Robert Crabtree, chief scientist at the Yellowstone Ecological Research Center based in Bozeman, has released a research paper questioning the modeling techniques used by Montana Fish, Wildlife and Parks. The paper has not been peer reviewed, something FWP brings into question. Photo by Holly Pippel
by Laura Lundquist
As the comment period on Montana’s new wolf management plan nears
its end, new research adds to questions about Montana’s wolf population
estimates. But the timing of the research paper—released before peer-review—raises
its own questions.
Last week, the Bozeman-based Yellowstone Ecological Research
Center, an independent organization, pre-released a research paper that
documented statistical problems with the population model developed and used by
Fish, Wildlife and Parks to estimate wolf populations. The paper was preprinted
by CABI, an international research organization based out of Britain.
Robert Crabtree, YERC’s chief scientist,
joined with statisticians from the University of Albert, Canada, to run
simulations to assess the iPOM, or integrated Patch Occupancy Model, that FWP
has used to estimate wolf populations since 2021 under the Gianforte
administration. Their results indicated that iPOM has a bias that produces high
population estimates, which could lure wildlife managers into thinking a
species is doing better than it might be in reality.
Because field monitoring requires a good deal of time and effort,
it costs more than computer modeling. So wildlife agencies dealing with elusive
species find modeling an attractive option. However, models require users to
make a number of assumptions and choose particular conditions. If those choices
and assumptions don’t reflect reality, then, as computer scientists say, “garbage
in equals garbage out.”
According to Crabtree and his colleagues, that’s the problem with
iPOM, and the result is that FWP’s estimates of the wolf populations are
significantly greater than what actually exists.
“We demonstrated iPOM has an inherent severe overestimation bias,
which inflates [the number of packs] and abundance by a factor of 2.5 times [150%
higher] by this one effect alone,” Crabtree wrote in the journal article.
The 2022 FWP commission approved an increase in the state killing quota to 456 wolves. This year, it was reduced to 313. Photo by Gary Kramer/USFWS
After wolves were delisted in Montana in 2011, biologists
directly monitored them for five years with field observations and radio
collars to make minimum counts for five years to be sure the delisting wasn’t
premature. They calculated wolf abundance by figuring out the number of packs
that actually existed and multiplying that by the pack size.
Then, from 2016 to 2020, FWP switched to a six-variable Patch
Occupancy Model developed by FWP and Sarah Sells of the U.S. Geological Survey
Montana Cooperative Wildlife Research Unit. Sells said at a public meeting
Wednesday night that FWP used POM prior to 2016 and was using it alongside
minimum counts.
In 2021, Sells and FWP changed the Patch Occupancy Model to iPOM
and eliminated one variable: the amount of territory overlap, according to the
2020 Annual Wolf Report. When the wolf numbers from previous years were plugged
into iPOM, each year’s population estimate increased. For example, in 2014, the
POM population estimate of approximately 920 wolves jumped to about 1,125 using
iPOM.
For his analysis, Crabtree told Mountain Journal he
contacted Sells this past spring to get the modeling code and the data she
used.
“I took painstaking effort to document everything, keep it
completely above-board, consulted and talked to Sarah Sells many times since
last spring. Everything is repeatable, and it’s up to the state now. We did
their work for them,” Crabtree said. “I substantiated the major bias, and I was
shocked to see what an overestimation bias there was.”
Crabtree’s paper said one problem with the iPOM is it uses three sub-models to
calculate the variables, and using multiple models can compound error. One
model calculates the total area occupied by wolves within a region while the
other calculates Territory Size of a pack. Dividing the Occupied Area by
Territory Size produces the estimated number of packs. A third sub-model
determines wolf abundance by multiplying the number of packs by the pack size,
just as before.
Crabtree’s article said rather than demographic models, the first
two sub-models are spatial models, which are used to model distribution, not
abundance. The Occupied Area sub-model assumes “closure,” meaning it assumes
the wolf population in a grid square doesn’t change; no wolves are born, die,
or leave or enter a grid square. However, the squares are large, about 232
square miles each. Such an assumption tends to overestimate the number of
wolves, as other researchers found in Wisconsin.
“Based on biological knowledge of seasonal variation in wolf pack
cohesion and dispersal combined with their normal high mobility and
wide-ranging behavior, we concluded that many of [the model] assumptions were
likely violated, especially the critical assumption of closure—no changes in
occupancy due to movements or demography,” Crabtree wrote.
“We demonstrated iPOM has an inherent severe overestimation bias, which inflates [the number of packs] and abundance by a factor of 2.5 times [150% higher] by this one effect alone.” – Robert Crabtree, Yellowstone Ecological Research Center
The Crabtree article said the model could be improved if the grid
size was reduced. But Wisconsin researcher Glenn Stauffer published in a 2021 Journal
of Wildlife Management study that even cell sizes as small as 39 square
miles caused populations to be overestimated. Stauffer also warned of problems
using spatial models to estimate abundance, concluding that “estimates rely on
somewhat subjective pack assignments, and likely deviate from true abundance to
an unknown and possibly variable degree.”
Montana State University biologist Scott Creel published a white
paper—not peer-reviewed—comparing iPOM’s territory size to what he’s documented
and found iPOM underestimated pack territory, which would lead to an
overestimation of the number of packs.
Crabtree’s paper also suggested that FWP should consider
replacing iPOM with an alternative, such as a hierarchical model using accurate
data, because “they provide a clear understanding of the relationship between
data and the ecological processes of wildlife populations.” Another option is
looking at genetic samples combined with capture-recapture methods.
In science, it’s not unusual for scientists to challenge each
other on new ideas, and battling articles can appear in various journals as
they seek to eliminate unsupported hypotheses. But for researchers, the
important thing about being published in a scientific journal is passing peer
review. Sells and other FWP biologists published the iPOM method in the journal
Ecological Applications in August 2022. Crabtree’s work hasn’t been
peer-reviewed yet.
When asked why he preprinted the article, Crabtree said it’s a
quicker way to get through the peer-review process because he’s able to submit
successive drafts to the CABI site and it’s a stronger paper after he
incorporates all the comments from other scientists. The study was funded by
the Jodar Family Foundation and the Rangeland Foundation.
According to several journals, including Springer Nature and the science and medical site PLOS, a preprint
is a full draft of a research paper that is shared publicly before it has been
peer reviewed. An increasing number of researchers are doing preprints,
especially since the pandemic, because they allow for early feedback, increased
visibility and ensure a researcher gets credit for a particular discovery. But scientists
know that, being public, it could also lead to a bad reputation if there are
too many preprints with no peer-reviewed follow up.
Crabtree told the Mountain Journal he intends to publish
in a scientific journal—probably to Ecological Applications—after he
gets feedback and suggestions from other researchers.
“We believe that peer review will address the severe misinterpretations in the Crabtree et al. analysis prior to being published." – Rebuttal statement from Montana FWP and Sarah Sells, USGS
Along the lines of feedback, Crabtree said Sells contacted him the
week of December 3 after she saw the preprint and said he misunderstood how
iPOM works. When asked for comment, FWP and Sells instead released a joint two-page response to
Crabtree’s paper on December 13.
They said the Crabtree article made the mistake of assuming that
iPOM used an entire grid square—232 square miles—where wolves were observed.
They said they assume that any wolves they observe occupy each square for only
about 20 percent of the time so that’s factored in to effectively decrease the
grid size. They use Crabtree’s assumption to show that his occupied area would
3.38 times the size of their iPOM calculations.
“We believe that peer review will address the severe
misinterpretations in the Crabtree et al. analysis prior to being published,
and if it is published in the peer-reviewed literature we will respond in
detail in that forum. However, as the article has already been posted online
ahead of the wolf plan comment deadline and distributed to the press, we
briefly address their key arguments,” the biologists wrote in their rebuttal. “We look forward to a scientific discussion of iPOM with Crabtree
et al. conducted under the scrutiny of scientific peer review ... Until that occurs,
however, the methods and conclusions of Crabtree et al. must be considered
preliminary and weakly supported at best.”
In response to FWP’s rebuttal, Crabtree said he ran the iPOM
model with the data he received from Sells, so they should have come out with
the same results.
“What they describe in their response are not methods described
in their publication. Is there another method they are using? Why wasn't this
communicated to us earlier when we asked about clarification of the POM
method used in iPOM? I even let [Sells] know that the overestimation
problem was about spatial resolution. Why didn't she respond?” Crabtree
said.
By trying to reproduce FWP’s iPOM results, Crabtree was doing
something that has led many scientists to voice concern about the “replication
crisis.” Increasingly, research written up in peer-reviewed articles is not
reproducible. Researchers have discovered over the past decade that lots of published
findings in fields like psychology, sociology, medicine and economics don’t hold up when other researchers try to
replicate them. But quality science, in addition to being peer-reviewed, must
be reproducible by other scientists.
Whether the disagreement on results is due to error on Crabtree’s
end or that of Sells’ remains to be seen.
During public comments to the FWP commission on December 14,
attention was brought to Crabtree’s paper several times. Some say that enough
questions have been raised about the iPOM that managers shouldn’t base wildlife
decisions on the estimates until the model has been more thoroughly reviewed.
Especially when there have been suggestions that FWP may use iPOM for grizzly
bear management.
A year ago, WildEarth Guardians and Project Coyote filed a complaint against FWP, the FWP
commission and the state of Montana, asking a Lewis and Clark County judge to stop the wolf season after the 2022
FWP commission approved an increase in the state killing quota to 456 wolves. This year, it was
reduced to 313. Part of the lawsuit takes FWP to task for changing the method
used to estimate wolf populations without going through a public process.
Another charge challenged the 2004 wolf management plan as being out-of-date,
which led to FWP writing a new plan, the draft of which is currently accepting
public comment.
Lizzy Pennock, WildEarth Guardians carnivore coexistence attorney, said
WildEarth Guardians has submitted comments challenging the use of iPOM in FWP’s
new wolf plan.
“We gave them [MSU biologist Scott] Creel’s statement in the
scoping period, and they didn’t consider it in the draft EIS. They just
regurgitated the same things: here’s why iPOM’s great. They didn’t engage in a
meaningful analysis of ‘here are some potential risks of iPOM and how we’re
going to manage for it,’” Pennock said.
Public comment on the draft wolf plan closes on December 19.
Mountain Journal is the only nonprofit, public-interest journalism organization of its kind dedicated to covering the wildlife and wild lands of Greater Yellowstone. We take pride in our work, yet to keep bold, independent journalism free, we need your support. Please donate here. Thank you.
About Laura Lundquist
Laura Lundquist earned a journalism degree from the University of Montana in 2010, and has since covered the environmental beat for newspapers in Twin Falls, Idaho and Bozeman, in addition to a year of court reporting in Hamilton. She's now a freelance environmental reporter with the Missoula Current Online Journal.
