Back to Stories

New Research Suggests Montana FWP Wolf Count High

Bozeman-based researcher says agency's model for counting wolves is wrong. FWP disagrees citing a lack of peer review.

Robert Crabtree, chief scientist at the Yellowstone Ecological Research Center based in Bozeman, has released a research paper questioning the modeling techniques used by Montana Fish, Wildlife and Parks. The paper has not been peer reviewed, something FWP brings into question. Photo by Holly Pippel
Robert Crabtree, chief scientist at the Yellowstone Ecological Research Center based in Bozeman, has released a research paper questioning the modeling techniques used by Montana Fish, Wildlife and Parks. The paper has not been peer reviewed, something FWP brings into question. Photo by Holly Pippel
by Laura Lundquist

As the comment period on Montana’s new wolf management plan nears its end, new research adds to questions about Montana’s wolf population estimates. But the timing of the research paper—released before peer-review—raises its own questions.

Last week, the Bozeman-based Yellowstone Ecological Research Center, an independent organization, pre-released a research paper that documented statistical problems with the population model developed and used by Fish, Wildlife and Parks to estimate wolf populations. The paper was preprinted by CABI, an international research organization based out of Britain. 

Robert Crabtree, YERC’s chief scientist, joined with statisticians from the University of Albert, Canada, to run simulations to assess the iPOM, or integrated Patch Occupancy Model, that FWP has used to estimate wolf populations since 2021 under the Gianforte administration. Their results indicated that iPOM has a bias that produces high population estimates, which could lure wildlife managers into thinking a species is doing better than it might be in reality.

Because field monitoring requires a good deal of time and effort, it costs more than computer modeling. So wildlife agencies dealing with elusive species find modeling an attractive option. However, models require users to make a number of assumptions and choose particular conditions. If those choices and assumptions don’t reflect reality, then, as computer scientists say, “garbage in equals garbage out.”

According to Crabtree and his colleagues, that’s the problem with iPOM, and the result is that FWP’s estimates of the wolf populations are significantly greater than what actually exists.

“We demonstrated iPOM has an inherent severe overestimation bias, which inflates [the number of packs] and abundance by a factor of 2.5 times [150% higher] by this one effect alone,” Crabtree wrote in the journal article.
The 2022 FWP commission approved an increase in the state killing quota to 456 wolves. This year, it was reduced to 313.  Photo by Gary Kramer/USFWS
The 2022 FWP commission approved an increase in the state killing quota to 456 wolves. This year, it was reduced to 313. Photo by Gary Kramer/USFWS

After wolves were delisted in Montana in 2011, biologists directly monitored them for five years with field observations and radio collars to make minimum counts for five years to be sure the delisting wasn’t premature. They calculated wolf abundance by figuring out the number of packs that actually existed and multiplying that by the pack size.

Then, from 2016 to 2020, FWP switched to a six-variable Patch Occupancy Model developed by FWP and Sarah Sells of the U.S. Geological Survey Montana Cooperative Wildlife Research Unit. Sells said at a public meeting Wednesday night that FWP used POM prior to 2016 and was using it alongside minimum counts.

In 2021, Sells and FWP changed the Patch Occupancy Model to iPOM and eliminated one variable: the amount of territory overlap, according to the 2020 Annual Wolf Report. When the wolf numbers from previous years were plugged into iPOM, each year’s population estimate increased. For example, in 2014, the POM population estimate of approximately 920 wolves jumped to about 1,125 using iPOM.

For his analysis, Crabtree told Mountain Journal he contacted Sells this past spring to get the modeling code and the data she used.

“I took painstaking effort to document everything, keep it completely above-board, consulted and talked to Sarah Sells many times since last spring. Everything is repeatable, and it’s up to the state now. We did their work for them,” Crabtree said. “I substantiated the major bias, and I was shocked to see what an overestimation bias there was.”

Crabtree’s paper said one problem with the iPOM is it uses three sub-models to calculate the variables, and using multiple models can compound error. One model calculates the total area occupied by wolves within a region while the other calculates Territory Size of a pack. Dividing the Occupied Area by Territory Size produces the estimated number of packs. A third sub-model determines wolf abundance by multiplying the number of packs by the pack size, just as before.

Crabtree’s article said rather than demographic models, the first two sub-models are spatial models, which are used to model distribution, not abundance. The Occupied Area sub-model assumes “closure,” meaning it assumes the wolf population in a grid square doesn’t change; no wolves are born, die, or leave or enter a grid square. However, the squares are large, about 232 square miles each. Such an assumption tends to overestimate the number of wolves, as other researchers found in Wisconsin.

“Based on biological knowledge of seasonal variation in wolf pack cohesion and dispersal combined with their normal high mobility and wide-ranging behavior, we concluded that many of [the model] assumptions were likely violated, especially the critical assumption of closure—no changes in occupancy due to movements or demography,” Crabtree wrote.
“We demonstrated iPOM has an inherent severe overestimation bias, which inflates [the number of packs] and abundance by a factor of 2.5 times [150% higher] by this one effect alone.” – Robert Crabtree, Yellowstone Ecological Research Center
The Crabtree article said the model could be improved if the grid size was reduced. But Wisconsin researcher Glenn Stauffer published in a 2021 Journal of Wildlife Management study that even cell sizes as small as 39 square miles caused populations to be overestimated. Stauffer also warned of problems using spatial models to estimate abundance, concluding that “estimates rely on somewhat subjective pack assignments, and likely deviate from true abundance to an unknown and possibly variable degree.”

Montana State University biologist Scott Creel published a white paper—not peer-reviewed—comparing iPOM’s territory size to what he’s documented and found iPOM underestimated pack territory, which would lead to an overestimation of the number of packs.

Crabtree’s paper also suggested that FWP should consider replacing iPOM with an alternative, such as a hierarchical model using accurate data, because “they provide a clear understanding of the relationship between data and the ecological processes of wildlife populations.” Another option is looking at genetic samples combined with capture-recapture methods.

In science, it’s not unusual for scientists to challenge each other on new ideas, and battling articles can appear in various journals as they seek to eliminate unsupported hypotheses. But for researchers, the important thing about being published in a scientific journal is passing peer review. Sells and other FWP biologists published the iPOM method in the journal Ecological Applications in August 2022. Crabtree’s work hasn’t been peer-reviewed yet.

When asked why he preprinted the article, Crabtree said it’s a quicker way to get through the peer-review process because he’s able to submit successive drafts to the CABI site and it’s a stronger paper after he incorporates all the comments from other scientists. The study was funded by the Jodar Family Foundation and the Rangeland Foundation.

According to several journals, including Springer Nature and the science and medical site PLOS, a preprint is a full draft of a research paper that is shared publicly before it has been peer reviewed. An increasing number of researchers are doing preprints, especially since the pandemic, because they allow for early feedback, increased visibility and ensure a researcher gets credit for a particular discovery. But scientists know that, being public, it could also lead to a bad reputation if there are too many preprints with no peer-reviewed follow up.

Crabtree told the Mountain Journal he intends to publish in a scientific journal—probably to Ecological Applications—after he gets feedback and suggestions from other researchers.
“We believe that peer review will address the severe misinterpretations in the Crabtree et al. analysis prior to being published." – Rebuttal statement from Montana FWP and Sarah Sells, USGS
Along the lines of feedback, Crabtree said Sells contacted him the week of December 3 after she saw the preprint and said he misunderstood how iPOM works. When asked for comment, FWP and Sells instead released a joint two-page response to Crabtree’s paper on December 13.

They said the Crabtree article made the mistake of assuming that iPOM used an entire grid square—232 square miles—where wolves were observed. They said they assume that any wolves they observe occupy each square for only about 20 percent of the time so that’s factored in to effectively decrease the grid size. They use Crabtree’s assumption to show that his occupied area would 3.38 times the size of their iPOM calculations.

“We believe that peer review will address the severe misinterpretations in the Crabtree et al. analysis prior to being published, and if it is published in the peer-reviewed literature we will respond in detail in that forum. However, as the article has already been posted online ahead of the wolf plan comment deadline and distributed to the press, we briefly address their key arguments,” the biologists wrote in their rebuttal. “We look forward to a scientific discussion of iPOM with Crabtree et al. conducted under the scrutiny of scientific peer review ... Until that occurs, however, the methods and conclusions of Crabtree et al. must be considered preliminary and weakly supported at best.”

In response to FWP’s rebuttal, Crabtree said he ran the iPOM model with the data he received from Sells, so they should have come out with the same results.

“What they describe in their response are not methods described in their publication. Is there another method they are using? Why wasn't this communicated to us earlier when we asked about clarification of the POM method used in iPOM? I even let [Sells] know that the overestimation problem was about spatial resolution. Why didn't she respond?” Crabtree said. 

By trying to reproduce FWP’s iPOM results, Crabtree was doing something that has led many scientists to voice concern about the “replication crisis.” Increasingly, research written up in peer-reviewed articles is not reproducible. Researchers have discovered over the past decade that lots of published findings in fields like psychology, sociology, medicine and economics don’t hold up when other researchers try to replicate them. But quality science, in addition to being peer-reviewed, must be reproducible by other scientists.

Whether the disagreement on results is due to error on Crabtree’s end or that of Sells’ remains to be seen.

During public comments to the FWP commission on December 14, attention was brought to Crabtree’s paper several times. Some say that enough questions have been raised about the iPOM that managers shouldn’t base wildlife decisions on the estimates until the model has been more thoroughly reviewed. Especially when there have been suggestions that FWP may use iPOM for grizzly bear management.

A year ago, WildEarth Guardians and Project Coyote filed a complaint against FWP, the FWP commission and the state of Montana, asking a Lewis and Clark County judge to stop the wolf season after the 2022 FWP commission approved an increase in the state  killing quota to 456 wolves. This year, it was reduced to 313. Part of the lawsuit takes FWP to task for changing the method used to estimate wolf populations without going through a public process. Another charge challenged the 2004 wolf management plan as being out-of-date, which led to FWP writing a new plan, the draft of which is currently accepting public comment.

Lizzy Pennock, WildEarth Guardians carnivore coexistence attorney, said WildEarth Guardians has submitted comments challenging the use of iPOM in FWP’s new wolf plan.

“We gave them [MSU biologist Scott] Creel’s statement in the scoping period, and they didn’t consider it in the draft EIS. They just regurgitated the same things: here’s why iPOM’s great. They didn’t engage in a meaningful analysis of ‘here are some potential risks of iPOM and how we’re going to manage for it,’” Pennock said.

Public comment on the draft wolf plan closes on December 19.

_______________________________________________________________________________________________________________________________

Mountain Journal is the only nonprofit, public-interest journalism organization of its kind dedicated to covering the wildlife and wild lands of Greater Yellowstone. We take pride in our work, yet to keep bold, independent journalism free, we need your support. Please donate here. Thank you.

Laura Lundquist
About Laura Lundquist

Laura Lundquist earned a journalism degree from the University of Montana in 2010, and has since covered the environmental beat for newspapers in Twin Falls, Idaho and Bozeman, in addition to a year of court reporting in Hamilton. She's now a freelance environmental reporter with the Missoula Current Online Journal.
Increase our impact by sharing this story.
GET OUR FREE NEWSLETTER
Defending Nature

Defend Truth &
Wild Places

SUPPORT US
SUPPORT US

Related Stories

May 24, 2024

Have Wolves Returned Yellowstone to its Natural State?
Wolves have affected the ecosystem in the park, but new study says they may be just one component of a trophic cascade.

December 22, 2023

Where the Rudder Meets the Road
In his new book, Crossings, author Ben Goldfarb charts a course through the complicated intersection of roads and ecology.

June 14, 2024

Are Toxic Agrichemicals Forcing Rapid Evolution in Yellowstone Elk?
New research suggests hazardous chemicals like pesticides and fertilizers used in farming, more so than wolves, contributed to a decline in Greater Yellowstone...