The DOI matches its Crossref record; 64 of the 64 references it cites were found in a registry or an index.
The area under the precision‐recall curve as a performance metric for rare binary events
Open accessAbstract
Species distribution models are used to study biogeographic patterns and guide decision‐making. The variable quality of these models makes it critical to assess whether a model's outputs are suitable for the intended use, but commonly used evaluation approaches are inappropriate for many ecological contexts. In particular, unrealistically high performance assessments have been associated with models for rare species and predictions over large geographic extents. We evaluated the area under the precision‐recall curve (AUC‐PR) as a performance metric for rare binary events, focusing on the assessment of species distribution models. Precision is the probability that a species is present given a predicted presence, while recall (more commonly called sensitivity) is the probability the model predicts presence in locations where the species has been observed. We simulated species at three levels of prevalence, compared AUC‐PR and the area under the receiver operating characteristic curve (AUC‐ROC) when the geographic extent of predictions was increased and assessed how well each metric reflected a model's utility to guide surveys for new populations. AUC‐PR was robust to species rarity and, unlike AUC‐ROC, not affected by an increasing geographic extent. The major advantages of AUC‐PR arise because it does not incorporate correctly predicted absences and is therefore less prone to exaggerate model performance for unbalanced datasets. AUC‐PR and precision were useful indicators of a model's utility for guiding surveys. We show that AUC‐PR has important advantages for evaluating models of rare species, and its benefits in the context of unbalanced binary responses will make it applicable for other ecological studies. By not considering the true negative quadrant of the confusion matrix, AUC‐PR ameliorates issues that arise when the geographic extent is increased beyond the species’ range or when a large number of background points are used when absence information is unavailable. However, no single metric captures all aspects of performance nor provides an absolute index that can be compared across datasets. Our results indicate AUC‐PR and precision can provide useful and intuitive metrics for evaluating a model's utility for guiding sampling, and can complement other metrics to help delineate a model's appropriate use.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The meaning and use of the area under a receiver operating characteristic (ROC) curve.
Hanley, J. A., & McNeil, B. J. (1982) · Radiology · doi:10.1148/radiology.143.1.7063747 · Cited by 21,856 · cited 1×
“AUC- ROC represents the probability that a randomly selected presence will have a higher continuous prediction than a randomly selected absence across all thresholds (Hanley & Mcneil, 1982).”
BackgroundRegularization Paths for Generalized Linear Models via Coordinate Descent
Friedman, J., Hastie, T., & Tibshirani, R. (2010) · Journal of Statistical Software · doi:10.18637/jss.v033.i01 · Cited by 19,927
Very high resolution interpolated climate surfaces for global land areas
Hijmans, R. J., Cameron, S. E., Parra, J. L., Jones, P. G., & Jarvis, A. (2005) · International Journal of Climatology · doi:10.1002/joc.1276 · Cited by 19,825
Classification and Regression by randomForest
Liaw, A. I., & Wiener, M. C. (2007) · Cited by 18,594
Maximum entropy modeling of species geographic distributions
Phillips, S. J., Anderson, R. P., & Schapire, R. E. (2006) · Ecological Modelling · doi:10.1016/j.ecolmodel.2005.03.026 · Cited by 15,396
Dormann, C. F., Elith, J., Bacher, S., Buchmann, C., Carl, G., Carré, G., Marquéz, J. R. G., Gruber, B., Lafourcade, B., Leitão, P. J., et al. (2013) · Ecography · doi:10.1111/j.1600-0587.2012.07348.x · Cited by 9,998
Novel methods improve prediction of species’ distributions from occurrence data
Elith*, J., H. Graham*, C., P. Anderson, R., Dudík, M., Ferrier, S., Guisan, A., J. Hijmans, R., Huettmann, F., R. Leathwick, J., Lehmann, A., et al. (2006) · Ecography · doi:10.1111/j.2006.0906-7590.04596.x · Cited by 9,088
A review of methods for the assessment of prediction errors in conservation presence/absence models
FIELDING, A. H., & BELL, J. F. (1997) · Environmental Conservation · doi:10.1017/s0376892997000088 · Cited by 6,956
A statistical explanation of MaxEnt for ecologists
Elith, J., Phillips, S. J., Hastie, T., Dudík, M., Chee, Y. E., & Yates, C. J. (2011) · Diversity and Distributions · doi:10.1111/j.1472-4642.2010.00725.x · Cited by 6,699
The relationship between Precision-Recall and ROC curves
Davis, J., & Goadrich, M. (2006) · Proceedings of the 23rd international conference on Machine learning - ICML '06 · doi:10.1145/1143844.1143874 · Cited by 6,559
Saito, T., & Rehmsmeier, M. (2015) · PLOS ONE · doi:10.1371/journal.pone.0118432 · Cited by 5,309
Two-sided confidence intervals for the single proportion: comparison of seven methods
Newcombe, R. G. (1998) · Statistics in Medicine · doi:10.1002/(sici)1097-0258(19980430)17:8<857::aid-sim777>3.0.co;2-e · Cited by 5,107
AUC: A misleading measure of the performance of predictive distribution models
Lobo, J. M., Jiménez‐Valverde, A., & Real, R. (2008) · Global Ecology and Biogeography · doi:10.1111/j.1466-8238.2007.00358.x · Cited by 3,499
Phillips, S. J., Dudík, M., Elith, J., Graham, C. H., Lehmann, A., Leathwick, J., & Ferrier, S. (2009) · Ecological Applications · doi:10.1890/07-2153.1 · Cited by 2,996
The art of modelling range-shifting species: The art of modelling range-shifting species
Elith, J., Kearney, M., & Phillips, S. (2010) · Methods in Ecology and Evolution · doi:10.1111/j.2041-210x.2010.00036.x · Cited by 2,813
Selecting thresholds of occurrence in the prediction of species distributions
Liu, C., Berry, P. M., Dawson, T. P., & Pearson, R. G. (2005) · Ecography · doi:10.1111/j.0906-7590.2005.03957.x · Cited by 2,701
Predicting species distributions for conservation decisions
Guisan, A., Tingley, R., Baumgartner, J. B., Naujokaitis‐Lewis, I., Sutcliffe, P. R., Tulloch, A. I. T., Regan, T. J., Brotons, L., McDonald‐Madden, E., Mantyka‐Pringle, C., et al. (2013) · Ecology Letters · doi:10.1111/ele.12189 · Cited by 2,068
Evaluating the predictive performance of habitat models developed using logistic regression
Pearce, J., & Ferrier, S. (2000) · Ecological Modelling · doi:10.1016/s0304-3800(00)00322-7 · Cited by 2,002
Barve, N., Barve, V., Jiménez-Valverde, A., Lira-Noriega, A., Maher, S. P., Peterson, A. T., Soberón, J., & Villalobos, F. (2011) · Ecological Modelling · doi:10.1016/j.ecolmodel.2011.02.011 · Cited by 1,881
Evaluating presence–absence models in ecology: the need to account for prevalence
Manel, S., Williams, H. C., & Ormerod, S. J. (2001) · Journal of Applied Ecology · doi:10.1046/j.1365-2664.2001.00647.x · Cited by 1,759
Evaluating the ability of habitat suitability models to predict species presences
Hirzel, A. H., Le Lay, G., Helfer, V., Randin, C., & Guisan, A. (2006) · Ecological Modelling · doi:10.1016/j.ecolmodel.2006.05.017 · Cited by 1,631
Rethinking receiver operating characteristic analysis applications in ecological niche modeling
Peterson, A. T., Papeş, M., & Soberón, J. (2008) · Ecological Modelling · doi:10.1016/j.ecolmodel.2007.11.008 · Cited by 1,526
Is my species distribution model fit for purpose? Matching data and models to applications
Guillera‐Arroita, G., Lahoz‐Monfort, J. J., Elith, J., Gordon, A., Kujala, H., Lentini, P. E., McCarthy, M. A., Tingley, R., & Wintle, B. A. (2015) · Global Ecology and Biogeography · doi:10.1111/geb.12268 · Cited by 961
Jiménez‐Valverde, A. (2012) · Global Ecology and Biogeography · doi:10.1111/j.1466-8238.2011.00683.x · Cited by 803
VanDerWal, J., Shoo, L. P., Graham, C., & Williams, S. E. (2009) · Ecological Modelling · doi:10.1016/j.ecolmodel.2008.11.010 · Cited by 793
Presence‐only modelling using MAXENT: when can we trust the inferences?
Yackulic, C. B., Chandler, R., Zipkin, E. F., Royle, J. A., Nichols, J. D., Campbell Grant, E. H., & Veran, S. (2013) · Methods in Ecology and Evolution · doi:10.1111/2041-210x.12004 · Cited by 752
On the selection of thresholds for predicting species occurrence with presence‐only data
Liu, C., Newell, G., & White, M. (2016) · Ecology and Evolution · doi:10.1002/ece3.1878 · Cited by 727
Not as good as they seem: the importance of concepts in species distribution modelling
Jiménez‐Valverde, A., Lobo, J. M., & Hortal, J. (2008) · Diversity and Distributions · doi:10.1111/j.1472-4642.2008.00496.x · Cited by 648
An evaluation of phrasal and clustered representations on a text categorization task
Lewis, D. D. (1992) · Proceedings of the 15th annual international ACM SIGIR conference on Research and development in information retrieval - SIGIR '92 · doi:10.1145/133160.133172 · Cited by 544
Using Niche‐Based Models to Improve the Sampling of Rare Species
GUISAN, A., BROENNIMANN, O., ENGLER, R., VUST, M., YOCCOZ, N. G., LEHMANN, A., & ZIMMERMANN, N. E. (2006) · Conservation Biology · doi:10.1111/j.1523-1739.2006.00354.x · Cited by 523
Fourcade, Y., Besnard, A. G., & Secondi, J. (2018) · Global Ecology and Biogeography · doi:10.1111/geb.12684 · Cited by 500
On Summary Measures of Skill in Rare Event Forecasting Based on Contingency Tables
Doswell, C. A., Davies-Jones, R., & Keller, D. L. (1990) · Weather and Forecasting · doi:10.1175/1520-0434(1990)005<0576:osmosi>2.0.co;2 · Cited by 477
Point process models for presence‐only analysis
Renner, I. W., Elith, J., Baddeley, A., Fithian, W., Hastie, T., Phillips, S. J., Popovic, G., & Warton, D. I. (2015) · Methods in Ecology and Evolution · doi:10.1111/2041-210x.12352 · Cited by 453
Leroy, B., Delsol, R., Hugueny, B., Meynard, C. N., Barhoumi, C., Barbet‐Massin, M., & Bellard, C. (2018) · Journal of Biogeography · doi:10.1111/jbi.13402 · Cited by 425
PRROC: computing and visualizing precision-recall and receiver operating characteristic curves in R
Grau, J., Grosse, I., & Keilwagen, J. (2015) · Bioinformatics · doi:10.1093/bioinformatics/btv153 · Cited by 412
Northrup, J. M., Hooten, M. B., Anderson, C. R., & Wittemyer, G. (2013) · Ecology · doi:10.1890/12-1688.1 · Cited by 405
Validation techniques for logistic regression models
Miller, M. E., Hui, S. L., & Tierney, W. M. (1991) · Statistics in Medicine · doi:10.1002/sim.4780100805 · Cited by 318
The continuing challenges of testing species distribution models
VAUGHAN, I. P., & ORMEROD, S. J. (2005) · Journal of Applied Ecology · doi:10.1111/j.1365-2664.2005.01052.x · Cited by 314
Boosting and Rocchio applied to text filtering
Schapire, R. E., Singer, Y., & Singhal, A. (1998) · Proceedings of the 21st annual international ACM SIGIR conference on Research and development in information retrieval · doi:10.1145/290941.290996 · Cited by 301
virtualspecies, an R package to generate virtual species distributions
Leroy, B., Meynard, C. N., Bellard, C., & Courchamp, F. (2016) · Ecography · doi:10.1111/ecog.01388 · Cited by 300
Finite-sample equivalence in statistical models for presence-only data
Fithian, W., & Hastie, T. (2013) · The Annals of Applied Statistics · doi:10.1214/13-aoas667 · Cited by 273
Poisson point process models solve the “pseudo-absence problem” for presence-only data in ecology
Warton, D. I., & Shepherd, L. C. (2010) · The Annals of Applied Statistics · doi:10.1214/10-aoas331 · Cited by 249
Modelling invasion for a habitat generalist and a specialist plant species
Evangelista, P. H., Kumar, S., Stohlgren, T. J., Jarnevich, C. S., Crall, A. W., Norman III, J. B., & Barnett, D. T. (2008) · Diversity and Distributions · doi:10.1111/j.1472-4642.2008.00486.x · Cited by 245
Self-citationFerro, C. A. T., & Stephenson, D. B. (2011) · Weather and Forecasting · doi:10.1175/waf-d-10-05030.1 · Cited by 236
Uncertainty of bioclimate envelope models based on the geographical distribution of species
Luoto, M., Pöyry, J., Heikkinen, R. K., & Saarinen, K. (2005) · Global Ecology and Biogeography · doi:10.1111/j.1466-822x.2005.00186.x · Cited by 209
Prevalence, thresholds and the performance of presence–absence models
Lawson, C. R., Hodgson, J. A., Wilson, R. J., & Richards, S. A. (2014) · Methods in Ecology and Evolution · doi:10.1111/2041-210x.12123 · Cited by 194
POC plots: calibrating species distribution models with presence‐only data
Phillips, S. J., & Elith, J. (2010) · Ecology · doi:10.1890/09-0760.1 · Cited by 191
Delimiting the geographical background in species distribution modelling
Acevedo, P., Jiménez‐Valverde, A., Lobo, J. M., & Real, R. (2012) · Journal of Biogeography · doi:10.1111/j.1365-2699.2012.02713.x · Cited by 182
The dependence of Cohen's kappa on the prevalence does not matter
Vach, W. (2005) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2004.02.021 · Cited by 182
Jiménez‐Valverde, A., Acevedo, P., Barbosa, A. M., Lobo, J. M., & Real, R. (2012) · Global Ecology and Biogeography · doi:10.1111/geb.12007 · Cited by 167
Evaluating diagnostic tests: The area under the ROC curve and the balance of errors
Hand, D. J. (2010) · Statistics in Medicine · doi:10.1002/sim.3859 · Cited by 145
Prevalence dependence in model goodness measures with special emphasis on true skill statistics
Somodi, I., Lepesi, N., & Botta‐Dukát, Z. (2017) · Ecology and Evolution · doi:10.1002/ece3.2654 · Cited by 140
Morán‐Ordóñez, A., Lahoz‐Monfort, J. J., Elith, J., & Wintle, B. A. (2016) · Global Ecology and Biogeography · doi:10.1111/geb.12545 · Cited by 132
Jiménez‐Valverde, A. (2014) · Biodiversity and Conservation · doi:10.1007/s10531-013-0606-1 · Cited by 115
How to assess the prediction accuracy of species presence–absence models without absence data?
Li, W., & Guo, Q. (2013) · Ecography · doi:10.1111/j.1600-0587.2013.07585.x · Cited by 112
Unachievable Region in Precision-Recall Space and Its Effect on Empirical Evaluation
Boyd, K., Costa, V. S., Davis, J. J., & Page, D. (2012) · PubMed · doi:10.48550/arxiv.1206.4667 · Cited by 96
Climate and very large wildland fires in the contiguous western USA
Stavros, E. N., Abatzoglou, J. T., Larkin, N. K., McKenzie, D., & Steel, E. A. (2014) · International Journal of Wildland Fire · doi:10.1071/wf13169 · Cited by 94
Using habitat suitability models to target invasive plant species surveys.
Crall, A., Jarnevich, C., Panke, B. J., Young, N., Renz, M., & Morisette, J. (2013) · Ecological Applications · doi:10.1890/12-0465.1 · Cited by 93
Self-citationSantika, T. (2011) · Global Ecology and Biogeography · doi:10.1111/j.1466-8238.2010.00581.x · Cited by 89
Advancing our thinking in presence‐only and used‐available analysis
Warton, D. I., & Aarts, G. M. (2013) · Journal of Animal Ecology · doi:10.1111/1365-2656.12071 · Cited by 68
Smith, A. B. (2013) · Diversity and Distributions · doi:10.1111/ddi.12031 · Cited by 66
Misleading prioritizations from modelling range shifts under climate change
Sofaer, H. R., Jarnevich, C. S., & Flather, C. H. (2018) · Global Ecology and Biogeography · doi:10.1111/geb.12726 · Cited by 66
Self-citationSpecies distribution modeling and prediction: A class imbalance problem
Johnson, R. A., Chawla, N. V., & Hellmann, J. J. (2012) · Conference on Intelligent Data Understanding · doi:10.1109/cidu.2012.6382186 · Cited by 63
An Analysis of Performance Measures for Binary Classifiers
Parker, C. (2011) · Proceedings · doi:10.1109/icdm.2011.21 · Cited by 57
Cited by
Valavi, R., Guillera‐Arroita, G., Lahoz‐Monfort, J. J., & Elith, J. (2022) · Ecological Monographs · doi:10.1002/ecm.1486 · Cited by 830
Want to model a species niche? A step-by-step guideline on correlative ecological niche modelling
Sillero, N., Arenas-Castro, S., Enriquez‐Urzelai, U., Vale, C. G., Sousa-Guedes, D., Martínez-Freiría, F., Real, R., & Barbosa, A. M. (2021) · Ecological Modelling · doi:10.1016/j.ecolmodel.2021.109671 · Cited by 456
Development and Delivery of Species Distribution Models to Inform Decision-Making
Sofaer, H. R., Jarnevich, C. S., Pearse, I. S., Smyth, R. L., Auer, S., Cook, G. L., Edwards, T. C., Guala, G. F., Howard, T. G., Morisette, J. T., et al. (2019) · BioScience · doi:10.1093/biosci/biz045 · Cited by 351
Self-citationLevy, T. J., Coppa, K., Cang, J., Barnaby, D. P., Paradis, M. D., Cohen, S. L., Makhnevich, A., van Klaveren, D., Kent, D. M., Davidson, K. W., et al. (2022) · Nature Communications · doi:10.1038/s41467-022-34646-2 · Cited by 14
“AUPR is a measure that is well-suited for imbalanced data, and its values range from the sample prevalence (indicating random performance) to 1 (indicating perfect classi fi cation) 67 .”
Background
Metadata
- Authors
- Sofaer, H. R., Hoeting, J. A., & Jarnevich, C. S.
- Venue
- Methods in Ecology and Evolution
- Publisher
- Wiley
- Published
- 2019-02-14
- Volume
- 10
- Issue
- 4
- Pages
- 565-577
- ISSN
- 2041-210X (ISSN-L)
- OpenAlex
- W2903942159
- Cited by
- 375
- References
- 64
- FWCI
- 10.89
- Type
- journal article
- Language
- en
- Open access
- hybrid
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.