CoChat
Sign Up
Cited by 830

The DOI matches its Crossref record; 6 of the 6 references it cites were found in a registry or an index.

Predictive performance of presence‐only species distribution models: a benchmark study with reproducible code

Valavi, Roozbeh, Guillera‐Arroita, Gurutzeta, Lahoz‐Monfort, José J., Elith, Jane·2022·Ecological Monographs
DOI 10.1002/ecm.1486
Open this paper in CoChat — free account

Abstract

Species distribution modeling (SDM) is widely used in ecology and conservation. Currently, the most available data for SDM are species presence‐only records (available through digital databases). There have been many studies comparing the performance of alternative algorithms for modeling presence‐only data. Among these, a 2006 paper from Elith and colleagues has been particularly influential in the field, partly because they used several novel methods (at the time) on a global data set that included independent presence–absence records for model evaluation. Since its publication, some of the algorithms have been further developed and new ones have emerged. In this paper, we explore patterns in predictive performance across methods, by reanalyzing the same data set (225 species from six different regions) using updated modeling knowledge and practices. We apply well‐established methods such as generalized additive models and MaxEnt, alongside others that have received attention more recently, including regularized regressions, point‐process weighted regressions, random forests, XGBoost, support vector machines, and the ensemble modeling framework biomod. All the methods we use include background samples (a sample of environments in the landscape) for model fitting. We explore impacts of using weights on the presence and background points in model fitting. We introduce new ways of evaluating models fitted to these data, using the area under the precision‐recall gain curve, and focusing on the rank of results. We find that the way models are fitted matters. The top method was an ensemble of tuned individual models. In contrast, ensembles built using the biomod framework with default parameters performed no better than single moderate performing models. Similarly, the second top performing method was a random forest parameterized to deal with many background samples (contrasted to relatively few presence records), which substantially outperformed other random forest implementations. We find that, in general, nonparametric techniques with the capability of controlling for model complexity outperformed traditional regression methods, with MaxEnt and boosted regression trees still among the top performing models. All the data and code with working examples are provided to make this study fully reproducible.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1002/ecm.1486

Cites

6 of 6 references verified0 citation errors · 0 not checkable

Statistics

Figures

Image 1
CC-BY-NC-ND · From the PDF
FIG. 2 AUCROC values for all modeling methods across all species. Each circular track is the AUCROC of a model for all species. Models and species are ordered by average AUCROC. The outer text labels indicate the regions. The numbers between NSW and NZ regions label the modeling method as mentioned in middle of the figure. The height of the outer histogram shows the log10(number of species presence-only records) in the training data set. Histogram colors follow three categories: light orange. < 100; pink, 100 -1,000; and dark brown, > 1,000 presences. This figure was created in Circos software (Krzywinski et al. 2009).
FIG. 2 AUCROC values for all modeling methods across all species. Each circular track is the AUCROC of a model for all species. Models and species are ordered by average AUCROC. The outer text labels indicate the regions. The numbers between NSW and NZ regions label the modeling method as mentioned in middle of the figure. The height of the outer histogram shows the log10(number of species presence-only records) in the training data set. Histogram colors follow three categories: light orange. < 100; pink, 100 -1,000; and dark brown, > 1,000 presences. This figure was created in Circos software (Krzywinski et al. 2009).CC-BY-NC-ND · From the PDF
FIG. 3 Mean AUCROC vs. mean COR for the models, summarized across all species. The gray bars (around each model) are standard errors estimated in a Bayesian mixed model, reflecting variation for an average species in an average region. The solid gray line is the least-squares line fitted on the models. The dashed gray lines delimit regions of low-, moderate-, and high-performance models, following the hierarchical clustering (represented in the top of the plot in Fig. 4). As a reminder, the ensemble model is the average of Lasso, GAM, MaxEnt, BRT, and RF down-sampled.
FIG. 3 Mean AUCROC vs. mean COR for the models, summarized across all species. The gray bars (around each model) are standard errors estimated in a Bayesian mixed model, reflecting variation for an average species in an average region. The solid gray line is the least-squares line fitted on the models. The dashed gray lines delimit regions of low-, moderate-, and high-performance models, following the hierarchical clustering (represented in the top of the plot in Fig. 4). As a reminder, the ensemble model is the average of Lasso, GAM, MaxEnt, BRT, and RF down-sampled.CC-BY-NC-ND · From the PDF
FIG. 4 P values of the post-hoc of the Friedman Aligned Rank test with Shaffer adjustments for AUCROC for a null hypothesis of no difference between pairs of models (values are rounded). The colors and the text inside each cell show different P values. Darker colors indicate higher P values (no significant difference in performance between models). The dendrogram on the top is based on both AUCROC and COR that clusters models in three main groups. The height of each bar shows the relative difference between each model/group.
FIG. 4 P values of the post-hoc of the Friedman Aligned Rank test with Shaffer adjustments for AUCROC for a null hypothesis of no difference between pairs of models (values are rounded). The colors and the text inside each cell show different P values. Darker colors indicate higher P values (no significant difference in performance between models). The dendrogram on the top is based on both AUCROC and COR that clusters models in three main groups. The height of each bar shows the relative difference between each model/group.CC-BY-NC-ND · From the PDF
FIG. 5 The difference between the performance of the default model with modified models in RF, SVM, and ensemble models. Other methods are presented in gray for reference. Bars and lines as in Fig. 3.
FIG. 5 The difference between the performance of the default model with modified models in RF, SVM, and ensemble models. Other methods are presented in gray for reference. Bars and lines as in Fig. 3.CC-BY-NC-ND · From the PDF
FIG. 6 Performance of models in NCEAS 2006, Phillips et al. (2017), and this study. For ease of comparison, this plot uses the same axes ranges and division lines as Fig. 3. The dashed and solid gray lines separate the low-, moderate-, and high-performance models, as in Fig. 3. The Phillips et al. (2017) models have no standard errors attached, as only their average performance was available to us. The MAXENTT from NCEAS 2006 study is a MaxEnt model with ' threshold ' feature (this feature is not included in the other studies).
FIG. 6 Performance of models in NCEAS 2006, Phillips et al. (2017), and this study. For ease of comparison, this plot uses the same axes ranges and division lines as Fig. 3. The dashed and solid gray lines separate the low-, moderate-, and high-performance models, as in Fig. 3. The Phillips et al. (2017) models have no standard errors attached, as only their average performance was available to us. The MAXENTT from NCEAS 2006 study is a MaxEnt model with ' threshold ' feature (this feature is not included in the other studies).CC-BY-NC-ND · From the PDF
FIG. 7 The average rank of AUCROC vs. (A) Pearson correlation between the predicted likelihood of presence and the presence -absence testing data (COR) and (B) area under the precision-recall gain curve (AUCPRG) for all the models. The gray dotted lines are the mean ranks. The values of the x and y -axis are reversed to have the lowest rank (better models) on the top-right corner and vice versa.
FIG. 7 The average rank of AUCROC vs. (A) Pearson correlation between the predicted likelihood of presence and the presence -absence testing data (COR) and (B) area under the precision-recall gain curve (AUCPRG) for all the models. The gray dotted lines are the mean ranks. The values of the x and y -axis are reversed to have the lowest rank (better models) on the top-right corner and vice versa.CC-BY-NC-ND · From the PDF
FIG. 9 Mean and standard error of the AUCROC for the species with < 30 and ≥ 30 presence points (with 61 and 164 species, respectively). Models are arranged based on median AUCROC in their corresponding group: tree-based, regression, and others.
FIG. 9 Mean and standard error of the AUCROC for the species with < 30 and ≥ 30 presence points (with 61 and 164 species, respectively). Models are arranged based on median AUCROC in their corresponding group: tree-based, regression, and others.CC-BY-NC-ND · From the PDF
Image 9
CC-BY-NC-ND · From the PDF
FIG. 11 Map of predicted relative likelihood of occurrence for the species nz30 in the NZ data set. The top-left figure shows the occurrence data used for fitting the models. The colored histogram to the left of each map shows the distribution of predicted values on all grid cells and the transparent gray histogram on top of it, demonstrates the distribution of predicted values on presence points (from testing data). The SVM weighted predicted values were very low, so its map is linearly rescaled between 0 and 1.
FIG. 11 Map of predicted relative likelihood of occurrence for the species nz30 in the NZ data set. The top-left figure shows the occurrence data used for fitting the models. The colored histogram to the left of each map shows the distribution of predicted values on all grid cells and the transparent gray histogram on top of it, demonstrates the distribution of predicted values on presence points (from testing data). The SVM weighted predicted values were very low, so its map is linearly rescaled between 0 and 1.CC-BY-NC-ND · From the PDF
FIG. 12 Model-fitting computation time for each method. Methods are arranged along the x -axis based on the median of times (from short to long). The ensemble model runtime was calculated by summing the computation times for all its underlying models (see Modeling methods ). XGBoost and MARS models were run with multiple CPU cores in parallel (eight cores).
FIG. 12 Model-fitting computation time for each method. Methods are arranged along the x -axis based on the median of times (from short to long). The ensemble model runtime was calculated by summing the computation times for all its underlying models (see Modeling methods ). XGBoost and MARS models were run with multiple CPU cores in parallel (eight cores).CC-BY-NC-ND · From the PDF

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Research works printed but not linked: 120

  1. 10.1007/978-3-540-30115-8_7 · DOI
  2. Ali A., 2015, Classification with class imbalance problem: a review, International Journal of Advances in Soft Computing and Its Applications, 7, 176
  3. 10.1111/j.1365-2664.2006.01214.x · DOI · Paper page
  4. 10.1111/j.1749-6632.2011.06440.x · DOI
  5. 10.1126/sciadv.aat4858 · DOI
  6. 10.1016/j.tree.2006.09.010 · DOI · Paper page
  7. 10.1002/ecs2.1825 · DOI
  8. 10.1016/S0378-1127(96)03753-X · DOI
  9. 10.1111/j.1600-0706.2012.00299.x · DOI
  10. 10.1111/j.2041-210X.2011.00172.x · DOI · Paper page
  11. 10.1016/j.ecolmodel.2016.10.004 · DOI
  12. 10.1016/j.tree.2008.10.008 · DOI · Paper page
  13. 10.1111/ddi.12144 · DOI
  14. Boyd K., 2012, Unachievable region in precision‐recall space and its effect on empirical evaluation
  15. 10.1111/2041-210X.12957 · DOI
  16. 10.1111/geb.12102 · DOI
  17. 10.32614/RJ-2016-017 · DOI
  18. Chen C., 2004, Using random forest to learn imbalanced data, University of California, Berkeley
  19. 10.1093/biomet/45.3-4.562 · DOI · Paper page
  20. Demšar J., 2006, Statistical comparisons of classifiers over multiple data sets, Journal of Machine Learning Research, 7, 30
  21. 10.1126/science.aat7526 · DOI
  22. 10.1002/ecm.1309 · DOI
  23. 10.1111/j.1365-2664.2006.01141.x · DOI
  24. 10.1111/ecog.03149 · DOI
  25. 10.1111/j.2006.0906-7590.04596.x · DOI · Paper page
  26. Elith J., 2019, Quantitative analyses in wildlife science
  27. 10.17161/bi.v15i2.13384 · DOI
  28. 10.1111/j.1472-4642.2007.00340.x · DOI
  29. Elith J., 2009, Spatial conservation prioritization: quantitative methods, 70, 10.1093/oso/9780199547760.003.0006 · DOI · Paper page
  30. 10.1111/j.1365-2656.2008.01390.x · DOI · Paper page
  31. 10.1111/j.1472-4642.2010.00725.x · DOI · Paper page
  32. 10.1007/s10980-009-9341-0 · DOI
  33. Fidler F., 2017, Metaresearch for evaluating reproducibility in ecology and evolution, BioScience, 67, 282
  34. 10.1111/2041-210X.12242 · DOI
  35. 10.1214/13-AOAS667 · DOI
  36. Flach P., 2015, Advances in neural information processing systems, 838
  37. 10.1017/CBO9780511810602 · DOI
  38. 10.1139/cjfr-2014-0562 · DOI
  39. 10.1016/S0167-9473(01)00065-2 · DOI · Paper page
  40. 10.18637/jss.v033.i01 · DOI · Paper page
  41. 10.1016/j.ins.2009.12.010 · DOI
  42. García S., 2008, An extension on “statistical comparisons of classifiers over multiple data sets” for all pairwise comparisons, Journal of Machine Learning Research, 9, 2677
  43. 10.1016/j.ecolmodel.2011.04.015 · DOI
  44. 10.1111/geb.12268 · DOI
  45. 10.1111/ele.12189 · DOI
  46. 10.1111/j.1461-0248.2005.00792.x · DOI
  47. 10.1017/9781139028271 · DOI
  48. 10.1016/j.ecolmodel.2004.07.012 · DOI
  49. 10.1016/j.ecolmodel.2019.108719 · DOI
  50. 10.1111/ddi.12892 · DOI
  51. 10.1111/ecog.04890 · DOI
  52. 10.7717/peerj.4278 · DOI
  53. 10.1111/j.1600-0587.2013.00321.x · DOI
  54. 10.1007/978-0-387-84858-7 · DOI · Paper page
  55. 10.1109/TKDE.2008.239 · DOI · Paper page
  56. 10.1007/s40823-016-0008-7 · DOI
  57. Herdter E., 2019, Using extreme gradient boosting (XGBoost) to evaluate the importance of a suite of environmental variables and to predict recruitment of young‐of‐the‐year spotted seatrout in Florida, bioRxiv, 543181
  58. 10.1016/j.ecolmodel.2006.05.017 · DOI
  59. Huang H., 2018, FVID: Fishing vessel type identification based on VMS trajectories, Journal of Ocean University of China, 17, 1
  60. Hughes‐Oliver J. M., 2018, Population and empirical PR curves for assessment of ranking algorithms
  61. 10.1111/2041-210X.13496 · DOI
  62. 10.1007/978-1-4614-7138-7 · DOI · Paper page
  63. 10.3233/IDA-2002-6504 · DOI · Paper page
  64. 10.1109/CIDU.2012.6382186 · DOI
  65. 10.1016/j.ecolmodel.2019.108927 · DOI
  66. 10.1186/1472-6947-11-51 · DOI
  67. 10.1093/oxfordjournals.pan.a004868 · DOI · Paper page
  68. 10.1101/gr.092759.109 · DOI
  69. Kull M., 2019, Advances in Neural Information Processing Systems, 12295
  70. 10.1111/2041-210X.12123 · DOI · Paper page
  71. 10.1016/j.ecolmodel.2006.05.022 · DOI
  72. 10.1016/j.ecolmodel.2012.07.003 · DOI
  73. 10.1111/j.1472-4642.2008.00491.x · DOI
  74. 10.1016/j.csda.2011.02.004 · DOI
  75. 10.1111/ecog.00845 · DOI
  76. 10.1111/j.1365-2699.2007.01720.x · DOI
  77. 10.7717/peerj.2849 · DOI
  78. 10.1111/j.1749-8198.2010.00351.x · DOI
  79. 10.1016/j.ecoinf.2019.100974 · DOI
  80. 10.1111/2041-210X.12261 · DOI · Paper page
  81. 10.1111/cobi.13669 · DOI
  82. 10.1002/ecm.1370 · DOI
  83. 10.1016/S0304-3800(00)00322-7 · DOI
  84. 10.1111/j.1365-2699.2006.01460.x · DOI
  85. 10.1111/ecog.03049 · DOI · Paper page
  86. 10.1016/j.ecolmodel.2005.03.026 · DOI · Paper page
  87. 10.1111/j.0906-7590.2008.5203.x · DOI · Paper page
  88. 10.1890/07-2153.1 · DOI · Paper page
  89. 10.1890/09-0760.1 · DOI
  90. 10.1890/12-1520.1 · DOI
  91. Plummer M., 2003, Proceedings of the 3rd International Workshop on Distributed Statistical Computing, 1
  92. 10.1111/bij.12517 · DOI
  93. 10.1007/978-3-319-96978-7_6 · DOI
  94. 10.1007/s10021-005-0054-1 · DOI · Paper page
  95. 10.1007/978-3-540-24694-7_32 · DOI
  96. Probst P., 2019, Tunability: Importance of hyperparameters of machine learning algorithms, Journal of Machine Learning Research, 20, 1
  97. R Core Team., 2020, R: A language and environment for statistical computing
  98. 10.1111/jbi.12227 · DOI · Paper page
  99. 10.1111/j.2007.0906-7590.05041.x · DOI
  100. 10.1016/j.ecolmodel.2005.10.003 · DOI
  101. 10.1111/2041-210X.12352 · DOI
  102. 10.1111/j.1541-0420.2012.01824.x · DOI
  103. 10.1111/ecog.02881 · DOI · Paper page
  104. 10.1111/ddi.12698 · DOI
  105. 10.1111/j.1365-2699.2004.01076.x · DOI · Paper page
  106. 10.1002/ece3.2332 · DOI
  107. 10.1029/2019WR024884 · DOI
  108. 10.1080/01621459.1986.10478341 · DOI
  109. 10.1111/2041-210X.13140 · DOI
  110. 10.1111/geb.12204 · DOI
  111. 10.1186/1471-2105-9-307 · DOI
  112. 10.1111/j.1600-0587.2008.05742.x · DOI
  113. 10.1111/2041-210X.13107 · DOI · Paper page
  114. 10.1111/ecog.05615 · DOI · Paper page
  115. 10.1111/j.1541-0420.2008.01116.x · DOI
  116. 10.1214/10-AOAS331 · DOI
  117. 10.1111/conl.12260 · DOI
  118. 10.1111/2041-210X.13106 · DOI
  119. 10.1111/2041-210x.12004 · DOI
  120. 10.1016/j.ecoinf.2019.05.003 · DOI

Cited by

Sorted by citations

See all 830

Metadata

Authors
Valavi, R., Guillera‐Arroita, G., Lahoz‐Monfort, J. J., & Elith, J.
Venue
Ecological Monographs
Publisher
Wiley
Published
2021-11-16
Volume
92
Issue
1
ISSN
  • 0012-9615
  • 1557-7015
DOI
10.1002/ecm.1486
Cited by
830
References
6
Language
en
Licence
creativecommons.org/licenses/by-nc-nd/4.0/ (vor)

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review