The DOI matches its Crossref record; 8 of the 8 references it cites were found in a registry or an index.
Assessing transferability of ecological models: an underappreciated aspect of statistical validation
Open accessAbstract
Summary: 1. Ecologists have long sought to distinguish relationships that are general from those that are idiosyncratic to a narrow range of conditions. Conventional methods of model validation and selection assess in‐ or out‐of‐sample prediction accuracy but do not assess model generality or transferability, which can lead to overestimates of performance when predicting in other locations, time periods or data sets. 2. We propose an intuitive method for evaluating transferability based on techniques currently in use in the area of species distribution modelling. The method involves cross‐validation in which data are assigned non‐randomly to groups that are spatially, temporally or otherwise distinct, thus using heterogeneity in the data set as a surrogate for heterogeneity among data sets. 3. We illustrate the method by applying it to distribution modelling of brook trout ( Salvelinus fontinalis Mitchill) and brown trout ( Salmo trutta Linnaeus) in western United States. We show that machine‐learning techniques such as random forests and artificial neural networks can produce models with excellent in‐sample performance but poor transferability, unless complexity is constrained. In our example, traditional linear models have greater transferability. 4. We recommend the use of a transferability assessment whenever there is interest in making inferences beyond the data set used for model fitting. Such an assessment can be used both for validation and for model selection and provides important information beyond what can be learned from conventional validation and selection techniques.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Breiman, L. (2001) · Machine Learning · doi:10.1023/a:1010933404324 · Cited by 133,538
The Elements of Statistical Learning
Hastie, T., Tibshirani, R., & Friedman, J. (2009) · Springer Series in Statistics · doi:10.1007/978-0-387-84858-7 · Cited by 25,970
The Elements of Statistical Learning
Hastie, T., Friedman, J., & Tibshirani, R. (2001) · Springer Series in Statistics · doi:10.1007/978-0-387-21606-5 · Cited by 24,162
Neural Networks for Pattern Recognition
Bishop, C. M. (1995) · doi:10.1093/oso/9780198538493.001.0001 · Cited by 12,287
A review of methods for the assessment of prediction errors in conservation presence/absence models
FIELDING, A. H., & BELL, J. F. (1997) · Environmental Conservation · doi:10.1017/s0376892997000088 · Cited by 6,956
Random forests for classification in ecology
Cutler, D. R., Edwards, T. C., Beard, K. H., Cutler, A., Hess, K. T., Gibson, J., & Lawler, J. J. (2007) · Ecology · doi:10.1890/07-0539.1 · Cited by 4,950
A survey of cross-validation procedures for model selection
Arlot, S., & Celisse, A. (2010) · Statistics Surveys · doi:10.1214/09-ss054 · Cited by 3,522
Model uncertainty, data mining and statistical inference
Chatfield, C. (1995) · Journal of the Royal Statistical Society. Series A (Statistics in Society) · doi:10.2307/2983440 · Cited by 1,103
Cited by
Cross‐validation strategies for data with temporal, spatial, hierarchical, or phylogenetic structure
Roberts, D. R., Bahn, V., Ciuti, S., Boyce, M. S., Elith, J., Guillera‐Arroita, G., Hauenstein, S., Lahoz‐Monfort, J. J., Schröder, B., Thuiller, W., et al. (2017) · Ecography · doi:10.1111/ecog.02881 · Cited by 3,689
Merow, C., Smith, M. J., & Silander, J. A. (2013) · Ecography · doi:10.1111/j.1600-0587.2013.07872.x · Cited by 3,392
Muscarella, R., Galante, P. J., Soley‐Guardia, M., Boria, R. A., Kass, J. M., Uriarte, M., & Anderson, R. P. (2014) · Methods in Ecology and Evolution · doi:10.1111/2041-210x.12261 · Cited by 1,962
Valavi, R., Elith, J., Lahoz‐Monfort, J. J., & Guillera‐Arroita, G. (2019) · Methods in Ecology and Evolution · doi:10.1111/2041-210x.13107 · Cited by 1,056
Meyer, H., Reudenbach, C., Wöllauer, S., & Nauss, T. (2019) · Ecological Modelling · doi:10.1016/j.ecolmodel.2019.108815 · Cited by 509
Fourcade, Y., Besnard, A. G., & Secondi, J. (2018) · Global Ecology and Biogeography · doi:10.1111/geb.12684 · Cited by 500
Machine learning-based global maps of ecological variables and the challenge of assessing them
Meyer, H., & Pebesma, E. (2022) · Nature Communications · doi:10.1038/s41467-022-29838-9 · Cited by 310
High-accuracy detection of malaria vector larval habitats using drone-based multispectral imagery
Carrasco-Escobar, G., Manrique, E., Ruiz-Cabrejos, J., Saavedra, M., Alava, F., Bickersmith, S., Prussing, C., Vinetz, J. M., Conn, J. E., Moreno, M., et al. (2019) · PLOS Neglected Tropical Diseases · doi:10.1371/journal.pntd.0007105 · Cited by 126
Nearest neighbour distance matching Leave‐One‐Out Cross‐Validation for map validation
Milà, C., Mateu, J., Pebesma, E., & Meyer, H. (2022) · Methods in Ecology and Evolution · doi:10.1111/2041-210x.13851 · Cited by 119
Ramasamy, M., Das, B., & Ramesh, R. (2022) · Journal of Pest Science · doi:10.1007/s10340-021-01411-1 · Cited by 107
Assessing and improving the transferability of current global spatial prediction models
Ludwig, M., Moreno‐Martinez, A., Hölzel, N., Pebesma, E., & Meyer, H. (2023) · Global Ecology and Biogeography · doi:10.1111/geb.13635 · Cited by 92
kNNDM CV: k-fold nearest-neighbour distance matching cross-validation for map accuracy estimation
Linnenbrink, J., Milà, C., Ludwig, M., & Meyer, H. (2024) · Geoscientific Model Development · doi:10.5194/gmd-17-5897-2024 · Cited by 51
“As an alternative, spatial CV methods such as block CV (Wenger and Olden, 2012; Valavi et al., 2019; Roberts et al., 2017) or buffered-LOO CV (Telford and Birks, 2009; Le Rest et al., 2014; Brenning, 2022) are often used.”
Models of alien species richness show moderate predictive accuracy and poor transferability
Capinha, C., Essl, F., Seebens, H., Pereira, H. M., & Kühn, I. (2018) · NeoBiota · doi:10.3897/neobiota.38.23518 · Cited by 16
“also expressing random noise), reducing their generality (Wenger and Olden 2012).”
BackgroundMethodInfluential
Metadata
- Authors
- Wenger, S. J., & Olden, J. D.
- Venue
- Methods in Ecology and Evolution
- Publisher
- Wiley
- Published
- 2012-01-23
- Volume
- 3
- Issue
- 2
- Pages
- 260-267
- ISSN
- 2041-210X (ISSN-L)
- OpenAlex
- W2156182980
- Cited by
- 642
- References
- 8
- FWCI
- 13.82
- Type
- journal article
- Language
- en
- Open access
- bronze
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.
