The DOI matches its Crossref record; 22 of the 22 references it cites were found in a registry or an index.
A comparison of resampling methods for remote sensing classification and accuracy assessment
Open accessAbstract
Abstract Maps that categorise the landscape into discrete units are a cornerstone of many scientific, management and conservation activities. The accuracy of these maps is often the primary piece of information used to make decisions about the mapping process or judge the quality of the final map. Variance is critical information when considering map accuracy, yet commonly reported accuracy metrics often do not provide that information. Various resampling frameworks have been proposed and shown to reconcile this issue, but have had limited uptake. In this paper, we compare the traditional approach of a single split of data into a training set (for classification) and test set (for accuracy assessment), to a resampling framework where the classification and accuracy assessment are repeated many times. Using a relatively simple vegetation mapping example and two common classifiers (maximum likelihood and random forest), we compare variance in mapped area estimates and accuracy assessment metrics (overall accuracy, kappa, user, producer, entropy, purity, quantity/allocation disagreement). Input field data points were repeatedly split into training and test sets via bootstrapping, Monte Carlo cross-validation (67:33 and 80:20 split ratios) and k-fold (5-fold) cross-validation. Additionally, within the cross-validation, we tested four designs: simple random, block hold-out, stratification by class, and stratification by both class and space. A classification was performed for every split of every methodological combination (100’s iterations each), creating sampling distributions for the mapped area of each class and the accuracy metrics. We found that regardless of resampling design, a single split of data into training and test sets results in a large variance in estimates of accuracy and mapped area. In the worst case, overall accuracy varied between ~40–80% in one resampling design, due only to random variation in partitioning into training and test sets. On the other hand, we found that all resampling procedures provided accurate estimates of error, and that they can also provide confidence intervals that are informative about the performance and uncertainty of the classifier. Importantly, we show that these confidence intervals commonly encompassed the magnitudes of increase or decrease in accuracy that are often cited in literature as justification for methodological or sampling design choices. We also show how a resampling approach enables generation of spatially continuous maps of classification uncertainty. Based on our results, we make recommendations about which resampling design to use and how it could be implemented. We also provide a fully worked mapping example, which includes traditional inference of uncertainty from the error matrix and provides examples for presenting the final map and its accuracy.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The Elements of Statistical Learning
Hastie, T., Tibshirani, R., & Friedman, J. (2009) · Springer Series in Statistics · doi:10.1007/978-0-387-84858-7 · Cited by 25,970
Status of land cover classification accuracy assessment
Foody, G. M. (2002) · Remote Sensing of Environment · doi:10.1016/s0034-4257(01)00295-4 · Cited by 4,580
Cross‐validation strategies for data with temporal, spatial, hierarchical, or phylogenetic structure
Roberts, D. R., Bahn, V., Ciuti, S., Boyce, M. S., Elith, J., Guillera‐Arroita, G., Hauenstein, S., Lahoz‐Monfort, J. J., Schröder, B., Thuiller, W., et al. (2017) · Ecography · doi:10.1111/ecog.02881 · Cited by 3,689
Self-citationGood practices for estimating area and assessing accuracy of land change
Olofsson, P., Foody, G. M., Herold, M., Stehman, S. V., Woodcock, C. E., & Wulder, M. A. (2014) · Remote Sensing of Environment · doi:10.1016/j.rse.2014.02.015 · Cited by 3,078
Predicting species distributions for conservation decisions
Guisan, A., Tingley, R., Baumgartner, J. B., Naujokaitis‐Lewis, I., Sutcliffe, P. R., Tulloch, A. I. T., Regan, T. J., Brotons, L., McDonald‐Madden, E., Mantyka‐Pringle, C., et al. (2013) · Ecology Letters · doi:10.1111/ele.12189 · Cited by 2,068
Self-citationDeath to Kappa: birth of quantity disagreement and allocation disagreement for accuracy assessment
Pontius, R. G., & Millones, M. (2011) · International Journal of Remote Sensing · doi:10.1080/01431161.2011.552923 · Cited by 2,046
Improvements on Cross-Validation: The 632+ Bootstrap Method
Efron, B., & Tibshirani, R. (1997) · Journal of the American Statistical Association · doi:10.1080/01621459.1997.10474007 · Cited by 1,636
Foody, G. M. (2004) · Photogrammetric Engineering & Remote Sensing · doi:10.14358/pers.70.5.627 · Cited by 1,065
Brenning, A. (2012) · 2012 IEEE International Geoscience and Remote Sensing Symposium · doi:10.1109/igarss.2012.6352393 · Cited by 266
MANAGING LANDSCAPES FOR CONSERVATION UNDER UNCERTAINTY
Burgman, M. A., Lindenmayer, D. B., & Elith, J. (2005) · Ecology · doi:10.1890/04-0906 · Cited by 201
Self-citation
Cited by
The global distribution and trajectory of tidal flats
Murray, N. J., Phinn, S. R., DeWitt, M., Ferrari, R., Johnston, R., Lyons, M. B., Clinton, N., Thau, D., & Fuller, R. A. (2019) · Nature · doi:10.1038/s41586-018-0805-8 · Cited by 1,063
Self-citationKey issues in rigorous accuracy assessment of land cover products
Stehman, S. V., & Foody, G. M. (2019) · Remote Sensing of Environment · doi:10.1016/j.rse.2019.05.018 · Cited by 651
High-resolution mapping of losses and gains of Earth’s tidal wetlands
Murray, N. J., Worthington, T. A., Bunting, P., Duce, S., Hagger, V., Lovelock, C. E., Lucas, R., Saunders, M. I., Sheaves, M., Spalding, M., et al. (2022) · Science · doi:10.1126/science.abm9583 · Cited by 489
Self-citationLi, R., Zheng, S., Zhang, C., Duan, C., Wang, L., & Atkinson, P. M. (2021) · ISPRS Journal of Photogrammetry and Remote Sensing · doi:10.1016/j.isprsjprs.2021.09.005 · Cited by 443
Nearest neighbour distance matching Leave‐One‐Out Cross‐Validation for map validation
Milà, C., Mateu, J., Pebesma, E., & Meyer, H. (2022) · Methods in Ecology and Evolution · doi:10.1111/2041-210x.13851 · Cited by 119
Metadata
- Authors
- Lyons, M. B., Keith, D. A., Phinn, S. R., Mason, T. J., & Elith, J.
- Venue
- Remote Sensing of Environment
- Publisher
- Elsevier BV
- Published
- 2018-04
- Volume
- 208
- Pages
- 145-153
- ISSN
- 0034-4257 (ISSN-L)
- OpenAlex
- W2790063226
- Cited by
- 275
- References
- 22
- FWCI
- 16.46
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: A comparison of resampling methods for remote sensing classification and accuracy assessment. https://app.cochat.ai/papers/10.1016/j.rse.2018.02.026
- The paper
- Lyons, M. B., Keith, D. A., Phinn, S. R., Mason, T. J., & Elith, J. (2018). A comparison of resampling methods for remote sensing classification and accuracy assessment. Remote Sensing of Environment. https://doi.org/10.1016/j.rse.2018.02.026
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.