The DOI matches its Crossref record; 34 of the 34 references it cites were found in a registry or an index.
Evaluation of Sampling and Cross-Validation Tuning Strategies for Regional-Scale Machine Learning Classification
Open accessAbstract
High spatial resolution (1–5 m) remotely sensed datasets are increasingly being used to map land covers over large geographic areas using supervised machine learning algorithms. Although many studies have compared machine learning classification methods, sample selection methods for acquiring training and validation data for machine learning, and cross-validation techniques for tuning classifier parameters are rarely investigated, particularly on large, high spatial resolution datasets. This work, therefore, examines four sample selection methods—simple random, proportional stratified random, disproportional stratified random, and deliberative sampling—as well as three cross-validation tuning approaches—k-fold, leave-one-out, and Monte Carlo methods. In addition, the effect on the accuracy of localizing sample selections to a small geographic subset of the entire area, an approach that is sometimes used to reduce costs associated with training data collection, is investigated. These methods are investigated in the context of support vector machines (SVM) classification and geographic object-based image analysis (GEOBIA), using high spatial resolution National Agricultural Imagery Program (NAIP) orthoimagery and LIDAR-derived rasters, covering a 2,609 km2 regional-scale area in northeastern West Virginia, USA. Stratified-statistical-based sampling methods were found to generate the highest classification accuracy. Using a small number of training samples collected from only a subset of the study area provided a similar level of overall accuracy to a sample of equivalent size collected in a dispersed manner across the entire regional-scale dataset. There were minimal differences in accuracy for the different cross-validation tuning methods. The processing time for Monte Carlo and leave-one-out cross-validation were high, especially with large training sets. For this reason, k-fold cross-validation appears to be a good choice. Classifications trained with samples collected deliberately (i.e., not randomly) were less accurate than classifiers trained from statistical-based samples. This may be due to the high positive spatial autocorrelation in the deliberative training set. Thus, if possible, samples for training should be selected randomly; deliberative samples should be avoided.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The Elements of Statistical Learning
Hastie, T., Tibshirani, R., & Friedman, J. (2009) · Springer Series in Statistics · doi:10.1007/978-0-387-84858-7 · Cited by 25,970
Cross validatory choice and assessment of statistical predictions
Stone, M. (1974) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.2517-6161.1974.tb00994.x · Cited by 10,881
A review of assessing the accuracy of classifications of remotely sensed data
Congalton, R. G. (1991) · Remote Sensing of Environment · doi:10.1016/0034-4257(91)90048-b · Cited by 7,724
Note on the Sampling Error of the Difference Between Correlated Proportions or Percentages
McNemar, Q. (1947) · Psychometrika · doi:10.1007/bf02295996 · Cited by 4,754
Object based image analysis for remote sensing
Blaschke, T. (2010) · ISPRS Journal of Photogrammetry and Remote Sensing · doi:10.1016/j.isprsjprs.2009.06.004 · Cited by 4,482
Support vector machines in remote sensing: a review
Mountrakis, G., Im, J., & Ogole, C. (2011) · ISPRS Journal of Photogrammetry and Remote Sensing · doi:10.1016/j.isprsjprs.2010.11.001 · Cited by 3,473
A survey of image classification methods and techniques for improving classification performance
Lu, D., & Weng, Q. (2007) · International Journal of Remote Sensing · doi:10.1080/01431160600746456 · Cited by 3,402
Baatz, M. (2000) · Medical Entomology and Zoology · Cited by 2,072
Implementation of machine-learning classification in remote sensing: an applied review
Maxwell, A. E., Warner, T. A., & Fang, F. (2018) · International Journal of Remote Sensing · doi:10.1080/01431161.2018.1433343 · Cited by 2,057
caret: Classification and Regression Training
Kuhn, M. (2007) · CRAN: Contributed Packages · doi:10.32614/cran.package.caret · Cited by 1,786
Cross-Validation of Regression Models
Picard, R. R., & Cook, R. D. (1984) · Journal of the American Statistical Association · doi:10.1080/01621459.1984.10478083 · Cited by 1,547
Foody, G. M. (2004) · Photogrammetric Engineering & Remote Sensing · doi:10.14358/pers.70.5.627 · Cited by 1,065
Duro, D. C., Franklin, S. E., & Dubé, M. G. (2012) · Remote Sensing of Environment · doi:10.1016/j.rse.2011.11.020 · Cited by 989
A review of supervised object-based land-cover image classification
Ma, L., Li, M., Ma, X., Cheng, L., Du, P., & Liu, Y. (2017) · ISPRS Journal of Photogrammetry and Remote Sensing · doi:10.1016/j.isprsjprs.2017.06.001 · Cited by 900
Cracknell, M. J., & Reading, A. M. (2014) · Computers & Geosciences · doi:10.1016/j.cageo.2013.10.008 · Cited by 674
Automated parameterisation for multi-scale image segmentation on multiple layers
Drăguţ, L., Csillik, O., Eisank, C., & Tiede, D. (2014) · ISPRS Journal of Photogrammetry and Remote Sensing · doi:10.1016/j.isprsjprs.2013.11.018 · Cited by 621
Deciduous Forests of Eastern North America
BRAUN, E. L. (1951) · Soil Science · doi:10.1097/00010694-195102000-00012 · Cited by 618
Fassnacht, F. E., Hartig, F., Latifi, H., Berger, C., Hernández, J., Corvalán, P., & Koch, B. (2014) · Remote Sensing of Environment · doi:10.1016/j.rse.2014.07.028 · Cited by 438
Sampling designs for accuracy assessment of land cover
Stehman, S. V. (2009) · International Journal of Remote Sensing · doi:10.1080/01431160903131000 · Cited by 407
Kashani, A., Olsen, M., Parrish, C., & Wilson, N. (2015) · Sensors · doi:10.3390/s151128099 · Cited by 376
Training set size requirements for the classification of a specific class
Foody, G. M., Mathur, A., Sanchez‐Hernandez, C., & Boyd, D. S. (2006) · Remote Sensing of Environment · doi:10.1016/j.rse.2006.03.004 · Cited by 295
Brenning, A. (2012) · 2012 IEEE International Geoscience and Remote Sensing Symposium · doi:10.1109/igarss.2012.6352393 · Cited by 266
Stehman, S. V. (2014) · International Journal of Remote Sensing · doi:10.1080/01431161.2014.930207 · Cited by 255
Sample size determination for image classification accuracy assessment and comparison
Foody, G. M. (2009) · International Journal of Remote Sensing · doi:10.1080/01431160903130937 · Cited by 248
The SAGE Handbook of Remote Sensing
Warner, T. A., Nellis, M. D., & Foody, G. M. (2009) · Sage eBooks · doi:10.4135/9780857021052 · Cited by 239
THE EFFECT OF TRAINING STRATEGIES ON SUPERVISED CLASSIFICATION AT DIFFERENT SPATIAL RESOLUTIONS
Chen, D., & Stow, D. A. (2002) · Photogrammetric Engineering & Remote Sensing · Cited by 220
Evaluation of SVM, RVM and SMLR for Accurate Image Classification With Limited Ground Data
Pal, M., & Foody, G. M. (2012) · IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing · doi:10.1109/jstars.2012.2215310 · Cited by 142
Stehman, S. V. (2011) · Remote Sensing Letters · doi:10.1080/01431161.2010.541950 · Cited by 108
Foody, G. M., Pal, M., Rocchini, D., Garzón‐López, C. X., & Bastin, L. (2016) · ISPRS International Journal of Geo-Information · doi:10.3390/ijgi5110199 · Cited by 105
Maxwell, A. E., Warner, T. A., Strager, M. P., Conley, J., & Sharp, A. (2015) · International Journal of Remote Sensing · doi:10.1080/01431161.2014.1001086 · Cited by 95
Establishing Qualitative Geographic Sample Size in the Presence of Spatial Autocorrelation
Griffith, D. A. (2013) · Annals of the Association of American Geographers · doi:10.1080/00045608.2013.776884 · Cited by 75
Statistics for Earth and Environmental Scientists
Schuenemeyer, J. H., & Drew, L. J. (2010) · Wiley eBooks · doi:10.1002/9780470650707 · Cited by 67
Li, K., Li, J., Liu, Y., & Castiglione, A. (2016) · Communications in computer and information science · doi:10.1007/978-981-10-0356-1 · Cited by 12
CONCEPT FOR A COMPOUND ANALYSIS IN ACTIVE LEARNING FOR REMOTE SENSING
Wuttke, S., Middelmann, W., & Stilla, U. (2015) · The international archives of the photogrammetry, remote sensing and spatial information sciences/International archives of the photogrammetry, remote sensing and spatial information sciences · doi:10.5194/isprsarchives-xl-3-w2-273-2015 · Cited by 4
Cited by
Hoque, M. A.-A., Pradhan, B., Ahmed, N., & Roy, S. (2019) · Science of The Total Environment · doi:10.1016/j.scitotenv.2019.07.132 · Cited by 138
Metadata
- Authors
- A. Ramezan, C., A. Warner, T., & E. Maxwell, A.
- Venue
- Remote Sensing
- Publisher
- MDPI AG
- Published
- 2019-01-18
- Volume
- 11
- Issue
- 2
- Pages
- 185
- ISSN
- 2072-4292 (ISSN-L)
- OpenAlex
- W2910355480
- Cited by
- 322
- References
- 34
- FWCI
- 17.00
- Type
- journal article
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.