The DOI matches its Crossref record; 51 of the 51 references it cites were found in a registry or an index.
Explaining the unsuitability of the kappa coefficient in the assessment and comparison of the accuracy of thematic maps obtained by image classification
Open accessAbstract
The kappa coefficient is not an index of accuracy, indeed it is not an index of overall agreement but one of agreement beyond chance. Chance agreement is, however, irrelevant in an accuracy assessment and is anyway inappropriately modelled in the calculation of a kappa coefficient for typical remote sensing applications. The magnitude of a kappa coefficient is also difficult to interpret. Values that span the full range of widely used interpretation scales, indicating a level of agreement that equates to that estimated to arise from chance alone all the way through to almost perfect agreement, can be obtained from classifications that satisfy demanding accuracy targets (e.g. for a classification with overall accuracy of 95% the range of possible values of the kappa coefficient is −0.026 to 0.900). Comparisons of kappa coefficients are particularly challenging if the classes vary in their abundance (i.e. prevalence) as the magnitude of a kappa coefficient reflects not only agreement in labelling but also properties of the populations under study. It is shown that all of the arguments put forward for the use of the kappa coefficient in accuracy assessment are flawed and/or irrelevant as they apply equally to other, sometimes easier to calculate, measures of accuracy. Calls for the kappa coefficient to be abandoned from accuracy assessments should finally be heeded and researchers are encouraged to provide a set of simple measures and associated outputs such as estimates of per-class accuracy and the confusion matrix when assessing and comparing classification accuracy.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The Measurement of Observer Agreement for Categorical Data
Landis, J. R., & Koch, G. G. (1977) · Biometrics · doi:10.2307/2529310 · Cited by 81,148
A Coefficient of Agreement for Nominal Scales
Cohen, J. (1960) · Educational and Psychological Measurement · doi:10.1177/001316446002000104 · Cited by 42,398
Statistical Methods for Rates and Proportions
Fleiss, J. L., Levin, B., & Paik, M. C. (2003) · Wiley Series in Probability and Statistics · doi:10.1002/0471445428 · Cited by 18,646
A review of assessing the accuracy of classifications of remotely sensed data
Congalton, R. G. (1991) · Remote Sensing of Environment · doi:10.1016/0034-4257(91)90048-b · Cited by 7,724
A review of methods for the assessment of prediction errors in conservation presence/absence models
FIELDING, A. H., & BELL, J. F. (1997) · Environmental Conservation · doi:10.1017/s0376892997000088 · Cited by 6,956
ALLOUCHE, O., TSOAR, A., & KADMON, R. (2006) · Journal of Applied Ecology · doi:10.1111/j.1365-2664.2006.01214.x · Cited by 5,172
A land use and land cover classification system for use with remote sensor data
Anderson, J. R., Hardy, E. E., Roach, J. T., & Witmer, R. E. (1976) · Professional Paper · doi:10.3133/pp964 · Cited by 4,827
The kappa statistic in reliability studies: use, interpretation, and sample size requirements
Sim, J., & Wright, C. C. (2005) · Physical Therapy · doi:10.1093/ptj/85.3.257 · Cited by 4,093
Assessing the Accuracy of Remotely Sensed Data: Principles and Practices
Congalton, R., & Green, K. (1998) · Mapping Science · doi:10.1201/9781420048568 · Cited by 3,424
Good practices for estimating area and assessing accuracy of land change
Olofsson, P., Foody, G. M., Herold, M., Stehman, S. V., Woodcock, C. E., & Wulder, M. A. (2014) · Remote Sensing of Environment · doi:10.1016/j.rse.2014.02.015 · Cited by 3,078
Self-citationHigh agreement but low Kappa: I. the problems of two paradoxes
Feinstein, A. R., & Cicchetti, D. V. (1990) · Journal of Clinical Epidemiology · doi:10.1016/0895-4356(90)90158-l · Cited by 3,034
Assessing the Accuracy of Remotely Sensed Data: Principles and Practices
Foody, G. (2010) · The Photogrammetric Record · doi:10.1111/j.1477-9730.2010.00574_2.x · Cited by 2,806
Self-citationByrt, T., Bishop, J., & Carlin, J. B. (1993) · Journal of Clinical Epidemiology · doi:10.1016/0895-4356(93)90018-v · Cited by 2,179
Death to Kappa: birth of quantity disagreement and allocation disagreement for accuracy assessment
Pontius, R. G., & Millones, M. (2011) · International Journal of Remote Sensing · doi:10.1080/01431161.2011.552923 · Cited by 2,046
High agreement but low kappa: II. Resolving the paradoxes
Cicchetti, D. V., & Feinstein, A. R. (1990) · Journal of Clinical Epidemiology · doi:10.1016/0895-4356(90)90159-m · Cited by 1,838
Evaluating presence–absence models in ecology: the need to account for prevalence
Manel, S., Williams, H. C., & Ormerod, S. J. (2001) · Journal of Applied Ecology · doi:10.1046/j.1365-2664.2001.00647.x · Cited by 1,759
Selecting and interpreting measures of thematic classification accuracy
Stehman, S. V. (1997) · Remote Sensing of Environment · doi:10.1016/s0034-4257(97)00083-7 · Cited by 1,729
Coefficient Kappa: Some Uses, Misuses, and Alternatives
Brennan, R. L., & Prediger, D. J. (1981) · Educational and Psychological Measurement · doi:10.1177/001316448104100307 · Cited by 1,570
Large sample standard errors of kappa and weighted kappa.
Fleiss, J. L., Cohen, J., & Everitt, B. S. (1969) · Psychological Bulletin · doi:10.1037/h0028106 · Cited by 1,520
Comparing global vegetation maps with the Kappa statistic
Monserud, R. A., & Leemans, R. (1992) · Ecological Modelling · doi:10.1016/0304-3800(92)90003-w · Cited by 1,106
Foody, G. M. (2004) · Photogrammetric Engineering & Remote Sensing · doi:10.14358/pers.70.5.627 · Cited by 1,065
Self-citationAccuracy Assessment: A User's Perspective
Story, M., & Congalton, R. G. (1983) · University of New Hampshire Scholars Repository (University of New Hampshire at Manchester) · Cited by 1,040
Olofsson, P., Foody, G. M., Stehman, S. V., & Woodcock, C. E. (2013) · Remote Sensing of Environment · doi:10.1016/j.rse.2012.10.031 · Cited by 1,029
Self-citationESTIMATING PREVALENCE FROM THE RESULTS OF A SCREENING TEST
ROGAN, W. J., & GLADEN, B. (1978) · American Journal of Epidemiology · doi:10.1093/oxfordjournals.aje.a112510 · Cited by 1,001
Beyond kappa: a review of interrater agreement measures
Banerjee, M., Capozzoli, M., McSweeney, L., & Sinha, D. (1999) · Canadian Journal of Statistics · doi:10.2307/3315487 · Cited by 976
A coefficient of agreement as a measure of thematic classification accuracy.
Rosenfield, G. H., & Fitzpatrick-Lins, K. (1986) · Photogrammetric Engineering & Remote Sensing · Cited by 936
QUANTIFICATION ERROR VERSUS LOCATION ERROR IN COMPARISON OF CATEGORICAL MAPS
Pontius, R. G. (2000) · Clark Digital Commons (Clark University) · Cited by 924
Misinterpretation and misuse of the kappa statistic
MACLURE, M., & WILLETT, W. C. (1987) · American Journal of Epidemiology · doi:10.1093/aje/126.2.161 · Cited by 808
Statistical Methods for Rates and Proportions
(2004) · Technometrics · doi:10.1198/tech.2004.s812 · Cited by 681
Congalton, R. G., Oderwald, R. G., & Mead, R. A. (1983) · University of New Hampshire Scholars Repository (University of New Hampshire at Manchester) · Cited by 680
Key issues in rigorous accuracy assessment of land cover products
Stehman, S. V., & Foody, G. M. (2019) · Remote Sensing of Environment · doi:10.1016/j.rse.2019.05.018 · Cited by 651
Self-citationComparative assessment of the measures of thematic classification accuracy
Liu, C., Frazier, P., & Kumar, L. (2007) · Remote Sensing of Environment · doi:10.1016/j.rse.2006.10.010 · Cited by 465
Harshness in image classification accuracy assessment
Foody, G. M. (2008) · International Journal of Remote Sensing · doi:10.1080/01431160701442120 · Cited by 357
Self-citationAssessing the accuracy of land cover change with imperfect ground reference data
Foody, G. M. (2010) · Remote Sensing of Environment · doi:10.1016/j.rse.2010.05.003 · Cited by 333
Self-citationBehavior and interpretation of the κ statistic: Resolution of the two paradoxes
Lantz, C. A., & Nebenzahl, E. (1996) · Journal of Clinical Epidemiology · doi:10.1016/0895-4356(95)00571-4 · Cited by 311
Bias and prevalence effects on kappa viewed in terms of sensitivity and specificity
Hoehler, F. K. (2000) · Journal of Clinical Epidemiology · doi:10.1016/s0895-4356(99)00174-2 · Cited by 288
Diversity of decision-making models and the measurement of interrater agreement
Uebersax, J. S. (1987) · Psychological Bulletin · doi:10.1037/0033-2909.101.1.140 · Cited by 254
The SAGE Handbook of Remote Sensing
Warner, T. A., Nellis, M. D., & Foody, G. M. (2009) · Sage eBooks · doi:10.4135/9780857021052 · Cited by 239
Self-citationSpatial analysis of remote sensing image classification accuracy
Comber, A., Fisher, P., Brunsdon, C., & Khmag, A. (2012) · Remote Sensing of Environment · doi:10.1016/j.rse.2012.09.005 · Cited by 237
Foody, G. M. (2009) · Remote Sensing of Environment · doi:10.1016/j.rse.2009.03.014 · Cited by 231
Self-citationYe, S., Pontius, R. G., & Rakshit, R. (2018) · ISPRS Journal of Photogrammetry and Remote Sensing · doi:10.1016/j.isprsjprs.2018.04.002 · Cited by 198
The dependence of Cohen's kappa on the prevalence does not matter
Vach, W. (2005) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2004.02.021 · Cited by 182
Assessing the Performance of Classification Methods
Hand, D. J. (2012) · International Statistical Review · doi:10.1111/j.1751-5823.2012.00183.x · Cited by 177
Assessing the classification accuracy of multisource remote sensing data
Fitzgerald, R. W., & Lees, B. G. (1994) · Remote Sensing of Environment · doi:10.1016/0034-4257(94)90103-1 · Cited by 166
Practical Implications of Design-Based Sampling Inference for Thematic Map Accuracy Assessment
Stehman, S. V. (2000) · Remote Sensing of Environment · doi:10.1016/s0034-4257(99)00090-5 · Cited by 135
Recommendations for using the relative operating characteristic (ROC)
Pontius, R. G., & Parmentier, B. (2014) · Landscape Ecology · doi:10.1007/s10980-013-9984-8 · Cited by 131
Measures of spatio-temporal accuracy for time series land cover data
Tsutsumida, N., & Comber, A. (2015) · International Journal of Applied Earth Observation and Geoinformation · doi:10.1016/j.jag.2015.04.018 · Cited by 105
Testing the equality of two dependent kappa statistics
Donner, A. P., Shoukri, M. M., Klar, N. S., & Bartfay, E. (2000) · Statistics in Medicine · doi:10.1002/(sici)1097-0258(20000215)19:3<373::aid-sim337>3.0.co;2-y · Cited by 91
Foody, G. M. (2011) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2011.2174156 · Cited by 30
Self-citationDifferences in Inter‐Rater Reliability and Accuracy for a Treatment Adherence Scale
Wu, S. M., Whiteside, U. S., & Neighbors, C. (2007) · Cognitive Behaviour Therapy · doi:10.1080/16506070701584367 · Cited by 21
Thematic Accuracy Quality Control by Means of a Set of Multinomials
López, F. J. A., Avi, J. R., Fernández, M. V. A., & Balboa, J. L. G. (2019) · Applied Sciences · doi:10.3390/app9204240 · Cited by 9
Cited by
Phan, T. N., Kuch, V., & Lehnert, L. W. (2020) · Remote Sensing · doi:10.3390/rs12152411 · Cited by 616
A review on deep learning in UAV remote sensing
Osco, L. P., Marcato Junior, J., Marques Ramos, A. P., de Castro Jorge, L. A., Fatholahi, S. N., de Andrade Silva, J., Matsubara, E. T., Pistori, H., Gonçalves, W. N., & Li, J. (2021) · International Journal of Applied Earth Observation and Geoinformation · doi:10.1016/j.jag.2021.102456 · Cited by 458
Tassi, A., & Vizzari, M. (2020) · Remote Sensing · doi:10.3390/rs12223776 · Cited by 407
Maxwell, A. E., Warner, T. A., & Guillén, L. A. (2021) · Remote Sensing · doi:10.3390/rs13132450 · Cited by 327
Comparison of Machine Learning Pixel-Based Classifiers for Detecting Archaeological Ceramics
Argyrou, A., Agapiou, A., Papakonstantinou, A., & Alexakis, D. D. (2023) · Drones · doi:10.3390/drones7090578 · Cited by 19
Metadata
- Authors
- Foody, G. M.
- Venue
- Remote Sensing of Environment
- Publisher
- Elsevier BV
- Published
- 2020-03
- Volume
- 239
- Pages
- 111630
- ISSN
- 0034-4257 (ISSN-L)
- OpenAlex
- W2999585470
- Cited by
- 665
- References
- 51
- FWCI
- 38.86
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.