The DOI matches its Crossref record; 38 of the 38 references it cites were found in a registry or an index.
Death to Kappa: birth of quantity disagreement and allocation disagreement for accuracy assessment
Open accessAbstract
The family of Kappa indices of agreement claim to compare a map's observed classification accuracy relative to the expected accuracy of baseline maps that can have two types of randomness: (1) random distribution of the quantity of each category and (2) random spatial allocation of the categories. Use of the Kappa indices has become part of the culture in remote sensing and other fields. This article examines five different Kappa indices, some of which were derived by the first author in 2000. We expose the indices' properties mathematically and illustrate their limitations graphically, with emphasis on Kappa's use of randomness as a baseline, and the often-ignored conversion from an observed sample matrix to the estimated population matrix. This article concludes that these Kappa indices are useless, misleading and/or flawed for the practical applications in remote sensing that we have seen. After more than a decade of working with these indices, we recommend that the profession abandon the use of Kappa indices for purposes of accuracy assessment and map comparison, and instead summarize the cross-tabulation matrix with two much simpler summary parameters: quantity disagreement and allocation disagreement. This article shows how to compute these two parameters using examples taken from peer-reviewed literature.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The Measurement of Observer Agreement for Categorical Data
Landis, J. R., & Koch, G. G. (1977) · Biometrics · doi:10.2307/2529310 · Cited by 81,148 · cited 2×
“It makes no sense to have a universal standard for accuracy in practical applications (Foody 2008 ), in spite of temptations to establish such standards (Landis and Koch 1977 , Monserud and Leemans 1992 ) because a universal standard is not related to any specific research question or study area.”
BackgroundA Coefficient of Agreement for Nominal Scales
Cohen, J. (1960) · Educational and Psychological Measurement · doi:10.1177/001316446002000104 · Cited by 42,398
Photogrammetric Engineering and Remote Sensing
Nicol, J., King, B., Quattrochi, D., Dowman, I. J., Ehlers, M., & Xiaoli, D. (2007) · UCL Discovery (University College London) · Cited by 9,325
A review of assessing the accuracy of classifications of remotely sensed data
Congalton, R. G. (1991) · Remote Sensing of Environment · doi:10.1016/0034-4257(91)90048-b · Cited by 7,724
A review of methods for the assessment of prediction errors in conservation presence/absence models
FIELDING, A. H., & BELL, J. F. (1997) · Environmental Conservation · doi:10.1017/s0376892997000088 · Cited by 6,956 · cited 2×
“…= ⎛ ⎜⎜⎜⎝ nijJ∑ j=1 nij ⎞ ⎟⎟⎟⎠ ⎛ ⎜⎜⎜⎝ NiJ∑ i=1 Ni ⎞ ⎟⎟⎟⎠ (1) 2.4 Parameters to summarize the population matrix There are numerous possible parameters to summarize the information in the 200 population matrix (Ma and Redmond 1995, Fielding and Bell 1997, Stehman 1997, Liu et al. 2007).”
BackgroundInfluentialALLOUCHE, O., TSOAR, A., & KADMON, R. (2006) · Journal of Applied Ecology · doi:10.1111/j.1365-2664.2006.01214.x · Cited by 5,172
Status of land cover classification accuracy assessment
Foody, G. M. (2002) · Remote Sensing of Environment · doi:10.1016/s0034-4257(01)00295-4 · Cited by 4,580
Assessing the Accuracy of Remotely Sensed Data: Principles and Practices
Congalton, R., & Green, K. (1998) · Mapping Science · doi:10.1201/9781420048568 · Cited by 3,424
Assessing the Accuracy of Remotely Sensed Data: Principles and Practices
Foody, G. (2010) · The Photogrammetric Record · doi:10.1111/j.1477-9730.2010.00574_2.x · Cited by 2,806
Measures of Association for Cross Classifications
Goodman, L. A., & Kruskal, W. H. (1979) · Springer Series in Statistics · doi:10.1007/978-1-4612-9995-0 · Cited by 2,244
Reliability of Content Analysis: The Case of Nominal Scale Coding
Scott, W. A. (1955) · Public Opinion Quarterly · doi:10.1086/266577 · Cited by 1,853
Selecting and interpreting measures of thematic classification accuracy
Stehman, S. V. (1997) · Remote Sensing of Environment · doi:10.1016/s0034-4257(97)00083-7 · Cited by 1,729
Coefficient Kappa: Some Uses, Misuses, and Alternatives
Brennan, R. L., & Prediger, D. J. (1981) · Educational and Psychological Measurement · doi:10.1177/001316448104100307 · Cited by 1,570
Comparing global vegetation maps with the Kappa statistic
Monserud, R. A., & Leemans, R. (1992) · Ecological Modelling · doi:10.1016/0304-3800(92)90003-w · Cited by 1,106
Foody, G. M. (2004) · Photogrammetric Engineering & Remote Sensing · doi:10.14358/pers.70.5.627 · Cited by 1,065
Comparing the input, output, and validation maps for several models of land change
Pontius, R. G., Boersma, W., Castella, J.-C., Clarke, K., de Nijs, T., Dietzel, C., Duan, Z., Fotsing, E., Goldstein, N., Kok, K., et al. (2008) · The Annals of Regional Science · doi:10.1007/s00168-007-0138-2 · Cited by 940
Self-citationA coefficient of agreement as a measure of thematic classification accuracy.
Rosenfield, G. H., & Fitzpatrick-Lins, K. (1986) · Photogrammetric Engineering & Remote Sensing · Cited by 936
Detecting important categorical land changes while accounting for persistence
Pontius, R. G., Shusas, E., & McEachern, M. (2003) · Agriculture Ecosystems & Environment · doi:10.1016/j.agee.2003.09.008 · Cited by 932
Self-citationQUANTIFICATION ERROR VERSUS LOCATION ERROR IN COMPARISON OF CATEGORICAL MAPS
Pontius, R. G. (2000) · Clark Digital Commons (Clark University) · Cited by 924
Self-citationDesign and Analysis for Thematic Map Accuracy Assessment
Stehman, S. V., & Czaplewski, R. L. (1998) · Remote Sensing of Environment · doi:10.1016/s0034-4257(98)00010-8 · Cited by 710
Congalton, R. G., Oderwald, R. G., & Mead, R. A. (1983) · University of New Hampshire Scholars Repository (University of New Hampshire at Manchester) · Cited by 680
Measures of Association for Cross Classifications*
Goodman, L. A., & Kruskal, W. H. (1954) · Journal of the American Statistical Association · doi:10.1080/01621459.1954.10501231 · Cited by 678
The Kappa Statistic: A Second Look
Eugenio, B. D., & Glass, M. (2004) · Computational Linguistics · doi:10.1162/089120104773633402 · Cited by 489
Comparative assessment of the measures of thematic classification accuracy
Liu, C., Frazier, P., & Kumar, L. (2007) · Remote Sensing of Environment · doi:10.1016/j.rse.2006.10.010 · Cited by 465
Sampling designs for accuracy assessment of land cover
Stehman, S. V. (2009) · International Journal of Remote Sensing · doi:10.1080/01431160903131000 · Cited by 407
QUANTIFICATION ERROR VERSUS LOCATION ERROR IN COMPARISON OF CATEGORICAL MAPS
Pontius (2000) · Photogrammetric Engineering & Remote Sensing · Cited by 394
Visser, H., & de Nijs, T. (2006) · Environmental Modelling & Software · doi:10.1016/j.envsoft.2004.11.013 · Cited by 370
Harshness in image classification accuracy assessment
Foody, G. M. (2008) · International Journal of Remote Sensing · doi:10.1080/01431160701442120 · Cited by 357
Pontius, R. G. (2002) · Clark Digital Commons (Clark University) · Cited by 300
Self-citationResults and implications of a study of fifteen years of satellite image classification experiments
Wilkinson, G. G. (2005) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2004.837325 · Cited by 255
Components of information for multiple resolution comparison between maps that share a real variable
Pontius, R. G., Thontteh, O., & Chen, H. (2008) · Environmental and Ecological Statistics · doi:10.1007/s10651-007-0043-y · Cited by 170
Self-citationAickin, M. (1990) · Biometrics · doi:10.2307/2531434 · Cited by 122
Accuracy Assessment for a Simulation Model of Amazonian Deforestation
Pontius, R. G., Walker, R. T., Yao-Kumah, R., Arima, E. Y., Aldrich, S., Caldas, M. M., & Vergara, D. (2007) · Annals of the Association of American Geographers · doi:10.1111/j.1467-8306.2007.00577.x · Cited by 112
Self-citationUncertainty in Extrapolations of Predictive Land-Change Models
Pontius, R. G., & Spencer, J. (2005) · Environment and Planning B Planning and Design · doi:10.1068/b31152 · Cited by 94
Self-citationMulti-method assessment of map similarity
Hagen‐Zanker, A. (2002) · Cited by 94
Wu, S.-S., Qiu, X., Usery, E. L., & Wang, L. (2008) · Annals of the Association of American Geographers · doi:10.1080/00045600802459028 · Cited by 83
Ruelland, D., Dezetter, A., Puech, C., & Ardoin‐Bardin, S. (2008) · International Journal of Remote Sensing · doi:10.1080/01431160701758699 · Cited by 43
Evaluating interrater agreement in SPICE-based assessments
Jung, H. (2003) · Computer Standards & Interfaces · doi:10.1016/s0920-5489(03)00018-7 · Cited by 22
Cited by
Good practices for estimating area and assessing accuracy of land change
Olofsson, P., Foody, G. M., Herold, M., Stehman, S. V., Woodcock, C. E., & Wulder, M. A. (2014) · Remote Sensing of Environment · doi:10.1016/j.rse.2014.02.015 · Cited by 3,078
Liu, X., Liang, X., Li, X., Xu, X., Ou, J., Chen, Y., Li, S., Wang, S., & Pei, F. (2017) · Landscape and Urban Planning · doi:10.1016/j.landurbplan.2017.09.019 · Cited by 1,744
Characterising performance of environmental models
Bennett, N. D., Croke, B. F. W., Guariso, G., Guillaume, J. H. A., Hamilton, S. H., Jakeman, A. J., Marsili-Libelli, S., Newham, L. T. H., Norton, J. P., Perrin, C., et al. (2013) · Environmental Modelling & Software · doi:10.1016/j.envsoft.2012.09.011 · Cited by 1,536
Olofsson, P., Foody, G. M., Stehman, S. V., & Woodcock, C. E. (2013) · Remote Sensing of Environment · doi:10.1016/j.rse.2012.10.031 · Cited by 1,029
Jokar Arsanjani, J., Helbich, M., Kainz, W., & Darvishi Boloorani, A. (2013) · International Journal of Applied Earth Observation and Geoinformation · doi:10.1016/j.jag.2011.12.014 · Cited by 752
Metadata
- Authors
- Pontius, R. G., & Millones, M.
- Venue
- International Journal of Remote Sensing
- Publisher
- Informa UK Limited
- Published
- 2011-08-10
- Volume
- 32
- Issue
- 15
- Pages
- 4407-4429
- ISSN
- 0143-1161 (ISSN-L)
- 1366-5901
- OpenAlex
- W2104896032
- Cited by
- 2,046
- References
- 38
- FWCI
- 174.39
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.