CoChat
Sign Up
Cited by 5,172

The DOI matches its Crossref record; 1 of the 1 references it cites were found in a registry or an index.

Assessing the accuracy of species distribution models: prevalence, kappa and the true skill statistic (TSS)

ALLOUCHE, OMRI, TSOAR, ASAF, KADMON, RONEN·2006·Journal of Applied Ecology
DOI 10.1111/j.1365-2664.2006.01214.x
Open this paper in CoChat — free account

Abstract

Summary: In recent years the use of species distribution models by ecologists and conservation managers has increased considerably, along with an awareness of the need to provide accuracy assessment for predictions of such models. The kappa statistic is the most widely used measure for the performance of models generating presence–absence predictions, but several studies have criticized it for being inherently dependent on prevalence, and argued that this dependency introduces statistical artefacts to estimates of predictive accuracy. This criticism has been supported recently by computer simulations showing that kappa responds to the prevalence of the modelled species in a unimodal fashion. In this paper we provide a theoretical explanation for the observed dependence of kappa on prevalence, and introduce into ecology an alternative measure of accuracy, the true skill statistic (TSS), which corrects for this dependence while still keeping all the advantages of kappa. We also compare the responses of kappa and TSS to prevalence using empirical data, by modelling distribution patterns of 128 species of woody plant in Israel. The theoretical analysis shows that kappa responds in a unimodal fashion to variation in prevalence and that the level of prevalence that maximizes kappa depends on the ratio between sensitivity (the proportion of correctly predicted presences) and specificity (the proportion of correctly predicted absences). In contrast, TSS is independent of prevalence. When the two measures of accuracy were compared using empirical data, kappa showed a unimodal response to prevalence, in agreement with the theoretical analysis. TSS showed a decreasing linear response to prevalence, a result we interpret as reflecting true ecological phenomena rather than a statistical artefact. This interpretation is supported by the fact that a similar pattern was found for the area under the ROC curve, a measure known to be independent of prevalence. Synthesis and applications . Our results provide theoretical and empirical evidence that kappa, one of the most widely used measures of model performance in ecology, has serious limitations that make it unsuitable for such applications. The alternative we suggest, TSS, compensates for the shortcomings of kappa while keeping all of its advantages. We therefore recommend the TSS as a simple and intuitive measure for the performance of species distribution models when predictions are expressed as presence–absence maps.

Overview

Integrity signals

Registry

Crossref · 10.1111/j.1365-2664.2006.01214.x

Cites

1 of 1 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Verification

Claims the paper makes

This paper’s full text isn’t openly available, so its claims and self-check weren’t run. The registry and citation checks still apply.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Citations

Cites

  • Bias, prevalence and kappa

    Byrt, T., Bishop, J., & Carlin, J. B. (1993) · Journal of Clinical Epidemiology · doi:10.1016/0895-4356(93)90018-v · Cited by 2,179 · cited 3×

    “the prevalence and bias adjusted kappa, PABAK, proposed by Byrt, Bishop & Carlin 1993; the adjusted kappa coefficient, κ′ described by Hoehler 2000).”
    BackgroundMethodInfluential

Show all 1 references

Research works printed but not linked: 67

  1. 10.1175/WAF854.1 · DOI
  2. 10.1111/j.1365-2486.2004.00828.x · DOI
  3. 10.1111/j.1365-2486.2005.01000.x · DOI
  4. 10.1111/j.1365-2664.2006.01136.x · DOI
  5. 10.1016/j.ecolmodel.2005.01.030 · DOI
  6. 10.1111/j.0906-7590.2004.03553.x · DOI
  7. 10.1111/j.1365-2486.2005.00997.x · DOI
  8. 10.1111/j.0906-7590.2004.03764.x · DOI
  9. 10.1016/0006-3207(95)00102-6 · DOI
  10. 10.1016/0895-4356(90)90159-M · DOI · Paper page
  11. 10.1177/001316446002000104 · DOI · Paper page
  12. 10.1046/j.1365-2699.2000.00408.x · DOI
  13. 10.1046/j.1365-2699.2000.00419.x · DOI
  14. 10.1175/1520-0434(1990)005<0576:OSMOSI>2.0.CO;2 · DOI
  15. 10.1175/1520-0434(2003)018<0953:OECMFS>2.0.CO;2 · DOI
  16. 10.1111/j.0021-8901.2004.00881.x · DOI
  17. 10.1016/S0304-3800(02)00327-7 · DOI
  18. 10.1017/S0376892997000088 · DOI · Paper page
  19. 10.1046/j.1365-2699.2003.00914.x · DOI
  20. 10.1111/j.1461-0248.2005.00792.x · DOI
  21. 10.1111/j.1365-2664.2006.01164.x · DOI
  22. 10.1148/radiology.143.1.7063747 · DOI · Paper page
  23. 10.1177/000456329303000601 · DOI
  24. 10.1016/S0895-4356(99)00174-2 · DOI
  25. 10.1038/28843 · DOI
  26. 10.1111/j.1461-0248.2004.00598.x · DOI
  27. 10.1890/1051-0761(2003)013[0853:ASAOFA]2.0.CO;2 · DOI
  28. 10.1016/0895-4356(95)00571-4 · DOI
  29. 10.1111/j.0906-7590.2005.03957.x · DOI · Paper page
  30. 10.1111/j.1523-1739.2003.00233.x · DOI
  31. 10.1175/1520-0434(2000)015<0103:VOQPFF>2.0.CO;2 · DOI
  32. 10.1111/j.0021-8901.2004.00943.x · DOI
  33. 10.1016/S0304-3800(99)00113-1 · DOI
  34. 10.1046/j.1365-2664.2001.00647.x · DOI · Paper page
  35. 10.1038/35012251 · DOI
  36. 10.1037/1040-3590.7.3.404 · DOI
  37. Nix H.A., 1986, Atlas of Elapid Snakes of Australia, 4
  38. 10.1111/j.0021-8901.2004.00910.x · DOI
  39. 10.1577/1548-8659(2002)131<0329:PMOFSD>2.0.CO;2 · DOI
  40. 10.1111/j.1472-4642.2004.00051.x · DOI
  41. 10.1111/j.0906-7590.2004.03822.x · DOI
  42. 10.1016/S0304-3800(00)00322-7 · DOI
  43. 10.1111/j.0906-7590.2004.03740.x · DOI
  44. Pearson R.G., 2006, Model‐based uncertainty in species’ range prediction, Journal of Biogeography, 10.1111/j.1365-2699.2006.01460.x · DOI
  45. 10.1046/j.1523-1739.2003.02206.x · DOI
  46. 10.1034/j.1600-0587.2003.03545.x · DOI
  47. Reese G.C., 2005, Factors affecting species distribution predictions. A simulation modelling experiment, Ecological Applications, 15, 556, 10.1890/03-5374 · DOI
  48. 10.1111/j.1366-9516.2004.00118.x · DOI
  49. 10.1111/j.0021-8901.2004.00903.x · DOI
  50. 10.17161/bi.v2i0.9 · DOI
  51. 10.1175/1520-0434(2002)017<0832:COWCRF>2.0.CO;2 · DOI
  52. 10.1111/j.1366-9516.2005.00185.x · DOI
  53. 10.1111/j.1365-2699.2004.01076.x · DOI · Paper page
  54. 10.1016/j.ecolmodel.2004.12.012 · DOI
  55. 10.2307/2845837 · DOI
  56. 10.1111/j.0906-7590.2004.03823.x · DOI
  57. 10.1080/136588199241391 · DOI
  58. 10.1016/S0304-3800(01)00388-X · DOI
  59. Thuiller W., 2003, BIOMOD: optimising predictions of species distributions and projecting potential future shifts under global change, Global Change Biology, 9, 1353, 10.1046/j.1365-2486.2003.00666.x · DOI
  60. 10.1111/j.1466-822X.2005.00162.x · DOI
  61. 10.1073/pnas.0409902102 · DOI
  62. 10.1111/j.1365-2486.2005.001018.x · DOI
  63. 10.1016/j.apmr.2003.12.002 · DOI
  64. 10.1016/j.biocon.2004.01.009 · DOI
  65. 10.1111/j.1523-1739.2003.00359.x · DOI
  66. 10.1111/j.1365-2664.2005.01052.x · DOI
  67. Zweig M.H., 1993, Receiver‐operating characteristics (ROC) plots: a fundamental evaluation tool in clinical medicine, Clinical Chemistry, 39, 561, 10.1093/clinchem/39.4.561 · DOI

Cited by

Sorted by citations

Metadata

Authors
ALLOUCHE, O., TSOAR, A., & KADMON, R.
Venue
Journal of Applied Ecology
Publisher
Wiley
Published
2006-09-12
Volume
43
Issue
6
Pages
1223-1232
ISSN
  • 0021-8901
  • 1365-2664
DOI
10.1111/j.1365-2664.2006.01214.x
Cited by
5,172
References
1
Language
en
Licence
onlinelibrary.wiley.com/termsAndConditions#vor (vor)

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review