CoChat
Sign Up
Cited by 377Reliability and Agreement in MeasurementPsychometric Methodologies and TestingMultidisciplinary Science and Engineering Research

Measuring inter-rater reliability for nominal data – which coefficients and confidence intervals are appropriate?

Open access
Zapf, Antonia, Castell, Stefanie, Morawietz, Lars, Karch, André·2016·BMC Medical Research Methodology
DOI 10.1186/s12874-016-0200-9PMC PMC4974794
Open this paper in CoChat — free accountOpen PDF

Abstract

BACKGROUND: Reliability of measurements is a prerequisite of medical research. For nominal data, Fleiss' kappa (in the following labelled as Fleiss' K) and Krippendorff's alpha provide the highest flexibility of the available reliability measures with respect to number of raters and categories. Our aim was to investigate which measures and which confidence intervals provide the best statistical properties for the assessment of inter-rater reliability in different situations. METHODS: We performed a large simulation study to investigate the precision of the estimates for Fleiss' K and Krippendorff's alpha and to determine the empirical coverage probability of the corresponding confidence intervals (asymptotic for Fleiss' K and bootstrap for both measures). Furthermore, we compared measures and confidence intervals in a real world case study. RESULTS: Point estimates of Fleiss' K and Krippendorff's alpha did not differ from each other in all scenarios. In the case of missing data (completely at random), Krippendorff's alpha provided stable estimates, while the complete case analysis approach for Fleiss' K led to biased estimates. For shifted null hypotheses, the coverage probability of the asymptotic confidence interval for Fleiss' K was low, while the bootstrap confidence intervals for both measures provided a coverage probability close to the theoretical one. CONCLUSIONS: Fleiss' K and Krippendorff's alpha with bootstrap confidence intervals are equally suitable for the analysis of reliability of complete nominal data. The asymptotic confidence interval for Fleiss' K should not be used. In the case of missing data or data or higher than nominal order, Krippendorff's alpha is recommended. Together with this article, we provide an R-script for calculating Fleiss' K and Krippendorff's alpha and their corresponding bootstrap confidence intervals.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1186/s12874-016-0200-9

Cites

9 of 9 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • The agreement on the eligibility of the included studies between the two authors resulted in a 0.78 kappa value.

    “The agreement on the eligibility of the included studies, performed by the two authors (DB and DR), resulted in a 0.78 kappa value.23”

    From Efficacy and safety of osteopathic manipulative treatment: an overview of systematic reviews (2022) · cites this paper as [23]

    The Methods section explains that Fleiss’ K can measure agreement between two or more raters. The case-study Results concern breast-cancer assessments, not eligibility decisions by DB and DR. A search for the eligibility assessment and 0.78 found no passage reporting the claimed result. (LLM, conf=0.76)

    Experimental — accuracy not yet measured

  • Inter-rater agreement between the two overview authors on the ranking of quality achieved a 0.89 kappa value.

    “Inter-rater agreement between the two overview authors (DB and FB) on the ranking of quality achieved a 0.89 kappa value.23”

    From Efficacy and safety of osteopathic manipulative treatment: an overview of systematic reviews (2022) · cites this paper as [23]

    The Methods discuss kappa as a measure of agreement, and the case-study Results report estimates for clinical measures, not quality rankings by overview authors. Searches for “0.89 kappa” and “ranking of quality” found no passage reporting the claimed agreement between DB and FB. (LLM, conf=0.77)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 26

  1. [2]Michels KB A renaissance for measurement error Int J Epidemiol 2001 30 3 421 2 10.1093/ije/30.3.421 11416054 · DOI
  2. [3]Roger VL Boerwinkle E Crapo JD Strategic transformation of population studies: recommendations of the working group on epidemiology and population sciences from the National Heart, Lung, and Blood Advisory Council and Board of External Experts Am J Epidemiol 2015 181 6 363 8 10.1093/aje/kwv011 25743324 PMC4375403 · DOI
  3. [4]Scott WA Reliability of content analysis: the case of nominal scale coding Public Opinion Quarterly 1955 XIX 321 5 10.1086/266577 · DOI · Paper page
  4. [5]Cohen J A coefficient of agreement for nominal scales Educ Psychol Meas 1960 20 37 46 10.1177/001316446002000104 · DOI · Paper page
  5. [6]Hallgren KA Computing inter-rater reliability for observational data:_an overview and tutorial Tutor Quant Methods Psychol 2012 8 1 23 4 22833776 PMC3402032
  6. [7]Thompson WD Walter SD A reappraisal of the kappa coefficient J Clin Epidemiol 1988 41 10 949 58 10.1016/0895-4356(88)90031-5 3057117 · DOI
  7. [8]Kraemer HC Bloch DA Kappa coefficients in epidemiology: an appraisal of a reappraisal J Clin Epidemiol 1988 41 10 959 68 10.1016/0895-4356(88)90032-7 3193139 · DOI
  8. [11]Siegel S Castellan NJ Jr Nonparametric Statistics for the Behavioral Sciences 1988 2 New York McGraw-Hill
  9. [12]Krippendorff K Estimating the reliability, systematic error, and random error of interval data Educ Psychol Meas 1970 30 61 70 10.1177/001316447003000105 · DOI
  10. [17]Fleiss JL Cohen J Everitt BS Large sample standard errors of kappa and weighted kappa Psychol Bull 1969 72 5 323 7 10.1037/h0028106 · DOI
  11. [18]Fleiss JL Nee JCM Landis JR Large sample variance of kappa in the case of different sets of raters Psychol Bull 1979 86 5 974 7 10.1037/0033-2909.86.5.974 · DOI
  12. [19]Klar N Lipsitz SR Parzen M Leong T An exact bootstrap confidence interval for k in small samples J Royal Stat Soc Series D (The Statistician) 2002 51 4 467 78 10.1111/1467-9884.00331 · DOI
  13. [20]Häußler S. Evaluation of reader heterogeneity in diagnostic trials. Master thesis at the University of Bremen, study course: Medical Biometry / Biostatistics; 2010.
  14. [22]Rao CR Linear Statistical Inference and its Applications 1973 2 New York Wiley
  15. [23]Efron B LePage R Billard L Six questions raised by the bootstrap. Exploring the limits of bootstrap Technical Report No. 139 1992 New York Division of Biostatistics, Stanford University. Wiley
  16. [24]Fleiss JL Davies M Jackknifing functions of multinomial frequencies, with an application to a measure of concordance Am J Epidemiol 1982 115 841 5 7091143
  17. [25]McKenzie DP Mackinnon AJ Péladeau N Onghena P Bruce PC Clarke DM Harrigan S McGorry PD Comparing correlated kappas by resampling: is one level of agreement significantly different from another? J Psychiatr Res 1996 30 6 483 92 10.1016/S0022-3956(96)00033-7 9023792 · DOI
  18. [26]Vanbelle S Albert A A bootstrap method for comparing correlated kappa coefficients J Stat Comput Simulation 2008 78 11 1009 15 10.1080/00949650701410249 · DOI
  19. [27]Krippendorff. Content analysis, an introduction to its methodology. SAGE Publications, 2nd Edition; 2004.
  20. [32]MacPherson P Choko AT Webb EL Thindwa D Squire SB Sambakunsi R van Oosterhout JJ Chunda T Chavula K Makombe SD Lalloo DG Corbett EL Development and validation of a global positioning system-based “map book” system for categorizing cluster residency status of community members living in high-density urban slums in Blantyre Malawi Am J Epidemiol 2013 177 10 1143 7 10.1093/aje/kws376 23589586 PMC3649638 · DOI
  21. [33]Devine A Taylor SJ Spencer A Diaz-Ordaz K Eldrige S Underwood M The agreement between proxy and self-completed EQ-5D for care home residents was better for index scores than individual domains J Clin Epidemiol 2014 67 9 1035 43 10.1016/j.jclinepi.2014.04.005 24837298 PMC4126106 · DOI
  22. [34]Johnson SR Tomlinson GA Hawker GA Granton JT Grosbein HA Feldmann BM A valid and reliable belief elicitation method for Bayesian priors J Clin Epidemiol 2010 63 4 370 83 10.1016/j.jclinepi.2009.08.005 19926253 · DOI
  23. [35]Hoy D Brooks P Woolf A Blyth F March L Bain C Baker P Smith E Buchbinder R Assessing risk of bias in prevalence studies: modification of an existing tool and evidence of interrater agreement J Clin Epidemiol 2012 65 9 934 9 10.1016/j.jclinepi.2011.11.014 22742910 · DOI · Paper page
  24. [36]Krippendorff K Salmond CT Commentars: A dissenting view on so-called paradoxes of reliability coefficients Communication Yearbook 36 2012 New York Routledge
  25. [37]Schuster C A mixture model approach to indexing rater agreement Br J Math Stat Psychol 2002 55 2 289 303 10.1348/000711002760554598 12473229 · DOI
  26. [38]Schuster C Smith DA Indexing systematic rater agreement with a latent-class model Psychol Methods 2002 7 3 384 95 10.1037/1082-989X.7.3.384 12243308 · DOI

Cited by

Sorted by citations

Authors
Zapf, A., Castell, S., Morawietz, L., & Karch, A.
Affiliations
  • Universitätsmedizin Göttingen (DE)
  • Helmholtz Centre for Infection Research (DE)
  • Klinikum Ernst von Bergmann (DE)
  • German Center for Infection Research (DE)
Corresponding author
Zapf, Antonia
ORCID
Venue
BMC Medical Research Methodology
Publisher
Springer Science and Business Media LLC
Published
2016-08-05
Volume
16
Issue
1
Pages
93-93
ISSN
1471-2288 (ISSN-L)
DOI
10.1186/s12874-016-0200-9
PMC
PMC4974794
OpenAlex
W2477907264
Cited by
377
References
9
Citations by year
2026: 31 · 2025: 43 · 2024: 63 · 2023: 54 · 2022: 62 · 2021: 41 · 2020: 41 · 2019: 25 · 2018: 13 · 2017: 4
FWCI
13.08
Type
journal article
Language
en
Open access
Licence
Locations
  • BMC Medical Research Methodology (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • HZI (Helmholtz Centre for Infection Research) (submittedVersion, cc-by-nc-sa)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
  • GoeScholar The Publication Server of the Georg-August-Universität Göttingen (Georg-August-Universität Göttingen) (submittedVersion, cc-by)
  • GRO (submittedVersion, cc-by)
Funders
Universiteit Maastricht
Topics
  • Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Reliability and Agreement in Measurement
  • Social Sciences › Decision Sciences › Management Science and Operations Research › Psychometric Methodologies and Testing
  • Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Multidisciplinary Science and Engineering Research
Keywords
inter-rater reliability, Fleiss' kappa, Krippendorff's alpha, nominal data, confidence intervals, bootstrap confidence intervals, asymptotic confidence intervals, coverage probability, missing data, measurement reliability
MeSH terms
Algorithms, Breast Neoplasms, Data Interpretation, Statistical, Female, Humans, Retrospective Studies, Reproducibility of Results, Observer Variation, Confidence Intervals

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review