The DOI matches its Crossref record; 30 of the 30 references it cites were found in a registry or an index.
Answering the Call for a Standard Reliability Measure for Coding Data
Abstract
In content analysis and similar methods, data are typically generated by trained human observers who record or transcribe textual, pictorial, or audible matter in terms suitable for analysis. Conclusions from such data can be trusted only after demonstrating their reliability. Unfortunately, the content analysis literature is full of proposals for so-called reliability coefficients, leaving investigators easily confused, not knowing which to choose. After describing the criteria for a good measure of reliability, we propose Krippendorff's alpha as the standard reliability measure. It is general in that it can be used regardless of the number of observers, levels of measurement, sample sizes, and presence or absence of missing data. To facilitate the adoption of this recommendation, we describe a freely available macro written for SPSS and SAS to calculate Krippendorff's alpha and illustrate its use with a simple example.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“We calculate Krippendorff's alpha (Hayes and Krippendorff, 2007) to measure inter-rater reliability (see Table 3).”
From LLMs cannot find reasoning errors, but can correct them given the error location (2024) · cites this paper as Hayes and Krippendorff, 2007
In the citing paper
“Based on Krippendorff's alpha (α = 0.92), the level of agreement between coders was high and well above the threshold of 0.70 (Hayes & Krippendorff, 2007).”
From Collaboration, Coordination, and Cooperation Among Organizations: Establishing the Distinctive Meanings of These Terms Through a Systematic Literature Review (2020) · cites this paper as Hayes & Krippendorff, 2007
In the citing paper
“we calculated the inter-coder reliability based on 25% of all data units and derived a kappa value of .96, which reflects excellent reliability.”
In the citing paper
“To assess the inter-annotator agreement of stage 1, we report Krippendorf's α (Hayes and Krippendorff, 2007) for each dimension for the 135 articles after training and before any discussion/adjudication: economic (0.44), foreign (0.68), social (0.39).”
From Political Ideology and Polarization: A Multi-dimensional Approach (2022) · cites this paper as Hayes and Krippendorff, 2007
In the citing paper
“This family of metrics does not correct for chance (Cohen, 1960;Hayes and Krippendorff, 2007) and overstates discriminative ability (Guerdan et al., 2025;Han et al., 2025;Collot et al., 2025).”
From Reliability without Validity: A Systematic, Large-Scale Evaluation of LLM-as-a-Judge Models Across Agreement, Consistency, and Bias (2026) · cites this paper as Hayes and Krippendorff, 2007
Citations
Cites
Coefficient Alpha and the Internal Structure of Tests
Cronbach, L. J. (1951) · Psychometrika · doi:10.1007/bf02310555 · Cited by 43,931
A Coefficient of Agreement for Nominal Scales
Cohen, J. (1960) · Educational and Psychological Measurement · doi:10.1177/001316446002000104 · Cited by 42,398
An Introduction to the Bootstrap
Efron, B., & Tibshirani, R. J. (1994) · doi:10.1201/9780429246593 · Cited by 40,500
Nonparametric Statistics for the Behavioral Sciences
Guitton, H., & Siegel, S. (1958) · Revue économique · doi:10.2307/3498751 · Cited by 35,279
Content Analysis: An Introduction to its Methodology.
Shelley, M., & Krippendorff, K. (1984) · Journal of the American Statistical Association · doi:10.2307/2288384 · Cited by 24,346
Self-citationStatistical Methods for Rates and Proportions
Fleiss, J. L., Levin, B., & Paik, M. C. (2003) · Wiley Series in Probability and Statistics · doi:10.1002/0471445428 · Cited by 18,646
Measuring nominal scale agreement among many raters.
Fleiss, J. L. (1971) · Psychological Bulletin · doi:10.1037/h0031619 · Cited by 8,733
Content Analysis for the Social Sciences and Humanities.
Parker, E. B., & Holsti, O. R. (1970) · American Sociological Review · doi:10.2307/2093233 · Cited by 4,553
Content Analysis in Mass Communication: Assessment and Reporting of Intercoder Reliability
Lombard, M., Snyder-Duch, J., & Bracken, C. C. (2002) · Human Communication Research · doi:10.1111/j.1468-2958.2002.tb00826.x · Cited by 3,068
An Introduction to the Bootstrap
Grimshaw, S. D., Efron, B., & Tibshirani, R. J. (1995) · Technometrics · doi:10.2307/1269918 · Cited by 2,281
Research works printed but not linked: 1
- 10.1111/j.1468-2958.2004.tb00738.x · DOI · Paper page
Cited by
Sorted by citations
Intercoder Reliability in Qualitative Research: Debates and Practical Guidelines
O’Connor, C., & Joffe, H. (2020) · International Journal of Qualitative Methods · doi:10.1177/1609406919899220 · Cited by 3,871
“The statistical foundations of these measures are beyond the scope of this article but are fully discussed elsewhere (Banerjee et al., 1999; Davey et al., 2010; Feng, 2013; Hallgren, 2012; Hayes & Krippendorff, 2007; Rust & Cooil, 1994).”
MethodComputing Inter-Rater Reliability for Observational Data: An Overview and Tutorial
Hallgren, K. A. (2012) · Tutorials in Quantitative Methods for Psychology · doi:10.20982/tqmp.08.1.p023 · Cited by 3,884
“ICCs use list-wise deletion for missing data, and therefore cannot accommodate datasets in fully-crossed designs with large amounts of missing data, and Krippendorff’s alpha (Hayes & Krippendorff, 2007) may be more suitable when problems are posed by missing data in fully-crossed designs.”
MethodInter-Coder Agreement for Computational Linguistics
Artstein, R., & Poesio, M. (2008) · Computational Linguistics · doi:10.1162/coli.07-034-r2 · Cited by 1,665
Xie, X., Huo, J., & Hailiang, Z. (2019) · Journal of Business Research · doi:10.1016/j.jbusres.2019.01.010 · Cited by 1,593
Poetz, M. K., & Schreier, M. (2012) · Journal of Product Innovation Management · doi:10.1111/j.1540-5885.2011.00893.x · Cited by 1,241
“Before the experts assessed the final quality of ideas in more detail, they were given training with regard to the evaluation criteria as well as their definition and proper application (Hayes and Krippendorff, 2007; Krippendorff, 2004).”
Metadata
- Authors
- Hayes, A. F., & Krippendorff, K.
- Affiliations
- The Ohio State University (US)
- University of Pennsylvania (US)
- Venue
- Communication Methods and Measures
- Publisher
- Informa UK Limited
- Published
- 2007-04
- Volume
- 1
- Issue
- 1
- Pages
- 77-89
- ISSN
- 1931-2458 (ISSN-L)
- 1931-2466
- OpenAlex
- W2061504941
- Cited by
- 4,225
- References
- 30
- Citations by year
- 2026: 284 · 2025: 278 · 2024: 280 · 2023: 289 · 2022: 331 · 2021: 379 · 2020: 369 · 2019: 296 · 2018: 260 · 2017: 300 · 2016: 248 · 2015: 205 · 2014: 206 · 2013: 150 · 2012: 107
- FWCI
- 50.07
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Communication Methods and Measures (publishedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Reliability and Agreement in Measurement
- Social Sciences › General Social Sciences › Computational and Text Analysis Methods
- Social Sciences › Decision Sciences › Management Science and Operations Research › Psychometric Methodologies and Testing
- Keywords
- Krippendorff's alpha, inter-rater reliability, content analysis, coding reliability, reliability coefficients, observer agreement, SAS macro, missing data
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
