The DOI matches its Crossref record; 30 of the 30 references it cites were found in a registry or an index.
Answering the Call for a Standard Reliability Measure for Coding Data
Abstract
In content analysis and similar methods, data are typically generated by trained human observers who record or transcribe textual, pictorial, or audible matter in terms suitable for analysis. Conclusions from such data can be trusted only after demonstrating their reliability. Unfortunately, the content analysis literature is full of proposals for so-called reliability coefficients, leaving investigators easily confused, not knowing which to choose. After describing the criteria for a good measure of reliability, we propose Krippendorff's alpha as the standard reliability measure. It is general in that it can be used regardless of the number of observers, levels of measurement, sample sizes, and presence or absence of missing data. To facilitate the adoption of this recommendation, we describe a freely available macro written for SPSS and SAS to calculate Krippendorff's alpha and illustrate its use with a simple example.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“We calculate Krippendorff's alpha (Hayes and Krippendorff, 2007) to measure inter-rater reliability (see Table 3).”
From LLMs cannot find reasoning errors, but can correct them given the error location (2024) · cites this paper as Hayes and Krippendorff, 2007
In the citing paper
“Based on Krippendorff's alpha (α = 0.92), the level of agreement between coders was high and well above the threshold of 0.70 (Hayes & Krippendorff, 2007).”
From Collaboration, Coordination, and Cooperation Among Organizations: Establishing the Distinctive Meanings of These Terms Through a Systematic Literature Review (2020) · cites this paper as Hayes & Krippendorff, 2007
In the citing paper
“we calculated the inter-coder reliability based on 25% of all data units and derived a kappa value of .96, which reflects excellent reliability.”
In the citing paper
“To assess the inter-annotator agreement of stage 1, we report Krippendorf's α (Hayes and Krippendorff, 2007) for each dimension for the 135 articles after training and before any discussion/adjudication: economic (0.44), foreign (0.68), social (0.39).”
From Political Ideology and Polarization: A Multi-dimensional Approach (2022) · cites this paper as Hayes and Krippendorff, 2007
In the citing paper
“This family of metrics does not correct for chance (Cohen, 1960;Hayes and Krippendorff, 2007) and overstates discriminative ability (Guerdan et al., 2025;Han et al., 2025;Collot et al., 2025).”
From Reliability without Validity: A Systematic, Large-Scale Evaluation of LLM-as-a-Judge Models Across Agreement, Consistency, and Bias (2026) · cites this paper as Hayes and Krippendorff, 2007
Citations
Cites
Coefficient Alpha and the Internal Structure of Tests
Cronbach, L. J. (1951) · Psychometrika · doi:10.1007/bf02310555 · Cited by 43,931
A Coefficient of Agreement for Nominal Scales
Cohen, J. (1960) · Educational and Psychological Measurement · doi:10.1177/001316446002000104 · Cited by 42,398
An Introduction to the Bootstrap
Efron, B., & Tibshirani, R. J. (1994) · doi:10.1201/9780429246593 · Cited by 40,500
Nonparametric Statistics for the Behavioral Sciences
Guitton, H., & Siegel, S. (1958) · Revue économique · doi:10.2307/3498751 · Cited by 35,279
Content Analysis: An Introduction to its Methodology.
Shelley, M., & Krippendorff, K. (1984) · Journal of the American Statistical Association · doi:10.2307/2288384 · Cited by 24,351
Self-citationStatistical Methods for Rates and Proportions
Fleiss, J. L., Levin, B., & Paik, M. C. (2003) · Wiley Series in Probability and Statistics · doi:10.1002/0471445428 · Cited by 18,646
Measuring nominal scale agreement among many raters.
Fleiss, J. L. (1971) · Psychological Bulletin · doi:10.1037/h0031619 · Cited by 8,733
Content Analysis for the Social Sciences and Humanities.
Parker, E. B., & Holsti, O. R. (1970) · American Sociological Review · doi:10.2307/2093233 · Cited by 4,553
Content Analysis in Mass Communication: Assessment and Reporting of Intercoder Reliability
Lombard, M., Snyder-Duch, J., & Bracken, C. C. (2002) · Human Communication Research · doi:10.1111/j.1468-2958.2002.tb00826.x · Cited by 3,068
An Introduction to the Bootstrap
Grimshaw, S. D., Efron, B., & Tibshirani, R. J. (1995) · Technometrics · doi:10.2307/1269918 · Cited by 2,281
Research works printed but not linked: 1
- 10.1111/j.1468-2958.2004.tb00738.x · DOI · Paper page
Cited by
Sorted by citations
Items 1,001–1,013 of 1,013
A Simple Guide to Inter-rater, Intra-rater and Test-retest Reliability for Animal Behaviour Studies
Harvey, N. D. (2021) · doi:10.31219/osf.io/8stpy · Cited by 31
LLMs cannot find reasoning errors, but can correct them given the error location
Tyen, G., Mansoor, H., Carbune, V., Chen, P., & Mak, T. (2024) · Findings of the Association for Computational Linguistics ACL 2024 · doi:10.18653/v1/2024.findings-acl.826 · Cited by 156
Derwael, C., Lavergne, O., Lovinfosse, P., Nechifor, V., Salvé, M., Waltregny, D., Hustinx, R., & Withofs, N. (2020) · EJNMMI Research · doi:10.1186/s13550-020-0596-4 · Cited by 23
Radically open dialectical behaviour therapy for refractory depression: the RefraMED RCT
Lynch, T. R., Hempel, R. J., Whalley, B., Byford, S., Chamba, R., Clarke, P., Clarke, S., Kingdon, D., O’Mahen, H., Remington, B., et al. (2018) · Efficacy and Mechanism Evaluation · doi:10.3310/eme05070 · Cited by 22
Keng, S.-L., Mohd Salleh Sahimi, H. B., Chan, L. F., Woon, L., Eu, C. L., Sim, S. H., & Wong, M. K. (2021) · BMC Psychiatry · doi:10.1186/s12888-021-03500-y · Cited by 20
SPARQL Query Verbalization for Explaining Semantic Search Engine Queries
Ell, B., Harth, A., & Simperl, E. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-07443-6_29 · Cited by 18
“We used Krippendorff’s alpha [8] to measure inter-rater agreement regarding whether our results are comparable or.”
MethodMerke, S., Ganushchak, L., & van Steensel, R. (2024) · Educational Research Review · doi:10.1016/j.edurev.2023.100572 · Cited by 15
Political Ideology and Polarization: A Multi-dimensional Approach
Sinno, B., Oviedo, B., Atwell, K., Alikhani, M., & Li, J. J. (2022) · Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2022.naacl-main.17 · Cited by 12
Devine, R. T., Kovatchev, V., Grumley Traynor, I., Smith, P., & Lee, M. (2023) · Psychological Assessment · doi:10.1037/pas0001186 · Cited by 12
Kohmäscher, A., Primaßin, A., Heiler, S., Avelar, P. D. C., Franken, M.-C., & Heim, S. (2023) · Journal of Speech, Language, and Hearing Research · doi:10.1044/2023_jslhr-23-00224 · Cited by 9
Pribuišis, K., Maskeliūnas, R., Ulozaitė-Stanienė, N., Padervinskis, E., Damaševičius, R., Blažauskas, T., & Uloza, V. (2025) · Journal of Voice · doi:10.1016/j.jvoice.2025.04.026 · Cited by 3
Exposing the Hidden Layer: Software Repositories in the Service of Seo Manipulation
Wu, M., Hong, G., Mai, W., Wu, X., Zhang, L., Pu, Y., Chai, H., Ying, L., Duan, H., & Yang, M. (2025) · 2025 IEEE/ACM 47th International Conference on Software Engineering (ICSE) · doi:10.1109/icse55347.2025.00147 · Cited by 3
Norman, J. D., Rivera, M. U., & Hughes, D. A. (2026) · arXiv · doi:10.48550/arxiv.2606.19544 · Cited by 1
“This family of metrics does not correct for chance (Cohen, 1960; Hayes and Krippendorff, 2007) and overstates discriminative ability (Guerdan et al., 2025; Han et al., 2025; Collot et al., 2025).”
Metadata
- Authors
- Hayes, A. F., & Krippendorff, K.
- Affiliations
- The Ohio State University (US)
- University of Pennsylvania (US)
- Venue
- Communication Methods and Measures
- Publisher
- Informa UK Limited
- Published
- 2007-04
- Volume
- 1
- Issue
- 1
- Pages
- 77-89
- ISSN
- 1931-2458 (ISSN-L)
- 1931-2466
- OpenAlex
- W2061504941
- Cited by
- 4,225
- References
- 30
- Citations by year
- 2026: 284 · 2025: 278 · 2024: 280 · 2023: 289 · 2022: 331 · 2021: 379 · 2020: 369 · 2019: 296 · 2018: 260 · 2017: 300 · 2016: 248 · 2015: 205 · 2014: 206 · 2013: 150 · 2012: 107
- FWCI
- 50.07
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Communication Methods and Measures (publishedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Reliability and Agreement in Measurement
- Social Sciences › General Social Sciences › Computational and Text Analysis Methods
- Social Sciences › Decision Sciences › Management Science and Operations Research › Psychometric Methodologies and Testing
- Keywords
- Krippendorff's alpha, inter-rater reliability, content analysis, coding reliability, reliability coefficients, observer agreement, SAS macro, missing data
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
