Answering the Call for a Standard Reliability Measure for Coding Data
Abstract
In content analysis and similar methods, data are typically generated by trained human observers who record or transcribe textual, pictorial, or audible matter in terms suitable for analysis. Conclusions from such data can be trusted only after demonstrating their reliability. Unfortunately, the content analysis literature is full of proposals for so-called reliability coefficients, leaving investigators easily confused, not knowing which to choose. After describing the criteria for a good measure of reliability, we propose Krippendorff's alpha as the standard reliability measure. It is general in that it can be used regardless of the number of observers, levels of measurement, sample sizes, and presence or absence of missing data. To facilitate the adoption of this recommendation, we describe a freely available macro written for SPSS and SAS to calculate Krippendorff's alpha and illustrate its use with a simple example.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Krippendorff's alpha is calculated to measure inter-rater reliability.
“We calculate Krippendorff's alpha (Hayes and Krippendorff, 2007) to measure inter-rater reliability (see Table 3).”
From LLMs cannot find reasoning errors, but can correct them given the error location (2024) · cites this paper as Hayes and Krippendorff, 2007
The level of agreement between coders was high (Krippendorff's alpha = 0.92) and well above the threshold of 0.70.
“Based on Krippendorff's alpha (α = 0.92), the level of agreement between coders was high and well above the threshold of 0.70 (Hayes & Krippendorff, 2007).”
From Collaboration, Coordination, and Cooperation Among Organizations: Establishing the Distinctive Meanings of These Terms Through a Systematic Literature Review (2020) · cites this paper as Hayes & Krippendorff, 2007
Inter-coder reliability based on 25% of all data units yielded a kappa value of .96, reflecting excellent reliability.
“we calculated the inter-coder reliability based on 25% of all data units and derived a kappa value of .96, which reflects excellent reliability.”
Inter-annotator agreement (Krippendorff's α) for 135 articles after training and before adjudication was economic 0.44, foreign 0.68, social 0.39.
“To assess the inter-annotator agreement of stage 1, we report Krippendorf's α (Hayes and Krippendorff, 2007) for each dimension for the 135 articles after training and before any discussion/adjudication: economic (0.44), foreign (0.68), social (0.39).”
From Political Ideology and Polarization: A Multi-dimensional Approach (2022) · cites this paper as Hayes and Krippendorff, 2007
Exact match does not correct for agreement expected by chance and systematically overstates discriminative ability.
“This family of metrics does not correct for chance (Cohen, 1960;Hayes and Krippendorff, 2007) and overstates discriminative ability (Guerdan et al., 2025;Han et al., 2025;Collot et al., 2025).”
From Reliability without Validity: A Systematic, Large-Scale Evaluation of LLM-as-a-Judge Models Across Agreement, Consistency, and Bias (2026) · cites this paper as Hayes and Krippendorff, 2007
Cites
Coefficient Alpha and the Internal Structure of Tests
Cronbach, L. J. (1951) · Psychometrika · doi:10.1007/bf02310555 · Cited by 43,931 · see citing works
A Coefficient of Agreement for Nominal Scales
Cohen, J. (1960) · Educational and Psychological Measurement · doi:10.1177/001316446002000104 · Cited by 42,398 · see citing works
An Introduction to the Bootstrap
Efron, B., & Tibshirani, R. J. (1994) · doi:10.1201/9780429246593 · Cited by 40,500 · see citing works
Nonparametric Statistics for the Behavioral Sciences
Guitton, H., & Siegel, S. (1958) · Revue économique · doi:10.2307/3498751 · Cited by 35,281 · see citing works
Content Analysis: An Introduction to its Methodology.
Shelley, M., & Krippendorff, K. (1984) · Journal of the American Statistical Association · doi:10.2307/2288384 · Cited by 24,305 · see citing works
Self-citationStatistical Methods for Rates and Proportions
Fleiss, J. L., Levin, B., & Paik, M. C. (2003) · Wiley Series in Probability and Statistics · doi:10.1002/0471445428 · Cited by 18,646 · see citing works
Measuring nominal scale agreement among many raters.
Fleiss, J. L. (1971) · Psychological Bulletin · doi:10.1037/h0031619 · Cited by 8,733 · see citing works
Content Analysis for the Social Sciences and Humanities.
Parker, E. B., & Holsti, O. R. (1970) · American Sociological Review · doi:10.2307/2093233 · Cited by 4,582 · see citing works
Content Analysis in Mass Communication: Assessment and Reporting of Intercoder Reliability
Lombard, M., Snyder-Duch, J., & Bracken, C. C. (2002) · Human Communication Research · doi:10.1111/j.1468-2958.2002.tb00826.x · Cited by 3,068 · see citing works
An Introduction to the Bootstrap
Grimshaw, S. D., Efron, B., & Tibshirani, R. J. (1995) · Technometrics · doi:10.2307/1269918 · Cited by 2,281 · see citing works
Reliability of Content Analysis: The Case of Nominal Scale Coding
Scott, W. A. (1955) · Public Opinion Quarterly · doi:10.1086/266577 · Cited by 1,853 · see citing works
Coefficient Kappa: Some Uses, Misuses, and Alternatives
Brennan, R. L., & Prediger, D. J. (1981) · Educational and Psychological Measurement · doi:10.1177/001316448104100307 · Cited by 1,590 · see citing works
Reliability of Nominal Data Based on Qualitative Judgments
Perreault, W. D., & Leigh, L. E. (1989) · Journal of Marketing Research · doi:10.2307/3172601 · Cited by 1,575 · see citing works
Nonparametric Statistics for the Behavioral Sciences
(2010) · Encyclopedia of Research Design · doi:10.4135/9781412961288.n273 · Cited by 1,414 · see citing works
Bootstrapping: A Nonparametric Approach to Statistical Inference.
Aa, Mooney, C. Z., & Duval, R. D. (1994) · Journal of the American Statistical Association · doi:10.2307/2290969 · Cited by 1,368 · see citing works
Reliability in content analysis: some common misconceptions and recommendations.
Krippendorff, K. (2004) · Human Communication Research · doi:10.1093/hcr/30.3.411 · Cited by 1,027 · see citing works
Self-citationMooney, C., & Duval, R. (1993) · doi:10.4135/9781412983532 · Cited by 893 · see citing works
Estimating the Reliability, Systematic Error and Random Error of Interval Data
Krippendorff, K. (1970) · Educational and Psychological Measurement · doi:10.1177/001316447003000105 · Cited by 511 · see citing works
Self-citationSmith, M. B., & Pool, I. D. S. (1960) · The American Journal of Psychology · doi:10.2307/1419974 · Cited by 374 · see citing works
Statistical Methods for Communication Science
Hayes, A. F. (2014) · Routledge eBooks · doi:10.4324/9781410613707 · Cited by 276 · see citing works
Self-citationA Note on the G Index of Agreement
Holley, J. W., & Guilford, J. P. (1964) · Educational and Psychological Measurement · doi:10.1177/001316446402400402 · Cited by 266 · see citing works
Communications Through Limited Response Questioning
Bennett, E., Alpert, R., & Goldstein, A. C. (1954) · Public Opinion Quarterly · doi:10.1086/266520 · Cited by 254 · see citing works
Comparing the Classification of Subjects by Two Independent Judges
Maxwell, A. E. (1970) · The British Journal of Psychiatry · doi:10.1192/bjp.116.535.651 · Cited by 249 · see citing works
Another look at interrater agreement.
Zwick, R. (1988) · Psychological Bulletin · doi:10.1037/0033-2909.103.3.374 · Cited by 229 · see citing works
Statistical Methods For Communication Science
Ries, L. D. (2006) · The American Statistician · doi:10.1198/tas.2006.s55 · Cited by 205 · see citing works
On Agreement Indices for Nominal Data
Popping, R. (1988) · Palgrave Macmillan UK eBooks · doi:10.1007/978-1-349-19051-5_6 · Cited by 89 · see citing works
On Generalizations Of The G Index And The Phi Coefficient To Nominal Scales
Janson, S., & Vegelius, J. (1979) · Multivariate Behavioral Research · doi:10.1207/s15327906mbr1402_9 · Cited by 68 · see citing works
A Call for Standardization in Content Analysis Reliability
Lombard, M. (2004) · Human Communication Research · doi:10.1111/j.1468-2958.2004.tb00739.x · Cited by 67 · see citing works
Pearson, K., Lee, A., Warren, E., Fry, A., & Fawcett, C. D. (1901) · Philosophical Transactions of the Royal Society of London Series A Containing Papers of a Mathematical or Physical Character · doi:10.1098/rsta.1901.0020 · Cited by 55 · see citing works
A SAS MACRO for Calculating Intercoder Agreement in Content Analysis
Kang, N., Kara, A., Laskey, H. A., & Seaton, F. B. (1993) · Journal of Advertising · doi:10.1080/00913367.1993.10673400 · Cited by 34 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Intercoder Reliability in Qualitative Research: Debates and Practical Guidelines
O’Connor, C., & Joffe, H. (2020) · International Journal of Qualitative Methods · doi:10.1177/1609406919899220 · Cited by 3,871 · see citing works
“The statistical foundations of these measures are beyond the scope of this article but are fully discussed elsewhere (Banerjee et al., 1999; Davey et al., 2010; Feng, 2013; Hallgren, 2012; Hayes & Krippendorff, 2007; Rust & Cooil, 1994).”
MethodComputing Inter-Rater Reliability for Observational Data: An Overview and Tutorial
Hallgren, K. A. (2012) · Tutorials in Quantitative Methods for Psychology · doi:10.20982/tqmp.08.1.p023 · Cited by 3,884 · see citing works
“ICCs use list-wise deletion for missing data, and therefore cannot accommodate datasets in fully-crossed designs with large amounts of missing data, and Krippendorff’s alpha (Hayes & Krippendorff, 2007) may be more suitable when problems are posed by missing data in fully-crossed designs.”
MethodInter-Coder Agreement for Computational Linguistics
Artstein, R., & Poesio, M. (2008) · Computational Linguistics · doi:10.1162/coli.07-034-r2 · Cited by 1,665 · see citing works
Xie, X., Huo, J., & Hailiang, Z. (2019) · Journal of Business Research · doi:10.1016/j.jbusres.2019.01.010 · Cited by 1,499 · see citing works
Poetz, M. K., & Schreier, M. (2012) · Journal of Product Innovation Management · doi:10.1111/j.1540-5885.2011.00893.x · Cited by 1,241 · see citing works
“Before the experts assessed the final quality of ideas in more detail, they were given training with regard to the evaluation criteria as well as their definition and proper application (Hayes and Krippendorff, 2007; Krippendorff, 2004).”
On the shoulders of giants: undertaking a structured literature review in accounting
Massaro, M., Dumay, J., & Guthrie, J. (2016) · Accounting Auditing & Accountability Journal · doi:10.1108/aaaj-01-2015-1939 · Cited by 984 · see citing works
Dombrowski, S. U., Knittle, K., Avenell, A., Araujo-Soares, V., & Sniehotta, F. F. (2014) · BMJ · doi:10.1136/bmj.g2646 · Cited by 816 · see citing works
Chaumon, M., Bishop, D., & Busch, N. A. (2015) · Journal of Neuroscience Methods · doi:10.1016/j.jneumeth.2015.02.025 · Cited by 833 · see citing works
Computing Krippendorff's Alpha-Reliability
Krippendorff, K. (2011) · ScholarlyCommons (University of Pennsylvania) · Cited by 809 · see citing works
Self-citationIntegrated reporting: A structured literature review
Dumay, J., Bernardi, C., Guthrie, J., & Demartini, P. (2016) · Accounting Forum · doi:10.1016/j.accfor.2016.06.001 · Cited by 783 · see citing works
A meta-analysis of web-delivered tailored health behavior change interventions
Lustria, M. L. A., Noar, S. M., Cortese, J., Van Stee, S. K., Glueckauf, R. L., & Lee, J. (2013) · Journal of Health Communication · doi:10.1080/10810730.2013.768727 · Cited by 689 · see citing works
Heritability of personality: A meta-analysis of behavior genetic studies.
Vukasović, T., & Bratko, D. (2015) · Psychological Bulletin · doi:10.1037/bul0000017 · Cited by 716 · see citing works
Big data analytics in E-commerce: a systematic review and agenda for future research
Akter, S., & Wamba, S. F. (2016) · Electronic Markets · doi:10.1007/s12525-016-0219-0 · Cited by 684 · see citing works
“De Swert (2012) to judge the inter-rater reliability of the coded variables (Hayes and Krippendorff, 2007).”
BackgroundMethodMarketing in the Metaverse: Conceptual understanding, framework, and research agenda
Barrera, K. G., & Shah, D. (2022) · Journal of Business Research · doi:10.1016/j.jbusres.2022.113420 · Cited by 712 · see citing works
Future work selves: How salient hoped-for identities motivate proactive career behaviors.
Strauss, K., Griffin, M. A., & Parker, S. K. (2011) · Journal of Applied Psychology · doi:10.1037/a0026423 · Cited by 686 · see citing works
“As proposed by Hayes and Krippendorff (2007), we used Krippendorff’s alpha as a measure of interrater reliability (Krippendorff, 1970, 2004).”
BackgroundMethodEx Machina: Personal Attacks Seen at Scale
Wulczyn, E., Thain, N., & Dixon, L. (2017) · Proceedings of the 26th International Conference on World Wide Web · doi:10.1145/3038912.3052591 · Cited by 636 · see citing works
Effectiveness of positive psychology interventions: a systematic review and meta-analysis
Carr, A., Cullen, K., Keeney, C., Canning, C., Mooney, O., Chinseallaigh, E., & O’Dowd, A. (2021) · The Journal of Positive Psychology · doi:10.1080/17439760.2020.1818807 · Cited by 645 · see citing works
“There was a 95% interrater agreement for screening full texts with a Krippendorff’s alpha (K-alpha, Hayes & Krippendorff, 2007) of 0.88.”
MethodInfluentialTrust and team performance: A meta-analysis of main effects, moderators, and covariates.
De Jong, B. A., Dirks, K. T., & Gillespie, N. (2016) · Journal of Applied Psychology · doi:10.1037/apl0000110 · Cited by 616 · see citing works
A literature review and perspectives in reverse logistics
Agrawal, S., Singh, R. K., & Murtaza, Q. (2015) · Resources Conservation and Recycling · doi:10.1016/j.resconrec.2015.02.009 · Cited by 624 · see citing works
Jacobson, N. C., & Newman, M. G. (2017) · Psychological Bulletin · doi:10.1037/bul0000111 · Cited by 576 · see citing works
Do people “personally experience” global warming, and if so how, and does it matter?
Akerlof, K., Maibach, E. W., Fitzgerald, D., Cedeno, A. Y., & Neuman, A. (2012) · Global Environmental Change · doi:10.1016/j.gloenvcha.2012.07.006 · Cited by 589 · see citing works
“A Krippendorff’s a inter-reliability statistic (Hayes and Krippendorff, 2007) of .800 or above was obtained for 27 variables, and above .700 for one variable (precipitation specifics).”
MethodRhee, C., Jones, T. M., Hamad, Y., Pande, A., Varon, J., O’Brien, C. L., Anderson, D. J., Warren, D. K., Dantes, R., Epstein, L. H., et al. (2019) · JAMA Network Open · doi:10.1001/jamanetworkopen.2018.7571 · Cited by 580 · see citing works
Using the flipped classroom to enhance EFL learning
Hsieh, J. C., Wu, W. V., & Marek, M. W. (2016) · Computer Assisted Language Learning · doi:10.1080/09588221.2015.1111910 · Cited by 578 · see citing works
“Inter-rater reliability was measured with Krippendorff’s alpha at .82, which is above the level considered the norm for good reliability (Hayes & Krippendorff, 2007).”
MethodA Systematic Review of Propensity Score Methods in the Social Sciences
Thoemmes, F. J., & Kim, E. S. (2011) · Multivariate Behavioral Research · doi:10.1080/00273171.2011.540475 · Cited by 577 · see citing works
Amendola, L. M., Jarvik, G. P., Leo, M. C., McLaughlin, H. M., Akkari, Y. M. N., Amaral, M. D., Berg, J. S., Biswas, S., Bowling, K. M., Conlin, L. K., et al. (2016) · The American Journal of Human Genetics · doi:10.1016/j.ajhg.2016.03.024 · Cited by 556 · see citing works
Process guidelines for establishing Intercoder Reliability in qualitative studies
MacPhail, C., Khoza, N., Abler, L., & Ranganathan, M. (2016) · Qualitative Research · doi:10.1177/1468794115577012 · Cited by 534 · see citing works
“Krippendorf’s alpha is better able to correct for chance and is equipped to handle multiple coders (Hayes and Krippendorff, 2007), but is computationally and conceptually difficult.”
BackgroundCastañer, X., & Oliveira, N. (2020) · Journal of Management · doi:10.1177/0149206320901565 · Cited by 525 · see citing works
“Based on Krippendorff’s alpha (α = 0.92), the level of agreement between coders was high and well above the threshold of 0.70 (Hayes & Krippendorff, 2007).”
InfluentialGrana, R. A., & Ling, P. M. (2014) · American Journal of Preventive Medicine · doi:10.1016/j.amepre.2013.12.010 · Cited by 514 · see citing works
Weigel, F. K., & Hazen, B. T. (2013) · Advances in healthcare information systems and administration book series · doi:10.4018/978-1-4666-4619-3.ch008 · Cited by 497 · see citing works
Xia, L., Shin, H., & Burns, A. C. (2019) · Journal of Business Research · doi:10.1016/j.jbusres.2019.04.042 · Cited by 539 · see citing works
“Krippendorff's alpha (Hayes & Krippendorff, 2007) for entertainment is 0.82 (a value above 0.70 indicates adequate reliability).”
BackgroundInfluentialHowlett, N., Trivedi, D., Troop, N. A., & Chater, A. M. (2019) · Translational Behavioral Medicine · doi:10.1093/tbm/iby010 · Cited by 448 · see citing works
“Inter-rater reliability throughout this review was assessed using Krippendorf’s α, a reliability coefficient that compares favourably to alternatives [39].”
MethodIssues and Best Practices in Content Analysis
Lacy, S. R., Watson, B. R., Riffe, D., & Lovejoy, J. (2015) · Journalism & Mass Communication Quarterly · doi:10.1177/1077699015607338 · Cited by 478 · see citing works
“Among the three coefficients commonly used for content analysis, Krippendorff proposes that Alpha should become the accepted coefficient (Hayes & Krippendorff, 2007; Krippendorff, 2004b).”
BackgroundMethodReframing climate change as a public health issue: an exploratory study of public reactions
Maibach, E. W., Nisbet, M., Baldwin, P., Akerlof, K., & Diao, G. (2010) · BMC Public Health · doi:10.1186/1471-2458-10-299 · Cited by 454 · see citing works
“To assess reliability, we used Krippendorff 's alpha [24,25], a conservative measure that corrects for chance agreement among coders; a K-alpha of .”
MethodAgreement and Information in the Reliability of Coding
Krippendorff, K. (2011) · Communication Methods and Measures · doi:10.1080/19312458.2011.568376 · Cited by 451 · see citing works
“…sizes and distributions of values in the reliability data to which the information measures respond, also affect the confidence intervals of the calculated reliability coefficients (for alpha, see Krippendorff, 2004, pp. 237-238 and Hayes & Krippendorff, 2007), providing access to Type I errors.”
Self-citationMoretti, F., van Vliet, L., Bensing, J. M., Deledda, G., Mazzi, M. A., Rimondini, M., Zimmermann, C., & Fletcher, I. J. (2011) · Patient Education and Counseling · doi:10.1016/j.pec.2011.01.005 · Cited by 453 · see citing works
“It ranges from 0 (complete disagreement) and 1 (complete agreement) and its use is recommended when data regard complex categories derived from content analysis [14,15].”
BackgroundReport of Findings in a DSM-5 Field Trial for Hypersexual Disorder
Reid, R. C., Carpenter, B. N., Hook, J. N., Garos, S., Manning, J. C., Gilliland, R., Cooper, E. B., McKittrick, H., Davtian, M., & Fong, T. (2012) · The Journal of Sexual Medicine · doi:10.1111/j.1743-6109.2012.02936.x · Cited by 430 · see citing works
Algorithmic bias in data-driven innovation in the age of AI
Akter, S., McCarthy, G., Sajib, S., Michael, K., Dwivedi, Y. K., D’Ambra, J., & Shen, K. N. (2021) · International Journal of Information Management · doi:10.1016/j.ijinfomgt.2021.102387 · Cited by 440 · see citing works
Corporate Governance Meets Corporate Social Responsibility: Mapping the Interface
Zaman, R., Jain, T., Samara, G., & Jamali, D. (2020) · Business & Society · doi:10.1177/0007650320973415 · Cited by 439 · see citing works
“We applied the K-alpha test to check the coding reliability (Hayes & Krippendorff, 2007; Krippendorff, 2012).”
MethodWhen Vaccines Go Viral: An Analysis of HPV Vaccine Coverage on YouTube
Briones, R., Nan, X., Madden, K., & Waks, L. (2012) · Health Communication · doi:10.1080/10410236.2011.610258 · Cited by 381 · see citing works
“72, an acceptable degree of reliability ( Hayes & Krippendorff, 2007 ).”
BackgroundOn Using Twitter to Monitor Political Sentiment and Predict Election Results
Bermingham, A., & Smeaton, A. F. (2011) · Arrow@dit (Dublin Institute of Technology) · Cited by 392 · see citing works
Public sector knowledge management: a structured literature review
Massaro, M., Dumay, J., & Garlatti, A. (2015) · Journal of Knowledge Management · doi:10.1108/jkm-11-2014-0466 · Cited by 402 · see citing works
Universal Principles in the Repair of Communication Problems
Dingemanse, M., Roberts, S. G., Baranova, J., Blythe, J., Drew, P., Floyd, S., Gísladóttir, R. S., Kendrick, K. H., Levinson, S. C., Manrique, E., et al. (2015) · PLoS ONE · doi:10.1371/journal.pone.0136100 · Cited by 383 · see citing works
Montoya, R. M., Horton, R. S., Vevea, J. L., Citkowicz, M., & Lauber, E. A. (2017) · Psychological Bulletin · doi:10.1037/bul0000085 · Cited by 389 · see citing works
Advanced, Analytic, Automated (AAA) Measurement of Engagement During Learning
D’Mello, S. K., Dieterle, E., & Duckworth, A. L. (2017) · Educational Psychologist · doi:10.1080/00461520.2017.1281747 · Cited by 350 · see citing works
“Several studies reported recognition rate, which is the proportion of cases in which computer estimates match the human-provided annotations and is highly flawed when there is class skew (i.e., uneven division among categories being discriminated; Hayes & Krippendorff, 2007).”
BackgroundCommunication related health crisis on social media: a case of COVID-19 outbreak
Yu, M., Li, Z., Yu, Z., He, J., & Zhou, J. (2020) · Current Issues in Tourism · doi:10.1080/13683500.2020.1752632 · Cited by 345 · see citing works
“Second, different from data generation based on frequency and category of the concepts, manual content analysis can extract extra latent content hidden in the reviews (Hayes & Krippendorff, 2007).”
MethodRethlefsen, M. L., Farrell, A. M., Osterhaus Trzasko, L. C., & Brigham, T. J. (2015) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2014.11.025 · Cited by 331 · see citing works
Richards, D. R., & Friess, D. A. (2015) · Ecological Indicators · doi:10.1016/j.ecolind.2015.01.034 · Cited by 327 · see citing works
Drones for parcel and passenger transportation: A literature review
Kellermann, R., Biehle, T., & Fischer, L. (2020) · Transportation Research Interdisciplinary Perspectives · doi:10.1016/j.trip.2019.100088 · Cited by 324 · see citing works
Zapf, A., Castell, S., Morawietz, L., & Karch, A. (2016) · BMC Medical Research Methodology · doi:10.1186/s12874-016-0200-9 · Cited by 377 · see citing works
Biggi, A., Gallamini, A., Chauvie, S., Hutchings, M., Kostakoğlu, L., Gregianin, M., Meignan, M. A., Małkowski, B., Hofman, M. S., & Barrington, S. F. (2013) · Journal of Nuclear Medicine · doi:10.2967/jnumed.112.110890 · Cited by 308 · see citing works
“negative) was measured using the Krippendorf a-coefficient (10).”
MethodDireito, A., Dale, L. P., Shields, E., Dobson, R., Whittaker, R., & Maddison, R. (2014) · BMC Public Health · doi:10.1186/1471-2458-14-646 · Cited by 308 · see citing works
“Despite 100% agreement, the calculation of alphas for these techniques was not possible because there was no variation in the reliability data matrix [19].”
MethodInfluentialIncreased Course Structure Improves Performance in Introductory Biology
Freeman, S., Haak, D., & Wenderoth, M. P. (2011) · CBE—Life Sciences Education · doi:10.1187/cbe.10-08-0105 · Cited by 306 · see citing works
“To assess the degree of agreement among the multiple raters, we calculated Krippendorff’s alpha—an appropriate measure for ordinal coding data from multiple raters (Hayes and Krippendorff, 2007)—and the intra-class r.”
MethodSchuurman, D., Baccarne, B., & Marez, L. D. (2012) · Journal of theoretical and applied electronic commerce research · doi:10.4067/s0718-18762012000300006 · Cited by 297 · see citing works
The social media see-saw: Positive and negative influences on adolescents’ affective well-being
Weinstein, E. (2018) · New Media & Society · doi:10.1177/1461444818755634 · Cited by 282 · see citing works
“We repeated the reliability process three times with fresh sub-sets of transcripts until we achieved Krippendorff’s alpha reliability estimates >.75 for each of the 10 sub-codes (Hayes and Krippendorff, 2007).”
MethodAssessing what to address in science communication
de Bruin, W. B., & Bostrom, A. (2013) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1212729110 · Cited by 297 · see citing works
“Measures of reliability typically reflect interrater agreement after correcting for chance agreement (48, 49).”
BackgroundProblems (and solutions) in the measurement of policy diffusion mechanisms
Maggetti, M., & Gilardi, F. (2015) · Journal of Public Policy · doi:10.1017/s0143814x1400035x · Cited by 291 · see citing works
Best Practices for QoE Crowdtesting: QoE Assessment With Crowdsourcing
Hoßfeld, T., Keimel, C., Hirth, M., Gardlo, B., Habigt, J., Diepold, K. J., & Tran‐Gia, P. (2014) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2013.2291663 · Cited by 283 · see citing works
“Krippendorff’s α [63] is a reliability coefficient to measure the agreement among test subjects.”
BackgroundMarzi, G., Balzano, M., & Marchiori, D. (2024) · MethodsX · doi:10.1016/j.mex.2023.102545 · Cited by 292 · see citing works
A framework for AI-powered service innovation capability: Review and agenda for future research
Akter, S., Hossain, M. A., Sajib, S., Sultana, S., Rahman, M., Vrontis, D., & McCarthy, G. (2023) · Technovation · doi:10.1016/j.technovation.2023.102768 · Cited by 302 · see citing works
Understanding the Third-Person Perception: Evidence From a Meta-Analysis
Sun, Y., Pan, Z., & Shen, L. (2008) · Journal of Communication · doi:10.1111/j.1460-2466.2008.00385.x · Cited by 277 · see citing works
“For the 16 variables coded, Krippendorff ’s alpha ranged from .73 to 1.00, all above the acceptable level (.667; Hayes & Krippendorff, 2007; Krippendorff, 2004).”
MethodInfluentialSchöeppe, S., Alley, S., Rebar, A. L., Hayman, M., Bray, N. A., Lippevelde, W. V., Gnam, J.-P., Bachert, P., Direito, A., & Vandelanotte, C. (2017) · International Journal of Behavioral Nutrition and Physical Activity · doi:10.1186/s12966-017-0538-3 · Cited by 280 · see citing works
“Krippendorff ’s alpha (Kalpha) was used to evaluate interrater reliability for the app quality assessment and the presence of BCTs in the apps [28].”
MethodContent Analysis—A Methodological Primer for Gender Research
Neuendorf, K. A. (2010) · Sex Roles · doi:10.1007/s11199-010-9893-0 · Cited by 271 · see citing works
“The aforementioned set of tests of reliability statistics’ characteristics, including Monte Carlo tests (Mooney 1997) and/or bootstrapping techniques (Hayes and Krippendorff 2007), could also provide new information on the statistics’ sampling distributions (e.”
BackgroundMethodInfluentialRetracted: Digital Communication Media Use and Psychological Well-Being: A Meta-Analysis
Liu, D., Baumeister, R. F., Yang, C.-C., & Hu, B. (2019) · Journal of Computer-Mediated Communication · doi:10.1093/jcmc/zmz013 · Cited by 260 · see citing works
RetractedLiu, B. F., & Kim, S. (2011) · Public Relations Review · doi:10.1016/j.pubrev.2011.03.005 · Cited by 253 · see citing works
“Intercoder reliability was assessed using Krippendorff’s alpha (Hayes & Krippendorff, 2007).”
MethodBorsci, S., Malizia, A., Schmettow, M., van der Velde, F., Tariverdiyeva, G., Balaji, D., & Chamberlain, A. (2022) · Personal and Ubiquitous Computing · doi:10.1007/s00779-021-01582-9 · Cited by 266 · see citing works
“The agreement among the participants was equal to 0.780, which is acceptably higher than the minimum level of .667 for the Krippendorff ’ s Alpha [35].”
MethodResultInfluentialSexy, Strong, and Secondary: A Content Analysis of Female Characters in Video Games across 31 Years
Lynch, T., Tompkins, J. E., van Driel, I. I., & Fritz, N. (2016) · Journal of Communication · doi:10.1111/jcom.12237 · Cited by 254 · see citing works
“Krippendorff’s α (Hayes & Krippendorff, 2007) served as the reliability coefficient for all a priori and post hoc analyses.”
MethodWhat Doesn’t Get Measured Does Exist: Improving the Accuracy of Computer-Aided Text Analysis
McKenny, A. F., Aguinis, H., Short, J. C., & Anglin, A. H. (2016) · Journal of Management · doi:10.1177/0149206316657594 · Cited by 257 · see citing works
Bene, M. (2016) · Information Communication & Society · doi:10.1080/1369118x.2016.1198411 · Cited by 257 · see citing works
“These values are considered to be high (Hayes & Krippendorff, 2007).”
BackgroundHow Social Media Is Changing Crisis Communication Strategies: Evidence from the Updated Literature
Cheng, Y. (2016) · Journal of Contingencies and Crisis Management · doi:10.1111/1468-5973.12130 · Cited by 237 · see citing works
“%) were randomly selected for a reliability check, and the result of composite inter-coder reliability reached 0.92 by applying Krippendorff ’s alpha (Hayes & Krippendorff, 2007).”
MethodInfluentialDefine the populist political communication style: the case of Italian political leaders on Twitter
Bracciale, R., & Martella, A. (2017) · Information Communication & Society · doi:10.1080/1369118x.2017.1328522 · Cited by 237 · see citing works
“The coefficients of Krippendorff’s α for the groups were highly satisfying in terms of almost all variables (Krippendorff’s α> 0.83) by conventions established in the field (Hayes & Krippendorff, 2007).”
MethodInfluentialGuarnieri, P., Streit, J. A. C., & Batista, L. C. (2019) · Resources Conservation and Recycling · doi:10.1016/j.resconrec.2019.104541 · Cited by 237 · see citing works
Hemmler, V. L., Kenney, A. W., Langley, S. D., Callahan, C. M., Gubbins, E. J., & Holder, S. (2020) · Qualitative Research · doi:10.1177/1468794120976072 · Cited by 241 · see citing works
“…in interpretivist work in favor of other guidelines, such as trustworthiness as it is demonstrated through credibility, transferability, and dependability (Hammer and Berland, 2014; Hayes and Krippendorff, 2007; Hill et al., 2005; Lincoln and Guba, 1985; Schoenfeld, 1992; Smith and McGannon, 2018).”
BackgroundMethodInfluentialHow pressure from stakeholders affects integrated reporting quality
Vitolla, F., Raimo, N., Rubino, M., & Garzoni, A. (2019) · Corporate Social Responsibility and Environmental Management · doi:10.1002/csr.1850 · Cited by 212 · see citing works
Scaffolding information problem solving in web-based collaborative inquiry learning
Raes, A., Schellens, T., Wever, B. D., & Vanderhoven, E. (2011) · Computers & Education · doi:10.1016/j.compedu.2011.11.010 · Cited by 235 · see citing works
“20 % of students' performance was re-coded by a second rater to check for interrater reliability by means of Krippendorff's alpha (Hayes & Krippendorff, 2007).”
MethodGarcía‐Cabezas, M. Á., John, Y. J., Barbas, H., & Zikopoulos, B. (2016) · Frontiers in Neuroanatomy · doi:10.3389/fnana.2016.00107 · Cited by 227 · see citing works
“Inter-rater agreement was estimated using Krippendorff’s alpha (Hayes and Krippendorff, 2007), and was computed using a custom script written in MATLAB (MATLAB and Statistics Toolbox Release R2015b, The MathWorks, Inc.”
MethodInfluentialWhat to Do (and Not to Do) with the Comparative Manifestos Project Data
Gemenis, K. (2013) · Political Studies · doi:10.1111/1467-9248.12015 · Cited by 226 · see citing works
“…or correlation coefficients such as Pearson's product–moment correlation coefficient are misleading when used as a measure of coding reliability (Hayes and Krippendorff, 2007, Such criticisms have prompted the project to report detailed data regarding the reliability scores of the CMP coders…”
BackgroundMethodInfluentialValues associated with luxury brand consumption and the role of gender
Roux, É., Tafani, É., & Vigneron, F. (2016) · Journal of Business Research · doi:10.1016/j.jbusres.2016.10.012 · Cited by 229 · see citing works
“Krippendorff's alpha (Hayes & Krippendorff, 2007) reveals a good intercoder reliability (≤0.81; 95% confidence interval ranging from 0.76 to 0.86).”
MethodInfluentialNguyen, D., & Hekman, E. (2022) · AI & Society · doi:10.1007/s00146-022-01511-1 · Cited by 255 · see citing works
“The present study deploys an automated content analysis (A.C.A.) using term frequency-inverse document frequency (TD-IDF) to determine the importance of words within a large corpus of news articles.”
InfluentialKnowledge management in small and medium enterprises: a structured literature review
Massaro, M., Handley, K., Bagnoli, C., & Dumay, J. (2016) · Journal of Knowledge Management · doi:10.1108/jkm-08-2015-0320 · Cited by 223 · see citing works
Algorithmic bias in machine learning-based marketing models
Akter, S., Dwivedi, Y. K., Sajib, S., Biswas, K., Bandara, R. J., & Michael, K. (2022) · Journal of Business Research · doi:10.1016/j.jbusres.2022.01.083 · Cited by 242 · see citing works
Why and when hierarchy impacts team effectiveness: A meta-analytic integration.
Greer, L. L., Jong, B. A. D., Schouten, M. E., & Dannals, J. E. (2018) · Journal of Applied Psychology · doi:10.1037/apl0000291 · Cited by 222 · see citing works
Cultural Brokerage and Creative Performance in Multicultural Teams
Jang, S. (2017) · Organization Science · doi:10.1287/orsc.2017.1162 · Cited by 223 · see citing works
“In line with suggestions by Krippendorff and colleagues (e.g., Krippendorff 2004, Hayes and Krippendorff 2007) and following other researchers (e.g., Ely 1995, Homan et al. 2007), I coded the dialogue of a random subset of 20 teams (24 percent of the total sample) using the same protocol to check…”
ResultBenefits of a Tangible Interface for Collaborative Learning and Interaction
Schneider, B., Jermann, P., Zufferey, G., & Dillenbourg, P. (2010) · IEEE Transactions on Learning Technologies · doi:10.1109/tlt.2010.36 · Cited by 218 · see citing works
“Concerning collaboration rating, a second judge rated 20 percent of the dyads; interreliability using Krippendorff’s alpha was 0.93 [55].”
MethodInfluentialAutomated identification of media bias in news articles: an interdisciplinary literature review
Hamborg, F., Donnay, K., & Gipp, B. (2018) · International Journal on Digital Libraries · doi:10.1007/s00799-018-0261-y · Cited by 217 · see citing works
“, the widely used Krippendorff’s Alpha [185], provide estimates for the reliability and robustness of the coding.”
MethodValidation of ICA-based myogenic artifact correction for scalp and source-localized EEG
McMenamin, B. W., Shackman, A. J., Maxwell, J. S., Bachhuber, D. R. W., Koppenhaver, A. M., Greischar, L. L., & Davidson, R. J. (2009) · NeuroImage · doi:10.1016/j.neuroimage.2009.10.010 · Cited by 211 · see citing works
“Interrater reliability, assessed prior to consensus using Krippendorff's alpha (Hayes and Krippendorff, 2007), was excellent, α=0.”
InfluentialLow agreement among reviewers evaluating the same NIH grant applications
Pier, E. L., Brauer, M., Filut, A., Kaatz, A., Raclaw, J., Nathan, M. J., Ford, C. E., & Carnes, M. (2018) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1714379115 · Cited by 210 · see citing works
“Second, to corroborate the findings from estimating the ICC, we computed agreement among reviewers for each variable using the Krippendorff’s alpha statistic (23).”
MethodInterobserver agreement of various thyroid imaging reporting and data systems
Grani, G., Lamartina, L., Cantisani, V., Maranghi, M., Lucia, P., & Durante, C. (2017) · Endocrine Connections · doi:10.1530/ec-17-0336 · Cited by 210 · see citing works
“Agreement on ordinal ratings was assessed with the Krippendorff α statistic (17).”
MethodA Systematic Review of Reviews of the Outcome of Noninstitutional Child Maltreatment
Carr, A., Duff, H., & Craddock, F. (2018) · Trauma Violence & Abuse · doi:10.1177/1524838018801334 · Cited by 214 · see citing works
“Percentage agreement and Krippendorff’s a (Hayes & Krippendorff, 2007) were used to determine interrater agreement and reliability.”
MethodHienerth, C., Lettl, C. U., & Keinz, P. (2013) · Journal of Product Innovation Management · doi:10.1111/jpim.12127 · Cited by 205 · see citing works
“The measure for reliability using Krippendorff’s alpha resulted in high agreement (.87; values above .80 are generally considered highly reliable, e.g., see Hayes and Krippendorff, 2007).”
MethodThe Evolution of Media Effects Theory: A Six-Stage Model of Cumulative Research
Neuman, W. R., & Guggenheim, L. (2011) · Communication Theory · doi:10.1111/j.1468-2885.2011.01381.x · Cited by 209 · see citing works
“For each of the variables, α was determined with SPSS following Hayes and Krippendorff (2007).”
MethodA systematic review and meta-analysis of the intergenerational transmission of criminal behavior
Besemer, S., Ahmad, S. I., Hinshaw, S. P., & Farrington, D. P. (2017) · Aggression and Violent Behavior · doi:10.1016/j.avb.2017.10.004 · Cited by 203 · see citing works
Improving Teamwork and Communication in Trauma Care Through In Situ Simulations
Miller, D. G., Crandall, C. S., Washington, C., & McLaughlin, S. A. (2012) · Academic Emergency Medicine · doi:10.1111/j.1553-2712.2012.01354.x · Cited by 201 · see citing works
“85 for overall decisionmaking).(8) We calculated median and interquartile range (IQR) scores for each of the CTS component measures and compared the distribution of the scores across the four phases of the study.”
MethodSocial movement web use in theory and practice: a content analysis of US movement websites
Stein, L. (2009) · New Media & Society · doi:10.1177/1461444809105350 · Cited by 201 · see citing works
Artstein, R. (2017) · Springer eBooks · doi:10.1007/978-94-024-0881-2_11 · Cited by 206 · see citing works
“And while confidence intervals for agreement coefficients can be estimated through resampling of the annotated items (Hayes and Krippendorff, 2007), this method cannot be used to quantify annotator variation, because resampling annotators would result in measuring agreement between an annotator and…”
MethodMeasuring and Reporting Intercoder Reliability in Plan Quality Evaluation Research
Stevens, M. R., Lyles, W., & Berke, P. R. (2014) · Journal of Planning Education and Research · doi:10.1177/0739456x13513614 · Cited by 197 · see citing works
“In response to an ever-growing demand in the social sciences literature for an effective approach to measuring intercoder reliability, communications scholars Hayes and Krippendorff (2007) suggest five criteria that a “good” measure of reliability should satisfy.”
BackgroundMethodInfluentialPanadero, E., Romero, M., & Strijbos, J.-W. (2013) · Studies in Educational Evaluation · doi:10.1016/j.stueduc.2013.10.005 · Cited by 195 · see citing works
Mas, F. D., Massaro, M., Rippa, P., & Secundo, G. (2023) · Technovation · doi:10.1016/j.technovation.2023.102716 · Cited by 195 · see citing works
Socially mediated populism: the communicative strategies of political leaders on Facebook
Mazzoleni, G., & Bracciale, R. (2018) · Humanities and Social Sciences Communications · doi:10.1057/s41599-018-0104-x · Cited by 193 · see citing works
“…of the people (Akkerman et al., 2014; Albertazzi and McDonnell, 2008; Biorcio, 2015; Bos et al., 2013; Caiani and Graziano, 2016; Canovan, 1981; Hawkins et al., 2012; Jagers and Walgrave, 2007; Mény and Surel, 2002; Mudde, 2004; Taggart, 2000; van Kessel and Castelein, 2016; Wirth et al., 2016).”
BackgroundResultInfluentialRevisiting the Epistemology of Fact-Checking
Amazeen, M. A. (2015) · Critical Review · doi:10.1080/08913811.2014.993890 · Cited by 192 · see citing works
“Using a macro in SPSS 20.0.0 to calculate Krippendorff’s alpha (Hayes and Krippendorff 2007), all variables emerged above this inter-coder reliability standard.”
InfluentialAslanidis, P. (2017) · Quality & Quantity · doi:10.1007/s11135-017-0517-4 · Cited by 192 · see citing works
- Authors
- Hayes, A. F., & Krippendorff, K.
- Affiliations
- The Ohio State University (US)
- University of Pennsylvania (US)
- Venue
- Communication Methods and Measures
- Publisher
- Informa UK Limited
- Published
- 2007-04
- Volume
- 1
- Issue
- 1
- Pages
- 77-89
- ISSN
- 1931-2458 (ISSN-L)
- 1931-2466
- OpenAlex
- W2061504941
- Cited by
- 4,225
- References
- 30
- Citations by year
- 2026: 284 · 2025: 278 · 2024: 280 · 2023: 289 · 2022: 331 · 2021: 379 · 2020: 369 · 2019: 296 · 2018: 260 · 2017: 300 · 2016: 248 · 2015: 205 · 2014: 206 · 2013: 150 · 2012: 107
- FWCI
- 50.07
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Communication Methods and Measures (publishedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Reliability and Agreement in Measurement
- Social Sciences › General Social Sciences › Computational and Text Analysis Methods
- Social Sciences › Decision Sciences › Management Science and Operations Research › Psychometric Methodologies and Testing
- Keywords
- Krippendorff's alpha, inter-rater reliability, content analysis, coding reliability, reliability coefficients, observer agreement, SAS macro, missing data
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
