The DOI matches its Crossref record; 30 of the 30 references it cites were found in a registry or an index.
Answering the Call for a Standard Reliability Measure for Coding Data
Abstract
In content analysis and similar methods, data are typically generated by trained human observers who record or transcribe textual, pictorial, or audible matter in terms suitable for analysis. Conclusions from such data can be trusted only after demonstrating their reliability. Unfortunately, the content analysis literature is full of proposals for so-called reliability coefficients, leaving investigators easily confused, not knowing which to choose. After describing the criteria for a good measure of reliability, we propose Krippendorff's alpha as the standard reliability measure. It is general in that it can be used regardless of the number of observers, levels of measurement, sample sizes, and presence or absence of missing data. To facilitate the adoption of this recommendation, we describe a freely available macro written for SPSS and SAS to calculate Krippendorff's alpha and illustrate its use with a simple example.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“We calculate Krippendorff's alpha (Hayes and Krippendorff, 2007) to measure inter-rater reliability (see Table 3).”
From LLMs cannot find reasoning errors, but can correct them given the error location (2024) · cites this paper as Hayes and Krippendorff, 2007
In the citing paper
“Based on Krippendorff's alpha (α = 0.92), the level of agreement between coders was high and well above the threshold of 0.70 (Hayes & Krippendorff, 2007).”
From Collaboration, Coordination, and Cooperation Among Organizations: Establishing the Distinctive Meanings of These Terms Through a Systematic Literature Review (2020) · cites this paper as Hayes & Krippendorff, 2007
In the citing paper
“we calculated the inter-coder reliability based on 25% of all data units and derived a kappa value of .96, which reflects excellent reliability.”
In the citing paper
“To assess the inter-annotator agreement of stage 1, we report Krippendorf's α (Hayes and Krippendorff, 2007) for each dimension for the 135 articles after training and before any discussion/adjudication: economic (0.44), foreign (0.68), social (0.39).”
From Political Ideology and Polarization: A Multi-dimensional Approach (2022) · cites this paper as Hayes and Krippendorff, 2007
In the citing paper
“This family of metrics does not correct for chance (Cohen, 1960;Hayes and Krippendorff, 2007) and overstates discriminative ability (Guerdan et al., 2025;Han et al., 2025;Collot et al., 2025).”
From Reliability without Validity: A Systematic, Large-Scale Evaluation of LLM-as-a-Judge Models Across Agreement, Consistency, and Bias (2026) · cites this paper as Hayes and Krippendorff, 2007
Citations
Cites
Coefficient Alpha and the Internal Structure of Tests
Cronbach, L. J. (1951) · Psychometrika · doi:10.1007/bf02310555 · Cited by 43,931
A Coefficient of Agreement for Nominal Scales
Cohen, J. (1960) · Educational and Psychological Measurement · doi:10.1177/001316446002000104 · Cited by 42,398
An Introduction to the Bootstrap
Efron, B., & Tibshirani, R. J. (1994) · doi:10.1201/9780429246593 · Cited by 40,500
Nonparametric Statistics for the Behavioral Sciences
Guitton, H., & Siegel, S. (1958) · Revue économique · doi:10.2307/3498751 · Cited by 35,279
Content Analysis: An Introduction to its Methodology.
Shelley, M., & Krippendorff, K. (1984) · Journal of the American Statistical Association · doi:10.2307/2288384 · Cited by 24,351
Self-citationStatistical Methods for Rates and Proportions
Fleiss, J. L., Levin, B., & Paik, M. C. (2003) · Wiley Series in Probability and Statistics · doi:10.1002/0471445428 · Cited by 18,646
Measuring nominal scale agreement among many raters.
Fleiss, J. L. (1971) · Psychological Bulletin · doi:10.1037/h0031619 · Cited by 8,733
Content Analysis for the Social Sciences and Humanities.
Parker, E. B., & Holsti, O. R. (1970) · American Sociological Review · doi:10.2307/2093233 · Cited by 4,553
Content Analysis in Mass Communication: Assessment and Reporting of Intercoder Reliability
Lombard, M., Snyder-Duch, J., & Bracken, C. C. (2002) · Human Communication Research · doi:10.1111/j.1468-2958.2002.tb00826.x · Cited by 3,068
An Introduction to the Bootstrap
Grimshaw, S. D., Efron, B., & Tibshirani, R. J. (1995) · Technometrics · doi:10.2307/1269918 · Cited by 2,281
Research works printed but not linked: 1
- 10.1111/j.1468-2958.2004.tb00738.x · DOI · Paper page
Cited by
Sorted by citations
Items 601–700 of 1,013
Overcoming consumer resistance to innovations – an analysis of adoption triggers
Reinhardt, R., Hietschold, N., & Gurtner, S. (2017) · R and D Management · doi:10.1111/radm.12259 · Cited by 50
Hintz, E. A., Betts, T., & Buzzanell, P. M. (2021) · Health Communication · doi:10.1080/10410236.2021.1953727 · Cited by 50
“Coders participated in 3 rounds of coding with 50 posts coded in each round (n = 150 total) to establish inter-coder reliability using Krippendorff’s alpha (KALPHA = .86; Hayes & Krippendorff, 2007).”
MethodStatistical Genre Analysis: Toward Big Data Methodologies in Technical Communication
Graham, S. S., Kim, S., DeVasto, D. M., & Keith, W. M. (2014) · Technical Communication Quarterly · doi:10.1080/10572252.2015.975955 · Cited by 52
Comparing Separate Process and Intertwined Models for Reactance
Kim, S., Levine, T. R., & Allen, M. R. (2013) · Communication Studies · doi:10.1080/10510974.2012.755639 · Cited by 50
“Modeldata fit was assessed examining v(2) statistic, CFI (Hu & Bentler, 1999), and the dual criteria (RMSEA and SRMR) recommended by Hu and Bentler (1999). Also considered were Akaike Information Criterion (AIC) (Akaike, 1987) and Bayesian Information Criterion (BIC) (Raftery, 1995) for comparison of the non-nested mo…”
BackgroundMethodCACTI: Free, Open-Source Software for the Sequential Coding of Behavioral Interactions
Glynn, L. H., Hallgren, K. A., Houck, J. M., & Moyers, T. B. (2012) · PLoS ONE · doi:10.1371/journal.pone.0039740 · Cited by 50
“We also have reported estimates of reliability for ELICIT using Krippendorff’s alpha [43], a measure of reliability than has been suggested for standard use in coding studies [44].”
MethodMair, G., von Kummer, R., Adami, A., White, P., Adams, M. E., Yan, B., Demchuk, A. M., Farrall, A., Sellar, R. J., Ramaswamy, R., et al. (2014) · Neuroradiology · doi:10.1007/s00234-014-1441-0 · Cited by 50
“K-alpha has been shown to provide a more robust measure of observer variance than kappa or ICC and provides several advantages to the user; it allows comparisons between any number of observers, it can handle both categorical and ordinal data, it is less prone to the influences of observer bias and result prevalence a…”
BackgroundOn the unreliability of bug severity data
Tian, Y., Ali, N., Lo, D., & Hassan, A. E. (2015) · Empirical Software Engineering · doi:10.1007/s10664-015-9409-1 · Cited by 50
“Because the Krip-pendorff’s alpha reliability generalizes across scales of measurement, can be used with any number of observes, with or without missing data (Hayes and Krippendorff 2007).”
BackgroundArdiana, P. A. (2019) · Australian Accounting Review · doi:10.1111/auar.12293 · Cited by 50
Ontologies relevant to behaviour change interventions: a method for their development
Wright, A. J., Norris, E., Finnerty, A. N., Marques, M. M., Johnston, M., Kelly, M. P., Hastings, J., West, R., & Michie, S. (2020) · Wellcome Open Research · doi:10.12688/wellcomeopenres.15908.3 · Cited by 50
Dade, A., & Hassenzahl, D. M. (2013) · International Journal of Sustainability in Higher Education · doi:10.1108/ijshe-08-2011-0053 · Cited by 50
Environmental Management Accounting
Filho, W. L., Azul, A. M., Brandli, L., Özuyar, P. G., & Wall, T. (2020) · Encyclopedia of the UN sustainable development goals · doi:10.1007/978-3-319-95726-5_300054 · Cited by 50
Esmark, A., & Schoop, S. R. (2017) · Journal of European Social Policy · doi:10.1177/0958928716688262 · Cited by 49
“Krippendorff’s alpha has been calculated using Hayes macro in SPSS (Hayes and Krippendorff, 2007).”
MethodDynamics of positive deviance in destigmatisation: celebrities and the media in the rise of veganism
Lundahl, O. (2018) · Consumption Markets & Culture · doi:10.1080/10253866.2018.1512492 · Cited by 63
“I assessed Krippendorff with the Kalpha macro (Hayes and Krippendorff 2007) in IBM SPSS 23 with 5000 bootstraps.”
MethodThe “Serial Effect” and the True Crime Podcast Ecosystem
Sherrill, L. A. (2020) · Journalism Practice · doi:10.1080/17512786.2020.1852884 · Cited by 49
Kepplinger, H. M., & Lemke, R. (2015) · Political Communication · doi:10.1080/10584609.2015.1022240 · Cited by 49
“In addition, for the second test Krippendorff’s α was calculated using the Hayes and Krippendorff (2007) SPSS tool.”
MethodAdi, A., Erickson, K., & Lilleker, D. G. (2014) · Policy & Internet · doi:10.1002/1944-2866.poi350 · Cited by 49
“The researchers used Krippendorff’s alpha, deemed an appropriate method of assessing the variation between coders in this type of content analysis research (Hayes & Krippendorff, 2007).”
MethodHolistic approach to learning and teaching introductory object-oriented programming
Thota, N., & Whitfield, R. (2010) · Computer Science Education · doi:10.1080/08993408.2010.486260 · Cited by 51
“Acceptable inter-rater reliability figures are reported as: 84.52% for average pairwise percent agreement; 0.817 for average pairwise Cohen’s k ; 0.817 for Krippendorff’s a (Hayes & Krippendorff, 2007).”
MethodInfluentialPenetration–Aspiration: Is Their Detection in FEES® Reliable Without Video Recording?
Hey, C., Pluschinski, P., Pajunk, R., Almahameed, A., Girth, L., Sader, R. A., Stöver, T., & Zaretsky, Y. (2015) · Dysphagia · doi:10.1007/s00455-015-9616-3 · Cited by 49
Aresi, G., Moore, S., & Marta, E. (2016) · Substance Use & Misuse · doi:10.1080/10826084.2016.1201116 · Cited by 49
“Eligible studies were independently appraised by two reviewers and inter-rater reliability was assessed using Krippendorff ’s alpha (α) (Hayes & Krippendorff, 2007).”
MethodBene, M. (2017) · Journal of Information Technology & Politics · doi:10.1080/19331681.2017.1367348 · Cited by 49
“These values are considered to be high (Hayes & Krippendorff, 2007).”
BackgroundBuild community before the storm: The National Weather Service's social media engagement
Olson, M. K., Sutton, J. N., Vos, S. C., Prestley, R., Renshaw, S. L., & Butts, C. T. (2019) · Journal of Contingencies and Crisis Management · doi:10.1111/1468-5973.12267 · Cited by 52
“Reliability was measured with Krippendorff's alpha (α), using the SPSS macro created by Hayes and Krippendorff (2007).”
Method“I Just Want to Feel Safe”: A Diary Study of Safety Perceptions on Social Media
Redmiles, E. M., Bodford, J. E., & Blackwell, L. (2019) · Proceedings of the International AAAI Conference on Web and Social Media · doi:10.1609/icwsm.v13i01.3356 · Cited by 51
“They achieved a Krippendorff’s alpha (Hayes and Krippendorff 2007) of 0.96 and percentage agreement of 0.99.”
BackgroundInfluentialPopulism and emotions: Italian political leaders’ communicative strategies to engage Facebook users
Martella, A., & Bracciale, R. (2021) · Innovation The European Journal of Social Science Research · doi:10.1080/13511610.2021.1951681 · Cited by 52
“Krippendorff’s α from 0.73 to 0.82 and an agreement percentage from 94% to 98% have been considered highly satisfactory values according to field norms (Hayes and Krippendorff 2007).”
InfluentialEffects of a digital intervention on the development of algebraic expertise
Bokhove, C., & Drijvers, P. H. M. (2011) · Computers & Education · doi:10.1016/j.compedu.2011.08.010 · Cited by 49
Dano, H., Altınay, S., Arnould, L., Blétard, N., Colpaert, C., Dedeurwaerdere, F., Dessauvagie, B. F., Duwel, V., Floris, G., Fox, S. B., et al. (2019) · Modern Pathology · doi:10.1038/s41379-019-0367-9 · Cited by 47
Bianchi, I. M., Grassi, A., Pham, T. O., Telías, I., Droghi, M. T., Vieira, F. N., Jonkman, A. H., Brochard, L. J., & Bellani, G. (2021) · Journal of Critical Care · doi:10.1016/j.jcrc.2021.12.002 · Cited by 48
Analyzing Dataset Annotation Quality Management in the Wild
Klie, J.-C., de Castilho, R. E., & Gurevych, I. (2024) · Computational Linguistics · doi:10.1162/coli_a_00516 · Cited by 66
“…D e : Compared with Fleiss’s κ , Krippendorff’s α is more powerful and versatile: It can deal with missing annotations, supports more than two annotations per instance, and can be generalized to handle even categorical, ordinal, hierarchical, or continuous data (Hayes and Krippendorff 2007).”
MethodThe management's reaction to new mandatory risk disclosure
Greco, G. (2012) · Corporate Communications An International Journal · doi:10.1108/13563281211220256 · Cited by 48
“Reproducibility evaluates whether a coding instrument, serving as a set of common instructions to different observers of the same set of phenomena, gives the same output within an acceptable margin of error (Krippendorf and Hayes, 2007).”
BackgroundSchönbucher, V. B., Maier, T., Mohler‐Kuo, M., Schnyder, U., & Landolt, M. A. (2014) · Archives of Sexual Behavior · doi:10.1007/s10508-013-0230-x · Cited by 48
“…but on the perpetrator’s side (e.g., somebody defended the perpetrator) SV sexual victimization a When participants did not differentiate between her/his mother or father but just talked from his/her parents reliability, as indicated by a Krippendorff’s alpha of 0.98 (Hayes & Krippendorff, 2007).”
MethodInfluentialPatterns and trends in event tourism study topics over 30 years
Kim, J., Boo, S., & Kim, Y. (2013) · International Journal of Event and Festival Management · doi:10.1108/17582951311307520 · Cited by 48
Bringing it all back home? Social media practices by Swedish municipalities
Larsson, A. O. (2013) · European Journal of Communication · doi:10.1177/0267323113502277 · Cited by 48
“this, the Krippendorf alpha macro for SPSS was used (Hayes and Krippendorff, 2007).”
MethodPoldner, E., Simons, P. R. J., Wijngaards, G., & van der Schaaf, M. F. (2011) · Educational Research Review · doi:10.1016/j.edurev.2011.11.002 · Cited by 53
Reputation Repair at the Expense of Providing Instructing and Adjusting Information Following Crises
Kim, S., Avery, E. J., & Lariscy, R. W. (2011) · International Journal of Strategic Communication · doi:10.1080/1553118x.2011.566903 · Cited by 50
“Intercoder reliability was assessed using Krippendorff's Alpha ( Hayes & Krippendorff, 2007 ), which is appropriate for measurement level variables from nominal to ratio and accounts for chance agreement, making it a more conservative estimate of reliability ( Lombard, Snyder-Duch, & Bracken, 2002 ).”
MethodDoes reflection have an effect upon case-solving abilities of undergraduate medical students?
Koole, S., Dornan, T., Aper, L., Scherpbier, A. J. J. A., Valcke, M., Cohen‐Schotanus, J., & Derese, A. (2012) · BMC Medical Education · doi:10.1186/1472-6920-12-75 · Cited by 48
“A Krippendorff ’s alpha coefficient [26] above 0.”
The euro crisis in metaphors and frames: Focus on the press in the Low Countries
Joris, W., d’Haenens, L., & Gorp, B. V. (2014) · European Journal of Communication · doi:10.1177/0267323114538852 · Cited by 48
“Training resulted in high scores of inter-coder reliability (Krippendorff’s alpha) executed on a sample of 10% of the articles (n = 108) (Hayes and Krippendorff, 2007): war (α = .”
MethodPier, E. L., Raclaw, J., Kaatz, A., Brauer, M., Carnes, M., Nathan, M. J., & Ford, C. E. (2017) · Research Evaluation · doi:10.1093/reseval/rvw025 · Cited by 48
“To quantify this variability in the Preliminary Reviewer Scores, we computed Krippendorff’s a (Hayes and Krippendorff 2007; Krippendorff 2013), an inter-rater reliability coefficient that accommodates more than two raters and allows for missing values.”
MethodDying Is Unexpectedly Positive
Goranson, A., Ritter, R. S., Waytz, A. G., Norton, M. I., & Gray, K. J. (2017) · Psychological Science · doi:10.1177/0956797617701186 · Cited by 48
“Interrater reliability was calculated using the KALPHA macro for SPSS (Hayes & Krippendorff, 2007) and was reasonable for both positive (Krippendorff’s α = .76) and negative (Krippendorff’s α = .79) affect.”
MethodInfluentialHan, D., Heuvelmans, M. A., Vliegenthart, R., Rook, M., Dorrius, M. D., de Jonge, G. J., Walter, J. E., van Ooijen, P. M. A., de Koning, H. J., & Oudkerk, M. (2017) · British Journal of Radiology · doi:10.1259/bjr.20170405 · Cited by 48
Child Sexual Abuse News: A Systematic Review of Content Analysis Studies
Popović, S. (2018) · Journal of Child Sexual Abuse · doi:10.1080/10538712.2018.1486935 · Cited by 48
“Krippendorff’s alpha (Kalpha) is the most appropriate reliability coefficient usually recommended as a standard reliability measure for content analysis due to a number of advantages (Hayes & Krippendorff, 2007; Riffe et al., 2005).”
MethodAs if sand were stone. New concepts and metrics to probe the ground on which to build trustable AI
Cabitza, F., Campagner, A., & Sconfienza, L. M. (2020) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-020-01224-9 · Cited by 46
Age differences in callosal contributions to cognitive processes
Fling, B. W., Chapekis, M., Reuter‐Lorenz, P. A., Anguera, J. A., Bo, J., Langan, J., Welsh, R. C., & Seidler, R. D. (2011) · Neuropsychologia · doi:10.1016/j.neuropsychologia.2011.05.004 · Cited by 48
“Four independent raters made all callosal measurements and inter-rater reliability was assessed with both Krippendorff's alpha (Hayes & Krippendorff, 2007) and intraclass correlation coefficient (ICC).”
MethodBusiness and Human Rights in South Africa: An Analysis of Antecedents of Human Rights Due Diligence
Hamann, R., Sinha, P. N., Kapfudzaruwa, F., & Schild, C. (2009) · Journal of Business Ethics · doi:10.1007/s10551-009-0293-8 · Cited by 48
Cooperative gestures: Effective signaling for humanoid robots
Riek, L. D., Rabinowitch, T.-C., Bremner, P., Pipe, T., Fraser, M., & Robinson, P. (2010) · ACM/IEEE International Conference on Human-Robot Interaction (HRI) · doi:10.1109/hri.2010.5453266 · Cited by 48
Hierarchical Mum Effect: A New Investigation of Organizational Ethics
Ploeger, N. A., Kelley, K. M., & Bisel, R. S. (2011) · Southern Communication Journal · doi:10.1080/1041794x.2010.500343 · Cited by 47
“Measures of initial coding reliability and coder drift were computed as Krippendorff's α on a sample of 15–20 responses (Hayes & Krippendorff, 2007 ; Krippendorff, 2004 ).”
MethodEinwiller, S. A., & Kim, S. (2020) · Policy & Internet · doi:10.1002/poi3.239 · Cited by 47
“As Krippendorff's α tests were satisfactory for all measured variables, ranging from 0.78 to 1.00 for all countries, the remainder of the sample was coded (Hayes & Krippendorff, 2007).”
MethodInfluentialChecking PolitiFact’s Fact-Checks
Nieminen, S., & Sankari, V. (2021) · Journalism Studies · doi:10.1080/1461670x.2021.1873818 · Cited by 56
Bloss, J., LePrevost, C. E., Zahra, A. G., Firnhaber, G. C., Cofie, L. E., Zepeda, R., & Lee, J. G. L. (2021) · Health Promotion Practice · doi:10.1177/15248399211033308 · Cited by 47
Data Market Design: A Systematic Literature Review
Driessen, S. W., Monsieur, G., & van den Heuvel, W. (2022) · IEEE Access · doi:10.1109/access.2022.3161478 · Cited by 47
“In order to assess the inter-rater reliability of the inclusion assessment, the Krippendorff test was used to assess the agreement achieved among observers who categorise a given set of objects [58].”
MethodUsing nudges to realize project performance management
Bukoye, O. T., EJOHWOMU, O. A., Roehrich, J. K., & Too, J. (2022) · International Journal of Project Management · doi:10.1016/j.ijproman.2022.10.003 · Cited by 48
Bossert, P. E., Dunn, M. P., & Thomsen, G. H. (2013) · Developmental Dynamics · doi:10.1002/dvdy.24021 · Cited by 48
“Several reliability statistics—the Pearson product-moment correlation coefficient (R), a weighted Cohen’s kappa (kw) (Cohen, 1968), and Krippendorff’s alpha (a) (Hayes and Krippendorff, 2007)—were determined for each temperature (Table 2).”
MethodThe micro-processes during repatriate knowledge transfer: the repatriates’ perspective
Burmeister, A., Deller, J., Osland, J. S., Szkudlarek, B. A., Oddou, G. R., & Blakeney, R. N. (2015) · Journal of Knowledge Management · doi:10.1108/jkm-01-2015-0011 · Cited by 47
“The intercoder reliability was calculated using the macro KALPHA, provided by Hayes and Krippendorff (2007). Krippendorff’s was satisfactory with 0.”
MethodSchneider, B., & Pea, R. D. (2014) · International Journal of Computer-Supported Collaborative Learning · doi:10.1007/s11412-014-9202-y · Cited by 49
“Inter-reliability index using Krippendorff ’ s alpha was 0.81 (a value higher than 0.8 is considered as a reliable agreement between judges; Hayes and Krippendorff 2007).”
MethodInfluentialHow reliable are case formulations? A systematic literature review
Flinn, L. C., Braham, L. G., & das Nair, R. (2014) · British Journal of Clinical Psychology · doi:10.1111/bjc.12073 · Cited by 47
“Although percentage agreement can be used as a measure of reliability, it has been described as flawed in many respects (Hayes & Krippendorff, 2007) as it does not take into consideration agreement based on chance (McHugh, 2012).”
MethodExtracting narrative timelines as temporal dependency structures
Kolomiyets, O., Bethard, S. J., & Moens, M. (2012) · Lirias · Cited by 47
Hedman, U. (2017) · Journalism · doi:10.1177/1464884917734054 · Cited by 47
Customer query handling in sales interactions
Singh, S. K., Marinova, D., Singh, J., & Evans, K. R. (2017) · Journal of the Academy of Marketing Science · doi:10.1007/s11747-017-0569-y · Cited by 47
“Two research assistants classified each word into two dimensions (inter-coder reliability = .89 after three iterations) (Hayes and Krippendorff 2007): (1) Bevaluating^ words that indicate the salesperson’s skill and expertise related to resolving (e.g., why, when, what, while, because) and (2)…”
MethodPlaying Multiple Positions: Student-Athlete Identity Salience and Conflict
Lu, L. D., Heinze, K. L., & Soderstrom, S. B. (2018) · Journal of Intercollegiate Sport · doi:10.1123/jis.2018-0034 · Cited by 47
“In order to assess intercoder reliability, we calculated Krippendorff’s alpha (also known as Kalpha) for each code (De Swert, 2012; Hayes & Krippendorff, 2007).”
MethodThe communication of uncertainty in health: A scoping review
Kalke, K., Studd, H., & Scherr, C. L. (2021) · Patient Education and Counseling · doi:10.1016/j.pec.2021.01.034 · Cited by 52
Norris, E., Wright, A. J., Hastings, J., West, R., Boyt, N., & Michie, S. (2021) · Wellcome Open Research · doi:10.12688/wellcomeopenres.16682.1 · Cited by 47
Leveraging member expertise to improve knowledge transfer and demonstrability in groups.
Bonner, B. L., & Baumann, M. R. (2011) · Journal of Personality and Social Psychology · doi:10.1037/a0025566 · Cited by 50
Natural affect data — Collection & annotation in a learning context
Afzal, S., & Robinson, P. (2009) · International Conference on Affective Computing and Intelligent Interaction and workshops · doi:10.1109/acii.2009.5349537 · Cited by 48
“Inter-coder reliability measures for nominal data include raw-agreement, Scott’s pi, Cohen’s kappa, Fleiss’ kappa, and Krippendorff’s alpha [ 14 ].”
MethodZhong, W., Hu, Q., & Kapucu, N. (2022) · Public Administration · doi:10.1111/padm.12855 · Cited by 51
Pérez, J., Díaz, J., Berrocal, J., López-Viana, R., & González-Prieto, Á. (2022) · Computing · doi:10.1007/s00607-022-01104-2 · Cited by 47
“After completing the coding process, Krippendorff’s α coefficients [26, 32] were computed (see also [18] for a thorough introduction to these techniques).”
The unbearable (technical) unreliability of automated facial emotion recognition
Cabitza, F., Campagner, A., & Mattioli, M. (2022) · Big Data & Society · doi:10.1177/20539517221129549 · Cited by 58
“In this context, then reliability of the annotations refers to, and is operationalized as, the degree of agreement for the labels provided by the raters involved, i.e. what in the literature is usually called the inter-rater reliability (Hayes and Krippendorff, 2007).”
BackgroundRese, A., & Tränkner, P. (2023) · International Journal of Information Management · doi:10.1016/j.ijinfomgt.2023.102699 · Cited by 52
Ali, W., Wilson, J., & Frynas, J. G. (2024) · Corporate Social Responsibility and Environmental Management · doi:10.1002/csr.2869 · Cited by 46
On-screen print: the role of captions as a supplemental literacy tool
Linebarger, D. L., Piotrowski, J. T., & Greenwood, C. R. (2009) · Journal of Research in Reading · doi:10.1111/j.1467-9817.2009.01407.x · Cited by 46
Spontaneous mentalizing captures variability in the cortical thickness of social brain regions
Warnell, K. R., & Redcay, E. (2014) · Social Cognitive and Affective Neuroscience · doi:10.1093/scan/nsu081 · Cited by 53
Wuzzit Trouble: The Influence of a Digital Math Game on Student Number Sense
Pope, H., & Mangram, C. (2015) · International Journal of Serious Games · doi:10.17083/ijsg.v2i4.88 · Cited by 46
“For this reason, we used Krippendorff’s alpha for continuous data [23].”
MethodSignaling Service Quality via Website e-CRM Features
Tian, J., & Wang, S. (2014) · Journal of Hospitality & Tourism Research · doi:10.1177/1096348014525634 · Cited by 49
“Krippendorff’s alpha (Hayes & Krippendorff, 2007) was computed to assess the degree to which independent coders agreed in their coding.”
MethodChandler, M., Rullman, S. D., Cousins, J. A., Esmail, N., Begin, E., Venicx, G., Eisenberg, C., & Studer, M. (2016) · Biological Conservation · doi:10.1016/j.biocon.2016.09.024 · Cited by 46
Rodríguez-Hidalgo, C., Tan, E. S., & Verlegh, P. W. J. (2017) · Computers in Human Behavior · doi:10.1016/j.chb.2017.04.007 · Cited by 46
Massé, F., & Margulies, J. D. (2020) · World Development · doi:10.1016/j.worlddev.2020.104958 · Cited by 46
The Partisan Politics of Law and Order
Wenzelburger, G. (2020) · doi:10.1093/oso/9780190920487.001.0001 · Cited by 46
Wang, H., Qi, H., & Ran, B. (2021) · Administration & Society · doi:10.1177/00953997211009890 · Cited by 46
“Based on Krippendorff’s alpha (α = .93), the level of agreement between the two coders was high and well above the threshold of .70 (Hayes & Krippendorff, 2007).”
Analysing creative behaviour in the later stage design process
Snider, C. M., Culley, S. J., & Dekoninck, E. A. (2013) · Design Studies · doi:10.1016/j.destud.2013.03.001 · Cited by 46
“Both Cohen’s kappa (Cohen, 1960) and Krippendorff’s alpha (Hayes & Krippendorff, 2007) were used as measures of reliability, with testing achieving values of 0.770 and 0.768 respectively after three training iterations.”
MethodInfluentialAmerican public opinion, advocacy, and policy in Congress: what the public wants and what it gets
(2014) · Choice Reviews Online · doi:10.5860/choice.52-1106 · Cited by 46
Simulating Opinion Dynamics with Networks of LLM-based Agents
Chuang, Y.-S., Goyal, A., Harlalka, N., Suresh, S., Hawkins, R., Yang, S., Shah, D., Hu, J., & Rogers, T. (2024) · Findings of the Association for Computational Linguistics: NAACL 2024 · doi:10.18653/v1/2024.findings-naacl.211 · Cited by 85
Sexual Selection and Humor in Courtship
Hall, J. A. (2015) · Evolutionary Psychology · doi:10.1177/1474704915598918 · Cited by 45
“Reliability was measured using Krippendorf’s a MACRO (Hayes & Krippendorf, 2007).”
MethodCognitive Discourse Analysis: An Introduction
Tenbrink, T. (2020) · Cited by 45
Grassa, R., Moumen, N., & Hussainey, K. (2020) · Pacific-Basin Finance Journal · doi:10.1016/j.pacfin.2020.101327 · Cited by 46
“Fourth, we compared the risk disclosure grid coded by both scholars (the main researcher and the independent reviewer) to ascertain if there were any significant discrepancies through the Krippendorff’s alpha reliability test (Hayes and Krippendorff, 2007; Krippendorff, 2010).”
MethodAffectionate communication and health: A meta-analysis
Hesse, C., Floyd, K., Rains, S. A., Mikkelson, A. C., Pauley, P. M., Woo, N. T., Custer, B. E., & Duncan, K. L. (2020) · Communication Monographs · doi:10.1080/03637751.2020.1805480 · Cited by 45
“Based on duplicate coding of 20 percent of the sample, we calculated interrater reliability using Krippendorff’s alpha (Hayes & Krippendorff, 2007).”
MethodBa, S., Zhan, Y., Huang, L., & Lu, G. (2025) · British Journal of Educational Technology · doi:10.1111/bjet.13605 · Cited by 52
Pettigrew, J., Miller-Day, M. A., Shin, Y., Hecht, M. L., Krieger, J. L. R., & Graham, J. A. (2012) · American Journal of Community Psychology · doi:10.1007/s10464-012-9539-1 · Cited by 46
“Inter-coder reliability (Krippendorff’s alpha (see Hayes and Krippendorff 2007) of .80 was reached for quantitative ratings of practice videos and coders participated in recording qualitative descriptions of at least two videos with trainers prior to the outset of coding selected videos.”
All Student Samples Differ: On Participant Selection in Communication Science
Meltzer, C. E., Naab, T., & Daschmann, G. (2012) · Communication Methods and Measures · doi:10.1080/19312458.2012.732625 · Cited by 48
“We calculated Krippendorff’s alpha (Hayes & Krippendorff, 2007) for intercoder and intracoder reliability.”
MethodCommunicating Sustainability Online: An Examination of Corporate, Nonprofit, and University Websites
Ott, H. K., Wang, R., & Bortree, D. S. (2016) · Mass Communication & Society · doi:10.1080/15205436.2016.1204554 · Cited by 45
“Coder reliability was calculated by using Krippendorff’s alpha (Hayes & Krippendorff, 2007).”
MethodStudent evaluation of teaching and matters of reliability
Clayson, D. E. (2017) · Assessment & Evaluation in Higher Education · doi:10.1080/02602938.2017.1393495 · Cited by 45
“When ascertaining validity is the primary issue, Hayes and Krippendorff (2007) state that reproducibility by raters is the strongest and most feasible test of reliability.”
BackgroundMethodCetto, A., Klier, M., Richter, A., & Zolitschka, J. F. (2018) · Computer Networks · doi:10.1016/j.comnet.2018.02.012 · Cited by 52
“After coding further 100 messages separately, Krippendorff alpha [82] as standard reliability measure for coding data [83] was used to estimate the inter-rater reliability and to ensure a consistent coding approach and reliable results.”
MethodWhy are many businesses instilling a DevOps culture into their organization?
Díaz, J., López-Fernández, D., Pérez, J., & González-Prieto, Á. (2021) · Empirical Software Engineering · doi:10.1007/s10664-020-09919-3 · Cited by 45
Crisis information, communication strategies and customer complaint behaviours: the case of COVID-19
Liu, X., Fu, X., Chang, H., & Li, Z. (2021) · Tourism Review · doi:10.1108/tr-01-2021-0004 · Cited by 45
A comprehensive framework of brand name classification
Arora, S. V., Kalro, A. D., & Sharma, D. (2015) · Journal of Brand Management · doi:10.1057/bm.2015.8 · Cited by 45
Rieger, A., Draws, T., Theune, M., & Tintarev, N. (2021) · Proceedings of the 32st ACM Conference on Hypertext and Social Media · doi:10.1145/3465336.3475101 · Cited by 48
Spatial Strategies in the Description of Complex Configurations
Tenbrink, T., Coventry, K. R., & Andonova, E. (2011) · Discourse Processes · doi:10.1080/0163853x.2010.549452 · Cited by 44
“Levels of agreement were very good or good: Krippendorff’s alpha (computed using the SPSS macro reported in Hayes & Krippendorff, 2007) for coding of content (orientation and location information) was 0.83, for choice of perspective 0.81, for choice of relatum in location descriptions 0.75, and for…”
MethodInfluentialNews Media Framing of Negative Campaigning
Pedersen, R. T. (2014) · Mass Communication & Society · doi:10.1080/15205436.2013.858749 · Cited by 44
“The coding for focus on negative=positive campaigning had an acceptable level of reliability with a Krippendorff’s a¼ 0.73, and reliability for coding of dominant frame had a high level of reliability, with a¼ 0.89 (Hayes & Krippendorff, 2007; Krippendorff, 2004).”
MethodInfluentialGuffler, K., & Wagner, U. (2017) · Peace and Conflict Journal of Peace Psychology · doi:10.1037/pac0000264 · Cited by 44
What reduces prejudice in the real world? A meta‐analysis of prejudice reduction field experiments
Hsieh, W., Faulkner, N., & Wickes, R. L. (2021) · British Journal of Social Psychology · doi:10.1111/bjso.12509 · Cited by 51
Cross-cultural variation in cooperation: A meta-analysis.
Spadaro, G., Graf, C., Jin, S., Arai, S., Inoue, Y., Lieberman, E., Rinderu, M. I., Yuan, M., Lissa, C. J. V., & Balliet, D. P. (2022) · Journal of Personality and Social Psychology · doi:10.1037/pspi0000389 · Cited by 56
Femvertising practices on social media: a comparison of luxury and non-luxury brands
Michaelidou, N., Siamagka, N., Hatzithomas, L., & Chaput, L. (2022) · Journal of Product & Brand Management · doi:10.1108/jpbm-09-2020-3074 · Cited by 51
Bridging the Gap: A Survey on Integrating (Human) Feedback for Natural Language Generation
Fernandes, P., Madaan, A., Liu, E., Farinhas, A., Martins, P. H., Bertsch, A., de Souza, J. G. C., Zhou, S., Wu, T., Neubig, G., et al. (2023) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00626 · Cited by 94
“Inter-rater reliability metrics, such as Cohen’s Kappa, Fleiss’ Kappa, or Krippendorff’s alpha, can assess anno-tator agreement (Hayes and Krippendorff, 2007; Fleiss, 1971; Cohen, 1960).”
Buff, Tough, and Rough: Representations of Muscularity in Action Motion Pictures
Morrison, T. G., & Halton, M. (2009) · The Journal of Men s Studies · doi:10.3149/jms.1701.57 · Cited by 44
“%) of the films coded using Krippendorff’s alpha (Hayes & Krippendorff, 2007).”
Method
Metadata
- Authors
- Hayes, A. F., & Krippendorff, K.
- Affiliations
- The Ohio State University (US)
- University of Pennsylvania (US)
- Venue
- Communication Methods and Measures
- Publisher
- Informa UK Limited
- Published
- 2007-04
- Volume
- 1
- Issue
- 1
- Pages
- 77-89
- ISSN
- 1931-2458 (ISSN-L)
- 1931-2466
- OpenAlex
- W2061504941
- Cited by
- 4,225
- References
- 30
- Citations by year
- 2026: 284 · 2025: 278 · 2024: 280 · 2023: 289 · 2022: 331 · 2021: 379 · 2020: 369 · 2019: 296 · 2018: 260 · 2017: 300 · 2016: 248 · 2015: 205 · 2014: 206 · 2013: 150 · 2012: 107
- FWCI
- 50.07
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Communication Methods and Measures (publishedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Reliability and Agreement in Measurement
- Social Sciences › General Social Sciences › Computational and Text Analysis Methods
- Social Sciences › Decision Sciences › Management Science and Operations Research › Psychometric Methodologies and Testing
- Keywords
- Krippendorff's alpha, inter-rater reliability, content analysis, coding reliability, reliability coefficients, observer agreement, SAS macro, missing data
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
