The DOI matches its Crossref record; 30 of the 30 references it cites were found in a registry or an index.
Answering the Call for a Standard Reliability Measure for Coding Data
Abstract
In content analysis and similar methods, data are typically generated by trained human observers who record or transcribe textual, pictorial, or audible matter in terms suitable for analysis. Conclusions from such data can be trusted only after demonstrating their reliability. Unfortunately, the content analysis literature is full of proposals for so-called reliability coefficients, leaving investigators easily confused, not knowing which to choose. After describing the criteria for a good measure of reliability, we propose Krippendorff's alpha as the standard reliability measure. It is general in that it can be used regardless of the number of observers, levels of measurement, sample sizes, and presence or absence of missing data. To facilitate the adoption of this recommendation, we describe a freely available macro written for SPSS and SAS to calculate Krippendorff's alpha and illustrate its use with a simple example.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“We calculate Krippendorff's alpha (Hayes and Krippendorff, 2007) to measure inter-rater reliability (see Table 3).”
From LLMs cannot find reasoning errors, but can correct them given the error location (2024) · cites this paper as Hayes and Krippendorff, 2007
In the citing paper
“Based on Krippendorff's alpha (α = 0.92), the level of agreement between coders was high and well above the threshold of 0.70 (Hayes & Krippendorff, 2007).”
From Collaboration, Coordination, and Cooperation Among Organizations: Establishing the Distinctive Meanings of These Terms Through a Systematic Literature Review (2020) · cites this paper as Hayes & Krippendorff, 2007
In the citing paper
“we calculated the inter-coder reliability based on 25% of all data units and derived a kappa value of .96, which reflects excellent reliability.”
In the citing paper
“To assess the inter-annotator agreement of stage 1, we report Krippendorf's α (Hayes and Krippendorff, 2007) for each dimension for the 135 articles after training and before any discussion/adjudication: economic (0.44), foreign (0.68), social (0.39).”
From Political Ideology and Polarization: A Multi-dimensional Approach (2022) · cites this paper as Hayes and Krippendorff, 2007
In the citing paper
“This family of metrics does not correct for chance (Cohen, 1960;Hayes and Krippendorff, 2007) and overstates discriminative ability (Guerdan et al., 2025;Han et al., 2025;Collot et al., 2025).”
From Reliability without Validity: A Systematic, Large-Scale Evaluation of LLM-as-a-Judge Models Across Agreement, Consistency, and Bias (2026) · cites this paper as Hayes and Krippendorff, 2007
Citations
Cites
Coefficient Alpha and the Internal Structure of Tests
Cronbach, L. J. (1951) · Psychometrika · doi:10.1007/bf02310555 · Cited by 43,931
A Coefficient of Agreement for Nominal Scales
Cohen, J. (1960) · Educational and Psychological Measurement · doi:10.1177/001316446002000104 · Cited by 42,398
An Introduction to the Bootstrap
Efron, B., & Tibshirani, R. J. (1994) · doi:10.1201/9780429246593 · Cited by 40,500
Nonparametric Statistics for the Behavioral Sciences
Guitton, H., & Siegel, S. (1958) · Revue économique · doi:10.2307/3498751 · Cited by 35,279
Content Analysis: An Introduction to its Methodology.
Shelley, M., & Krippendorff, K. (1984) · Journal of the American Statistical Association · doi:10.2307/2288384 · Cited by 24,351
Self-citationStatistical Methods for Rates and Proportions
Fleiss, J. L., Levin, B., & Paik, M. C. (2003) · Wiley Series in Probability and Statistics · doi:10.1002/0471445428 · Cited by 18,646
Measuring nominal scale agreement among many raters.
Fleiss, J. L. (1971) · Psychological Bulletin · doi:10.1037/h0031619 · Cited by 8,733
Content Analysis for the Social Sciences and Humanities.
Parker, E. B., & Holsti, O. R. (1970) · American Sociological Review · doi:10.2307/2093233 · Cited by 4,553
Content Analysis in Mass Communication: Assessment and Reporting of Intercoder Reliability
Lombard, M., Snyder-Duch, J., & Bracken, C. C. (2002) · Human Communication Research · doi:10.1111/j.1468-2958.2002.tb00826.x · Cited by 3,068
An Introduction to the Bootstrap
Grimshaw, S. D., Efron, B., & Tibshirani, R. J. (1995) · Technometrics · doi:10.2307/1269918 · Cited by 2,281
Research works printed but not linked: 1
- 10.1111/j.1468-2958.2004.tb00738.x · DOI · Paper page
Cited by
Sorted by citations
Items 301–400 of 1,013
Enflo, L., Sundberg, J., Romedahl, C., & McAllister, A. M. (2013) · Journal of Speech Language and Hearing Research · doi:10.1044/1092-4388(2013/12-0040 · Cited by 90
Using Twitter as a means of coping with emotions and uncontrollable crises
Brummette, J., & Sisco, H. F. (2014) · Public Relations Review · doi:10.1016/j.pubrev.2014.10.009 · Cited by 90
A Systematic Review of the Outcome of Child Abuse in Long-Term Care
Carr, A., Duff, H., & Craddock, F. (2018) · Trauma Violence & Abuse · doi:10.1177/1524838018789154 · Cited by 91
Physics Teachers’ Challenges in Using History and Philosophy of Science in Teaching
Henke, A., & Höttecke, D. (2014) · Science & Education · doi:10.1007/s11191-014-9737-3 · Cited by 93
“71 (Hayes and Krippendorff 2007).”
How adolescents develop and implement their ideas? On self-regulation of creative action
Zielińska, A., Lebuda, I., Ivčević, Z., & Karwowski, M. (2022) · Thinking Skills and Creativity · doi:10.1016/j.tsc.2022.100998 · Cited by 89
A Threatening Space?: Stigmatization and the Framing of Autism in the News
Holton, A. E., Farrell, L. C., & Fudge, J. L. (2014) · Communication Studies · doi:10.1080/10510974.2013.855642 · Cited by 102
“To determine intercoder reliability, Krippendorff’s alpha, which is a vigorous measure of reliability for two or more coders for any sample size (Hayes & Krippendorff, 2007), was used.”
MethodA Qualitative Study of the Benefits and Costs of Logging From Developers’ Perspectives
Li, H., Shang, W., Adams, B., Sayagh, M., & Hassan, A. E. (2020) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2020.2970422 · Cited by 87
“In this work, we use Krippendorff’s α [10, 18] to measure the inter-coder agreement of our coding results.”
BackgroundMethodFacilitating Communicative Ability of EFL Learners via High-Immersion Virtual Reality.
Yang, F.-C. O., Lo, F.-Y., Hsieh, J. C., & Wu, W. V. (2020) · DOAJ (DOAJ: Directory of Open Access Journals) · Cited by 87
Cheng, C., Wang, H.-Y., & Ebrahimi, O. V. (2021) · Frontiers in Psychiatry · doi:10.3389/fpsyt.2021.626197 · Cited by 89
“Inter-rater agreement was evaluated using Krippendorff alpha coefficients (43), and the results showed no discrepancies because no subjective codings were required (Krippendorff alpha= 100%).”
MethodKumar, K., Kumari, R., Nandy, M., Sarim, M., & Kumar, R. (2022) · Management of Environmental Quality An International Journal · doi:10.1108/meq-08-2021-0196 · Cited by 100
A Review of Information Systems Research on Online Social Networks
Berger, K., Klier, J., Klier, M., & Probst, F. (2014) · Communications of the Association for Information Systems · doi:10.17705/1cais.03508 · Cited by 89
Mourão, R. R., Brown, D. K., & Sylvie, G. (2021) · Journalism · doi:10.1177/1464884918778722 · Cited by 89
“Krippendorff’s alpha scores were used to calculate inter-coder reliability (Hayes and Krippendorff, 2007), using ReCal (Freelon, 2013).”
MethodSustainable Supply Chain Management, Performance Measurement, and Management: A Review
Kumar, A., Shrivastav, S. K., Shrivastava, A. K., Panigrahi, R. R., Mardani, A., & Cavallaro, F. (2023) · Sustainability · doi:10.3390/su15065290 · Cited by 109
“A reliability A reliability test for content analysis has been carried out using the method proposed by [16] in SPSS. 16] in SPSS.”
MethodAnCora-CO: Coreferentially annotated corpora for Spanish and Catalan
Recasens, M. V., & Martí, M. A. (2009) · Language Resources and Evaluation · doi:10.1007/s10579-009-9108-x · Cited by 86
“Krippendorff s a was computed with the freely available KALPHA macro written for SPSS (Hayes and Krippendorff 2007), yielding the following results: a = .”
MethodThe customer as enabler of value (co)-creation in the solution business
Petri, J., & Jacob, F. (2016) · Industrial Marketing Management · doi:10.1016/j.indmarman.2016.03.009 · Cited by 86
Exploring Users' Appropriateness as a Proxy for Experts When Screening New Product/Service Ideas
Magnusson, P., Wästlund, E., & Netz, J. (2014) · Journal of Product Innovation Management · doi:10.1111/jpim.12251 · Cited by 85
“Thus, αC is a good measure of the reliability of aggregated measures based on many observers, while αK is better if the objective is a measure where absolute agreement is of interest (Hayes and Krippendorff, 2007).”
BackgroundThe love of money results in objectification
Wang, X., & Krumhuber, E. G. (2017) · British Journal of Social Psychology · doi:10.1111/bjso.12158 · Cited by 77
“Inter-rater agreement was very high, j = .96 (Hayes & Krippendorff, 2007), demonstrating good reliability.”
BackgroundMarker versus Markerless Augmented Reality. Which Has More Impact on Users?
Brito, P. Q., & Stoyanova, J. (2017) · International Journal of Human-Computer Interaction · doi:10.1080/10447318.2017.1393974 · Cited by 92
“The Krippendorff Alpha reliability measure (Hayes & Krippendorff, 2007) of the judgments for 35% of the data commonly coded ranged from 0.81 to 0.97.”
MethodInfluentialZhang, Y., Sun, J., Yang, Z., & Wang, Y. (2018) · International Journal of Project Management · doi:10.1016/j.ijproman.2018.09.003 · Cited by 83
“Krippendorff's alpha was used to assess inter-rater reliability (Hayes and Krippendorff, 2007).”
MethodEnglishBot: An AI-Powered Conversational System for Second Language Learning
Ruan, S., Jiang, L., Xu, Q., Liu, Z., Davis, G. M., Brunskill, E., & Landay, J. A. (2021) · ACM International Conference on Intelligent User Interfaces (IUI) · doi:10.1145/3397481.3450648 · Cited by 113
“Inter-rater reliability between the two graders was measured to be 0.722 using Krippendorff’s alpha [19, 36].”
MethodInfluentialScience blogging: an exploratory study of motives, styles, and audience reactions
Mahrt, M., & Puschmann, C. (2014) · Journal of Science Communication · doi:10.22323/2.13030205 · Cited by 84
Natural Sentences as Valid Units for Coded Political Texts
Däubler, T., Benoit, K. R., Mikhaylov, S., & Laver, M. (2012) · British Journal of Political Science · doi:10.1017/s0007123412000105 · Cited by 83
Cardoso, L., Dias, F., de Araújo, A. F., & Marques, I. A. (2018) · Annals of Tourism Research · doi:10.1016/j.annals.2018.11.001 · Cited by 92
Bornschein, R., Schmidt, L., & Maier, E. (2020) · Journal of Public Policy & Marketing · doi:10.1177/0743915620902143 · Cited by 101
Identifying and Measuring Annotator Bias Based on Annotators’ Demographic Characteristics
Kuwatly, H. A., Wich, M., & Groh, G. (2020) · Workshop on Online Abuse and Harms (WOAH) · doi:10.18653/v1/2020.alw-1.21 · Cited by 102
How to Conduct a Deductive Content Analysis in Counseling Research
McKibben, W. B., Cade, R., Purgason, L. L., & Wahesh, E. (2020) · Counseling Outcome Research and Evaluation · doi:10.1080/21501378.2020.1846992 · Cited by 88
“Freelon’s (2013) ReCal2 (for 2 coders) and ReCal 3 (for 3þ coders) provide multiple IRR statistics, and Hayes and Krippendorff (2007) created a macro for SPSS that calculates a with CIs.”
BackgroundExamining the extent of and drivers for materiality assessment disclosures in sustainability reports
Farooq, M. B., Zaman, R., Sarraj, D., & Khalid, F. (2021) · Sustainability Accounting Management and Policy Journal · doi:10.1108/sampj-04-2020-0113 · Cited by 84
The Effects of Message Framing and Visual Image on Persuasion
Seo, K., Dillard, J. P., & Shen, F. (2013) · Communication Quarterly · doi:10.1080/01463373.2013.822403 · Cited by 90
“When the reliability of the emotion versus cognition coding was estimated using Krippendorff’s aK (Hayes & Krippendorff, 2007), the resulting values were .62 for flossing and .62 for sunscreen.”
MethodCommenting on Health: A Framing Analysis of User Comments in Response to Health Articles Online
Holton, A. E., Lee, N., & Coleman, R. (2014) · Journal of Health Communication · doi:10.1080/10810730.2013.837554 · Cited by 86
How Far are We from Fully Automatic High Quality Grammatical Error Correction?
Bryant, C., & Ng, H. T. (2015) · doi:10.3115/v1/p15-1068 · Cited by 83
Information, Community, and Action on Sina-Weibo: How Chinese Philanthropic NGOs Use Social Media
Zhou, H., & Pan, Q. (2016) · VOLUNTAS International Journal of Voluntary and Nonprofit Organizations · doi:10.1007/s11266-016-9685-4 · Cited by 83
“The triple coding has a Kalpha of 0.8619 (with only 0.0341 probability to fall below 0.80), indicating good intercoder reliability (Hayes and Krippendorff 2007).”
BackgroundInfluentialJelsma, J. G. M., Mertens, V., Forsberg, L., & Forsberg, L. (2015) · Contemporary Clinical Trials · doi:10.1016/j.cct.2015.05.001 · Cited by 82
“For this reason, a specific KALPHA macro for SPSS [48], which can be downloaded here http://www.”
BackgroundMethodInfluentialWhen Journalists Tweet: Disclosure, Participatory, and Personal Transparency
Hedman, U. (2016) · Social Media + Society · doi:10.1177/2056305115624528 · Cited by 81
“When journalists adapt to the norms and practices of blogging, Hayes et al. (2007) found that they could contribute to journalistic accountability by explaining the rationale behind the news and humanizing the reporter, “allowing him or her to talk about the experience of covering a particular story or about personal…”
BackgroundMethodInfluentialLike Trainer, Like Bot? Inheritance of Bias in Algorithmic Content Moderation
Binns, R., Veale, M., Kleek, M. V., & Shadbolt, N. (2017) · Social Informatics · doi:10.1007/978-3-319-67256-4_32 · Cited by 182
“Considering the dataset before training classifiers, we measured agreement on toxicity within male and female annotators with Krippendorff’s alpha (due to amenability to missing data, see [20])) 3 , and average measures of how toxicity scores per comment differs between male and female annotators.”
MethodFortune 100 companies’ Facebook strategies: corporate ability versus social responsibility
Kim, S., Kim, S., & Sung, K. H. (2014) · Journal of Communication Management · doi:10.1108/jcom-01-2012-0006 · Cited by 81
Skills, knowledge and senior managers’ demonstrations of safety commitment
Fruhen, L. S., Mearns, K. J., Flin, R. H., & Kirwan, B. (2013) · Safety Science · doi:10.1016/j.ssci.2013.08.024 · Cited by 93
Liu, Y., & Merritt, D. H. (2018) · Children and Youth Services Review · doi:10.1016/j.childyouth.2018.03.019 · Cited by 89
Awan, U., & Sroufe, R. P. (2019) · International Journal of Innovation Management · doi:10.1142/s1363919620500498 · Cited by 80
Chan, M., Hu, P., & Mak, M. K. F. (2020) · Journalism & Mass Communication Quarterly · doi:10.1177/1077699020961519 · Cited by 89
“Intercoder reliability checks for every variable of every case were conducted incrementally after every set of 50 articles using Krippendorff’s alpha (Hayes & Krippendorff, 2007).3 Any disagreement among the three coders was deliberated upon to reach a consensus before moving on to the next set of…”
Methodkrippendorffsalpha: An R Package for Measuring Agreement Using Krippendorff's Alpha Coefficient
Hughes, J. (2021) · The R Journal · doi:10.32614/rj-2021-046 · Cited by 80
“Situating Krippendorff’s Alpha among statistical procedures Since Krippendorff’s α is defined in terms of discrepancies (Krippendorff, 2013), at first glance one might conclude, erroneously, that α is a measure of dis -agreement and so answers the wrong question.”
MethodInfluentialFindings of the IWSLT 2022 Evaluation Campaign
Anastasopoulos, A., Barrault, L., Bentivogli, L., Boito, M. Z., Bojar, O., Cattoni, R., Currey, A., Dinu, G., Duh, K., Elbayad, M., et al. (2022) · International Conference on Spoken Language Translation (IWSLT) · doi:10.18653/v1/2022.iwslt-1.10 · Cited by 82
“Inter-annotator agreement as measured by the Krippendorff’s ↵ coefficient (Hayes and Krippendorff, 2007) was high, with an average ↵ of 0.”
MethodInfluentialDouma, R. K., Soer, R., Krijnen, W. P., Reneman, M. F., & van der Schans, C. P. (2014) · BMC Sports Science Medicine and Rehabilitation · doi:10.1186/2052-1847-6-10 · Cited by 80
“To answer the primary objective of this study, reliability of the three repeated measurements, the intraclass correlation coefficients (ICC) two-way random effects model including lower and upper confidence limits (LCL and UCL), as well as the limits of agreement (LOA) were calculated [18].”
MethodPlanetary Boundaries and Sustainability Indicators. A Survey of Corporate Reporting Boundaries
Antonini, C., & Larrínaga, C. (2017) · Sustainable Development · doi:10.1002/sd.1667 · Cited by 79
“Furthermore, a key factor in reliability is the agreement among independent observers (Hayes and Krippendorff, 2007): 27% of the sustainability reports were analysed by the authors separately to test the research instrument, with any discrepancy about the coding procedure being discussed and agreed.”
BackgroundRetweeting Risk Communication: The Role of Threat and Efficacy
Vos, S. C., Sutton, J. N., Yu, Y., Renshaw, S. L., Olson, M. K., Gibson, C. B., & Butts, C. T. (2018) · Risk Analysis · doi:10.1111/risa.13140 · Cited by 82
“Reliability was assessed with Krippendorff’s alpha, which was calculated using the KALPHA macro for SPSS (Hayes & Krippendorff, 2007; Krippendorff, 2004).”
MethodLarrinaga, C., Rossi, A., Luque-Vílchez, M., & Núñez-Nickel, M. (2018) · Journal of Business Ethics · doi:10.1007/s10551-018-4014-z
“80), showing an acceptable level of agreement between the two coders (Hayes and Krippendorff 2007).”
BackgroundMethodInfluentialWuestefeld, A., Fuermaier, A. B. M., Bernardo-Filho, M., de Sá-Caputo, D. D. C., Rittweger, J., Schöenau, E., Stark, C., Marin, P. J., Seixas, A., Judex, S., et al. (2020) · PLoS ONE · doi:10.1371/journal.pone.0235905 · Cited by 81
Bojić, L., Zagovora, O., Zelenkauskaitė, A., Vuković, V., Čabarkapa, M., Jerković, S. V., & Jovančević, A. (2025) · Scientific Reports · doi:10.1038/s41598-025-96508-3 · Cited by 119
“To address RQ1, which focused on assessing the inter-rater reliability among LLMs and human annotators, Krippendorff’s alpha was calculated for both human annotators and LLMs in each dimension 37 .”
InfluentialCabana, F., Boissy, P., Tousignant, M., Moffet, H., Corriveau, H., & Dumais, R. (2010) · Telemedicine Journal and e-Health · doi:10.1089/tmj.2009.0106 · Cited by 77
Cervical Spinal Cord, Root, and Bony Spine Injuries
Hindman, B. J., Palecek, J. P., Posner, K. L., Traynelis, V. C., Lee, L. A., Sawin, P. D., Tredway, T. L., Todd, M. M., & Domino, K. B. (2011) · Anesthesiology · doi:10.1097/aln.0b013e3182104859 · Cited by 79
“001 Immediately postprocedure 28 (65) — 26 (79) 1 (11) — Within 24 h postprocedure 4 (9) — 3 (9) 1 (11) — 24 h postprocedure 5 (12) — 0 (0) 5 (56) — Cannot determine 6 (14) — 4 (12) 2 (22) —”
Evaluation of dysphagia by novel real-time MRI
Olthoff, A., Carstens, P.-O., Zhang, S., von Fintel, E., Friede, T., Lotz, J., Frahm, J., & Schmidt, J. (2016) · Neurology · doi:10.1212/wnl.0000000000003337 · Cited by 79
“The interrater agreement was assessed by the Krippendorff a coefficient.(20) Associations between variables were assessed by Pearson correlation coefficients.”
MethodSiegert, I., Böck, R., & Wendemuth, A. (2013) · Journal on Multimodal User Interfaces · doi:10.1007/s12193-013-0129-9 · Cited by 82
“Another advantage of α is its greater reliability utilizing two or more coders and its robustness dealing with incomplete data [36].”
BackgroundUniversals and cultural diversity in the expression of gratitude
Floyd, S., Rossi, G., Baranova, J., Blythe, J., Dingemanse, M., Kendrick, K. H., Zinken, J., & Enfield, N. J. (2018) · Royal Society Open Science · doi:10.1098/rsos.180391 · Cited by 88
“The two specific questions focused on in this study were reliably answered for the categories considered (see below), achieving a Krippendorf’s α [34] of greater than 0.75 and greater than 0.68, respectively.”
InfluentialRidings, C. M., & Wasko, M. M. (2010) · Journal of the Association for Information Systems · doi:10.17705/1jais.00220 · Cited by 78
The moderating effect of firm size on relational capital and firm performance
Corvino, A., Caputo, F., Pironti, M., Doni, F., & Martini, S. B. (2019) · Journal of Intellectual Capital · doi:10.1108/jic-03-2019-0044 · Cited by 78
Tenbrink, T. (2014) · Language and Cognition · doi:10.1017/langcog.2014.19 · Cited by 80
“Where applicable and feasible, inter-coder reliability should be tested statistically, for example using Krippendorff's Alpha (using the tool provided by Hayes & Krippendorff, 2007, for the SPSS/PASW statistics software package).”
MethodQuantitative text analysis and the study of EU lobbying and interest groups
Bunea, A., & Ibenskas, R. (2015) · European Union Politics · doi:10.1177/1465116515577821 · Cited by 77
“for the Krippendorff’s Alpha coefficient using the SPSS algorithm designed for this, based on Hayes and Krippendorff (2007). 7.”
MethodBirds of a feather flock together? Party leaders on Twitter during the 2013 Norwegian elections
Larsson, A. O., & Ihlen, Ø. (2015) · European Journal of Communication · doi:10.1177/0267323115595525 · Cited by 77
“SPSS was then used to perform a series of descriptive statistical analyses, presented in Figures 1 and 3.”
MethodEvaluation of ergonomic working conditions among standing sewing machine operators in Sri Lanka
Nagaraj, T. S., Jeyapaul, R., & Mathiyazhagan, K. (2019) · International Journal of Industrial Ergonomics · doi:10.1016/j.ergon.2019.01.006 · Cited by 77
Neto, G. C. D. O., Silva, A. D. C., & Filho, M. G. (2022) · International Journal of Environmental Science and Technology · doi:10.1007/s13762-022-04234-4 · Cited by 78
“Thus, the data were checked independently by two researchers to minimize errors and bias by the researcher (Hayes and Krippendorff 2007) and filter the selected research.”
BackgroundMethodPresseau, J., Sniehotta, F. F., Francis, J., & Campbell, N. C. (2009) · Implementation Science · doi:10.1186/1748-5908-4-77 · Cited by 85
Exploring social and cognitive presences in communities of inquiry to perform higher cognitive tasks
Morueta, R. T., López, P. M., Gómez, Á. H., & Harris, V. W. (2016) · The Internet and Higher Education · doi:10.1016/j.iheduc.2016.07.004 · Cited by 78
“The statistical package macro (KALPHA) by Hayes and Krippendorff (2007) was used for the calculation of the interrater and intra-rater reliability coefficient Krippendorff's alpha.”
MethodHumor in romantic relationships: A meta‐analysis
Hall, J. A. (2017) · Personal Relationships · doi:10.1111/pere.12183 · Cited by 76
Non-invasive Fetal ECG Signal Quality Assessment for Multichannel Heart Rate Estimation
Andreotti, F., Gräßer, F., Malberg, H., & Zaunseder, S. (2017) · IEEE Transactions on Biomedical Engineering · doi:10.1109/tbme.2017.2675543 · Cited by 78
“To cope with those “less/more serious” classification errors, the Krippendorff’s alpha coefficient (α) [38] was suggested.”
BackgroundIndividual variation in gender/sex category definitions.
Schudson, Z. C., Beischel, W. J., & van Anders, S. M. (2019) · Psychology of Sexual Orientation and Gender Diversity · doi:10.1037/sgd0000346 · Cited by 76
Visual frames of migrants and refugees in the main Western European media
Amores, J. J., Calderón, C. A., & Stanek, M. (2019) · Economics & Sociology · doi:10.14254/2071-789x.2019/12-3/10 · Cited by 76
“Reliability was checked with the KALPHA (Kippendorf's alpha) macro for SPSS, as it is the most recommended reliability measure (Hayes and Krippendorff, 2007), and all variables resulted in rates greater than 0.67.”
MethodInfluentialBanović, M., Krystallis, A., Guerrero, L., & Reinders, M. J. (2016) · Food Research International · doi:10.1016/j.foodres.2016.07.010 · Cited by 76
Learning in the wild: coding for learning and practice on Reddit
Haythornthwaite, C., Kumar, P., Gruzd, A. A., Gilbert, S., Valle, M. E. D., & Paulin, D. (2018) · Learning Media and Technology · doi:10.1080/17439884.2018.1498356 · Cited by 85
“According to Hayes and Krippendorff (2007), human coders as observers are trained to make judgments of kind (e.g., what category does this unit belong to?), magnitude (e.g., how pronounced is the unit attribute?), or frequency (how often is it occurring?). It is likely our coders differed in their judgments of kind, m…”
BackgroundDesign principles for engaging and retaining virtual citizen scientists
Wald, D. M., Longo, J., & Dobell, A. R. (2015) · Conservation Biology · doi:10.1111/cobi.12627 · Cited by 75
Modave, F. P., Bian, J., Leavitt, T., Bromwell, J., Harris, C. S., & Vincent, H. (2015) · JMIR mhealth and uhealth · doi:10.2196/mhealth.4669 · Cited by 80
“Inter-rater reliability was assessed using the Krippendorff alpha coefficient [24].”
Parker, S. K., Andrei, D. M., & den Broeck, A. V. (2019) · Journal of Applied Psychology · doi:10.1037/apl0000383 · Cited by 85
“Sufficient interrater agreement (computed with a syntax by Hayes & Krippendorff, 2007) was achieved across all three dimensions: strategy ( .78; 95% CI [0.61, 0.92]), reason ( .79; 95% CI [0.66, 0.92]), and job titles ( .95; 95% CI [0.88, 1.00]), therefore the remaining responses were coded by one…”
MethodInfluentialiElectrodes: A Comprehensive Open-Source Toolbox for Depth and Subdural Grid Electrode Localization
Blenkmann, A. O., Phillips, H. N., Princich, J. P., Rowe, J. B., Bekinschtein, T. A., Muravchik, C. H., & Kochen, S. S. (2017) · Frontiers in Neuroinformatics · doi:10.3389/fninf.2017.00014 · Cited by 77
“Krippendorff’s alpha has the capability to accept several raters and ordinal ratings (Hayes and Krippendorff, 2007).”
BackgroundFlaherty, S. J., McCarthy, M. B., Collins, A., & McAuliffe, F. M. (2017) · Public Health Nutrition · doi:10.1017/s1368980017002889 · Cited by 76
“Krippendorff’s alpha was used to evaluate inter-rater reliability((62)).”
MethodNurse–physician collaboration: a meta-analytical investigation of survey scores
Sollami, A., Caricati, L., & Sarli, L. (2014) · Journal of Interprofessional Care · doi:10.3109/13561820.2014.955912 · Cited by 74
“Three judges (C. F., G. M. and A. S.) independently evaluated each survey with an inter-rater agreement (Krippendorff’s alpha, see Hayes & Krippendorff, 2007) ranging between 0.82 and 1, indicating strong agreement among judges.”
MethodInfluentialInter-Annotator Agreement in Sentiment Analysis: Machine Learning Perspective
Bobicev, V., & Sokolova, M. (2017) · International conference Recent advances in natural language processing · doi:10.26615/978-954-452-049-6_015 · Cited by 78
The Role of Figurative Complexity in the Comprehension and Appreciation of Advertisements
Sobrino, P. P., Littlemore, J., & Houghton, D. (2018) · Applied Linguistics · doi:10.1093/applin/amy039 · Cited by 74
Hsieh, J. C., & Lee, J. S. (2021) · Computer Assisted Language Learning · doi:10.1080/09588221.2021.1969410 · Cited by 97
“With regard to students’ digital storytelling performances, inter-rater reliability was measured with Krippendorff ’s alpha at .85, considered the norm for good reliability (Hayes & Krippendorff, 2007).”
MethodFeldman, L. (2013) · Mass Communication & Society · doi:10.1080/15205436.2012.735742 · Cited by 75
“Intercoder reliability, calculated using Krippendorff’s alpha for ratio-level data (Hayes & Krippendorff, 2007), was .96.”
MethodPollet, T. V., Stulp, G., Henzi, S. P., & Barrett, L. (2015) · American Journal of Primatology · doi:10.1002/ajp.22405 · Cited by 75
“In field settings, standard inter-rater reliability procedures (e.g. Cohen’s kappa, Krippendorf alpha [Hayes & Krippendorff, 2007] often are not feasible because field assistants rarely collect exactly the same data on the same animals at the same time (i.e., they usually do not overlap in their…”
MethodJournalists as Knowledge Brokers
Gesualdo, N., Weber, M. S., & Yanovitzky, I. (2019) · Journalism Studies · doi:10.1080/1461670x.2019.1632734 · Cited by 73
“Inter-coder reliability was assessed using Krippendorff’s alpha (Krippendorff 2013), which handles multiple coders; nominal, ordinal, interval, ratio, and other metrics; missing data; and small sample sizes (Hayes and Krippendorff 2007).”
MethodEitel, A. (2015) · Learning and Instruction · doi:10.1016/j.learninstruc.2015.10.003 · Cited by 83
“A second rater scored 23% of the open questions assessing recall and troubleshooting performance ( from 18 out of 79 participants ) , and Krippendorff ' s alpha was calculated as the measure of inter - rater agreement ( Hayes & Krippendorff , 2007 ) .”
MethodPadilla‐Walker, L. M., Coyne, S. M., Fraser, A. M., & Stockdale, L. (2013) · Journal of Communication · doi:10.1111/jcom.12022 · Cited by 82
Confessing their Crime: Factors Influencing the Offender’s Decision to Confess to the Police
Deslauriers‐Varin, N., Lussier, P., & St‐Yves, M. (2009) · Justice Quarterly · doi:10.1080/07418820903218966 · Cited by 73
“Reliability was computed for this subsample (n = 70) using the Krippendorff’s alpha (Hayes & Krippendorff, 2007) for each of these variables.”
BackgroundMethodExploring holistic intuitive idea screening in the light of formal criteria
Magnusson, P., Netz, J., & Wästlund, E. (2014) · Technovation · doi:10.1016/j.technovation.2014.03.003 · Cited by 80
Ribeaud, D., & Eisner, M. (2010) · European Journal of Criminology · doi:10.1177/1477370810378116 · Cited by 78
Ward, K. M., Atkinson, J. P., Smith, C. A., & Windsor, R. A. (2013) · Intellectual and developmental disabilities · doi:10.1352/1934-9556-51.01.022 · Cited by 74
“A Krippendorff’s a (Krippendorff, 1970, 2004) was calculated to examine interrater reliability using an SPSS macro (Hayes & Krippendorff, 2007) resulting in a coefficient of 0.94.”
MethodInfluentialCooperative gestures: effective signaling for humanoid robots
Riek, L. D., Rabinowitch, T.-C., Bremner, P., Pipe, T., Fraser, M., & Robinson, P. (2010) · UWE Research Repository (UWE Bristol) · doi:10.5555/1734454.1734474 · Cited by 74
Family-based treatments for serious juvenile offenders: A multilevel meta-analysis.
Dopp, A. R., Borduin, C. M., White, M., & Kuppens, S. (2017) · Journal of Consulting and Clinical Psychology · doi:10.1037/ccp0000183 · Cited by 81
Bondjers, K., Hyland, P., Roberts, N. P., Bisson, J. I., Willebrand, M., & Arnberg, F. K. (2019) · European Journal of Psychotraumatology · doi:10.1080/20008198.2019.1665617 · Cited by 73
“Inter-rater agreement was examined for 12% of the sample using Krippendorff’s alpha (Hayes and Krippendorff, 2007).”
MethodBeyond East and West: Cognitive Style in Latin America
de Oliveira, S., & Nisbett, R. E. (2017) · Journal of Cross-Cultural Psychology · doi:10.1177/0022022117730816 · Cited by 82
“Krippendorff’s alpha was .98 for image height (95% confidence interval [CI] = [0.97, 0.99]), .96 for image width (95% CI = [0.92, 0.99]), and .89 for face height (95% CI = [0.83, 0.95]; computed using a script by Hayes & Krippendorff, 2007).”
MethodInfluentialFortune 1000 communication strategies on Facebook and Twitter
Tao, W., & Wilson, C. (2015) · Journal of Communication Management · doi:10.1108/jcom-01-2013-0004 · Cited by 72
Wells, R. S., Kolek, E. A., Williams, E. A., & Saunders, D. B. (2015) · The Journal of Higher Education · doi:10.1080/00221546.2015.11777361 · Cited by 94
Erkens, M., & Bodemer, D. (2018) · Computers & Education · doi:10.1016/j.compedu.2018.10.009 · Cited by 85
Neto, F. G. D. O., Torkar, R., FELDT, R. H., Gren, L., Furia, C. A., & Huang, Z. (2019) · Journal of Systems and Software · doi:10.1016/j.jss.2019.07.002 · Cited by 79
“We use Krippendorff’s αK due to its following properties [42]:”
MethodCrossing Domain‐Specific Boundaries in Search of Innovation: Exploring the Potential of Pyramiding*
Poetz, M. K., & Prügl, R. (2010) · Journal of Product Innovation Management · doi:10.1111/j.1540-5885.2010.00759.x · Cited by 72
“(2) Before the coders assessed the interview transcripts in more detail, they were given training with regard to the evaluation criteria as well as their definition and proper application (Krippendorff 2004, Hayes and Krippendorff 2007).”
The game frame and political efficacy: Beyond the spiral of cynicism
Pedersen, R. T. (2012) · European Journal of Communication · doi:10.1177/0267323112454089 · Cited by 71
Vliegenthart, R. (2011) · American Behavioral Scientist · doi:10.1177/0002764211419488 · Cited by 71
“The Krippendorff’s alpha score for all the coding decisions combined was .80, which is considered a good score (Hayes & Krippendorff, 2007).”
MethodUsing TikTok to Educate, Influence, or Inspire? A Content Analysis of Health-Related EduTok Videos
O’Donnell, N. H., Jerin, S. I., & Mu, D. (2023) · Journal of Health Communication · doi:10.1080/10810730.2023.2234866 · Cited by 71
“The KALPHA macro for SPSS was used to generate Krippendorff’s alpha reliability estimates (Hayes & Krippendorff, 2007).”
MethodBourke, M. E., & Grenyer, B. F. S. (2010) · Psychotherapy Research · doi:10.1080/10503307.2010.504242 · Cited by 71
“For the high-level cluster category (e.g., Cluster C), interrater reliability using Krippendorff ’s a (A. F. Hayes & Krippendorff, 2007) demonstrated acceptable correspondence (WSO, .79; WSS, .76; ROS, .79; RSO, .79; RSS, .80).”
MethodFinestack, L. H., Palmer, M., & Abbeduto, L. (2012) · American Journal of Speech-Language Pathology · doi:10.1044/1058-0360(2011/10-0095 · Cited by 74
Progress feedback effects on students' writing mastery goal, self-efficacy beliefs, and performance
Duijnhouwer, H., Prins, F. J., & Stokking, K. M. (2010) · Educational Research and Evaluation · doi:10.1080/13803611003711393 · Cited by 83
“For the ordinal and nominal variables we calculated Krippendorff's alpha (see Table 1 ), because it allows for missing categories (Hayes & Krippendorff, 2007 ).”
Method
Metadata
- Authors
- Hayes, A. F., & Krippendorff, K.
- Affiliations
- The Ohio State University (US)
- University of Pennsylvania (US)
- Venue
- Communication Methods and Measures
- Publisher
- Informa UK Limited
- Published
- 2007-04
- Volume
- 1
- Issue
- 1
- Pages
- 77-89
- ISSN
- 1931-2458 (ISSN-L)
- 1931-2466
- OpenAlex
- W2061504941
- Cited by
- 4,225
- References
- 30
- Citations by year
- 2026: 284 · 2025: 278 · 2024: 280 · 2023: 289 · 2022: 331 · 2021: 379 · 2020: 369 · 2019: 296 · 2018: 260 · 2017: 300 · 2016: 248 · 2015: 205 · 2014: 206 · 2013: 150 · 2012: 107
- FWCI
- 50.07
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Communication Methods and Measures (publishedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Reliability and Agreement in Measurement
- Social Sciences › General Social Sciences › Computational and Text Analysis Methods
- Social Sciences › Decision Sciences › Management Science and Operations Research › Psychometric Methodologies and Testing
- Keywords
- Krippendorff's alpha, inter-rater reliability, content analysis, coding reliability, reliability coefficients, observer agreement, SAS macro, missing data
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
