The DOI matches its Crossref record; 30 of the 30 references it cites were found in a registry or an index.
Answering the Call for a Standard Reliability Measure for Coding Data
Abstract
In content analysis and similar methods, data are typically generated by trained human observers who record or transcribe textual, pictorial, or audible matter in terms suitable for analysis. Conclusions from such data can be trusted only after demonstrating their reliability. Unfortunately, the content analysis literature is full of proposals for so-called reliability coefficients, leaving investigators easily confused, not knowing which to choose. After describing the criteria for a good measure of reliability, we propose Krippendorff's alpha as the standard reliability measure. It is general in that it can be used regardless of the number of observers, levels of measurement, sample sizes, and presence or absence of missing data. To facilitate the adoption of this recommendation, we describe a freely available macro written for SPSS and SAS to calculate Krippendorff's alpha and illustrate its use with a simple example.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“We calculate Krippendorff's alpha (Hayes and Krippendorff, 2007) to measure inter-rater reliability (see Table 3).”
From LLMs cannot find reasoning errors, but can correct them given the error location (2024) · cites this paper as Hayes and Krippendorff, 2007
In the citing paper
“Based on Krippendorff's alpha (α = 0.92), the level of agreement between coders was high and well above the threshold of 0.70 (Hayes & Krippendorff, 2007).”
From Collaboration, Coordination, and Cooperation Among Organizations: Establishing the Distinctive Meanings of These Terms Through a Systematic Literature Review (2020) · cites this paper as Hayes & Krippendorff, 2007
In the citing paper
“we calculated the inter-coder reliability based on 25% of all data units and derived a kappa value of .96, which reflects excellent reliability.”
In the citing paper
“To assess the inter-annotator agreement of stage 1, we report Krippendorf's α (Hayes and Krippendorff, 2007) for each dimension for the 135 articles after training and before any discussion/adjudication: economic (0.44), foreign (0.68), social (0.39).”
From Political Ideology and Polarization: A Multi-dimensional Approach (2022) · cites this paper as Hayes and Krippendorff, 2007
In the citing paper
“This family of metrics does not correct for chance (Cohen, 1960;Hayes and Krippendorff, 2007) and overstates discriminative ability (Guerdan et al., 2025;Han et al., 2025;Collot et al., 2025).”
From Reliability without Validity: A Systematic, Large-Scale Evaluation of LLM-as-a-Judge Models Across Agreement, Consistency, and Bias (2026) · cites this paper as Hayes and Krippendorff, 2007
Citations
Cites
Coefficient Alpha and the Internal Structure of Tests
Cronbach, L. J. (1951) · Psychometrika · doi:10.1007/bf02310555 · Cited by 43,931
A Coefficient of Agreement for Nominal Scales
Cohen, J. (1960) · Educational and Psychological Measurement · doi:10.1177/001316446002000104 · Cited by 42,398
An Introduction to the Bootstrap
Efron, B., & Tibshirani, R. J. (1994) · doi:10.1201/9780429246593 · Cited by 40,500
Nonparametric Statistics for the Behavioral Sciences
Guitton, H., & Siegel, S. (1958) · Revue économique · doi:10.2307/3498751 · Cited by 35,279
Content Analysis: An Introduction to its Methodology.
Shelley, M., & Krippendorff, K. (1984) · Journal of the American Statistical Association · doi:10.2307/2288384 · Cited by 24,351
Self-citationStatistical Methods for Rates and Proportions
Fleiss, J. L., Levin, B., & Paik, M. C. (2003) · Wiley Series in Probability and Statistics · doi:10.1002/0471445428 · Cited by 18,646
Measuring nominal scale agreement among many raters.
Fleiss, J. L. (1971) · Psychological Bulletin · doi:10.1037/h0031619 · Cited by 8,733
Content Analysis for the Social Sciences and Humanities.
Parker, E. B., & Holsti, O. R. (1970) · American Sociological Review · doi:10.2307/2093233 · Cited by 4,553
Content Analysis in Mass Communication: Assessment and Reporting of Intercoder Reliability
Lombard, M., Snyder-Duch, J., & Bracken, C. C. (2002) · Human Communication Research · doi:10.1111/j.1468-2958.2002.tb00826.x · Cited by 3,068
An Introduction to the Bootstrap
Grimshaw, S. D., Efron, B., & Tibshirani, R. J. (1995) · Technometrics · doi:10.2307/1269918 · Cited by 2,281
Research works printed but not linked: 1
- 10.1111/j.1468-2958.2004.tb00738.x · DOI · Paper page
Cited by
Sorted by citations
Items 101–200 of 1,013
Twitter versus Facebook: Comparing incivility, impoliteness, and deliberative attributes
Öz, M., Zheng, P., & Chen, G. M. (2017) · New Media & Society · doi:10.1177/1461444817749516 · Cited by 198
“Intercoder reliability was tested using Krippendorff’s alpha, which is the preferable test because it treats coders as independent, controls for agreement by chance, and is applicable to variables measured at nominal, ordinal, or interval-ratio levels (Hayes and Krippendorff, 2007).”
MethodThe Impacts of Service Failure and Recovery Efforts on Airline Customers’ Emotions and Satisfaction
Xu, X., Liu, W., & Gürsoy, D. (2018) · Journal of Travel Research · doi:10.1177/0047287518789285 · Cited by 191
“Krippendorff’s alpha value measures the agreement for nominal, ordinal, interval, and ratio data, which is fully comparable to the reliabilities reported for metric data (Hayes and Krippendorff 2007).”
ResultAkter, S., Bandara, R., Hani, U., Fosso Wamba, S., Foropon, C., & Papadopoulos, T. (2019) · International Journal of Information Management · doi:10.1016/j.ijinfomgt.2019.01.020 · Cited by 193
“We used Krippendorff's alpha (or, Kalpha) as a reliability measure to ensure further rigor in the thematic analysis (Krippendorff, 2004, 2007) following the guidelines of Hayes (2011) and De Swert (2012)to confirm the inter-rater reliability of the coded variables (Hayes & Krippendorff, 2007).”
MethodInvolvement of Fathers in Pediatric Obesity Treatment and Prevention Trials: A Systematic Review
Morgan, P. J., Young, M. D., Lloyd, A., Wang, M. L., Eather, N., Miller, A., Murtagh, E., Barnes, A. T., & Pagoto, S. (2017) · PEDIATRICS · doi:10.1542/peds.2016-2635 · Cited by 190
Rim, H., Lee, Y., & Yoo, S. (2019) · Public Relations Review · doi:10.1016/j.pubrev.2019.101869 · Cited by 190
Homburg, C., Vollmayr, J., & Hahn, A. (2014) · Journal of Marketing · doi:10.1509/jm.12.0179 · Cited by 189
Discrimination in grading: experimental evidence from primary school teachers
Sprietsma, M. (2012) · Empirical Economics · doi:10.1007/s00181-012-0609-x · Cited by 187
“To this purpose, we compute the Krippendorff’s alpha (Hayes and Krippendorff 2007) based on the rankings of the essays by teacher from 1 (best essay) to 10 (worst essay).”
MethodEisend, M., Plagemann, J., & Sollwedel, J. (2014) · Journal of Advertising · doi:10.1080/00913367.2013.857621 · Cited by 212
“Intercoder reliability was assessed by Krippendorff ’s a,which can easily be compared to standard reliability measures (Hayes and Krippendorff 2007).”
Method‘Let’s Get Them Involved’ . . . to Some Extent: Analyzing Online News Participation
Almgren, S., & Olsson, T. (2015) · Social Media + Society · doi:10.1177/2056305115621934 · Cited by 189
“The Krippendorff alpha values were calculated with KALPHA in the Statistical Package for the Social Sciences (SPSS) (Hayes & Krippendorff, 2007; Krippendorff, 2004).”
MethodLeveraging the power of place in citizen science for effective conservation decision making
Newman, G., Chandler, M., Clyde, M., McGreavy, B., Haklay, M., Ballard, H., Gray, S. A., Scarpino, R., Hauptfeld, R., Mellor, D., et al. (2017) · Biological Conservation · doi:10.1016/j.biocon.2016.07.019 · Cited by 179
“We calculated Krippendorff's alpha using the “kalpha” macro in SPSS for the independently coded 30 project subsample (Hayes and Krippendorff, 2007 - www.afhayes. com/public/kalpha.pdf).”
MethodThe effects of screen media content on young children’s executive functioning
Huber, B., Yeates, M., Meyer, D., Fleckhammer, L., & Kaufman, J. (2018) · Journal of Experimental Child Psychology · doi:10.1016/j.jecp.2018.01.006 · Cited by 206
Aggarwal, P., & Singh, A. K. (2018) · Social Responsibility Journal · doi:10.1108/srj-03-2018-0078 · Cited by 184
The False positive problem of automatic bot detection in social science research
Rauchfleisch, A., & Kaiser, J. (2020) · PLOS ONE · doi:10.1371/journal.pone.0241045 · Cited by 184
“While the percentage agreement is already an interesting indicator, we would highly recommend the use of a measurement like Krippendorff’s alpha [45] that takes into account chance agreement.”
MethodGuidelines of the minimum sample size requirements for Kappa agreement test
Bujang, M. A., & Baharum, N. (2022) · Epidemiology Biostatistics and Public Health · doi:10.2427/12267 · Cited by 182
A structured evaluation of business process improvement approaches
Zellner, G. (2011) · Business Process Management Journal · doi:10.1108/14637151111122329 · Cited by 181
“…was necessary because in content analysis, people (observers) are employed to systematically interpret the units of analysis (e.g. text), but “when relying on human observers, researchers must worry about the quality of the data – specifically, their reliability” (Hayes and Krippendorff, 2007).”
BackgroundInfluentialWould you follow your own route description? Cognitive strategies in urban route planning
Hölscher, C., Tenbrink, T., & Wiener, J. M. (2011) · Cognition · doi:10.1016/j.cognition.2011.06.005 · Cited by 183
“For this analysis, we computed Krippendorff's Alpha (using the SPSS macro described in Hayes & Krippendorff, 2007) for a representative subset of the data, with the entirely satisfactory result of 0.84.”
MethodInfluentialDirect versus Indirect Treatment for Preschool Children who Stutter: The RESTART Randomized Trial
de Sonneville-Koedoot, C., Stolk, E., Rietveld, T., & Franken, M. (2015) · PLoS ONE · doi:10.1371/journal.pone.0133758 · Cited by 147
“Since the data on %SS did not meet the assumptions needed to calculate CIs for the intraclass correlation coefficient (ICC), interjudge reliability of the speech samples was assessed using Krippendorff's alpha [41] with the option 'interval data' for the macro developed by Hayes (2013) [42].”
MethodThe 2008 Presidential Election, 2.0: A Content Analysis of User-Generated Political Facebook Groups
Woolley, J. K., Limperos, A. M., & Oliver, M. B. (2010) · Mass Communication & Society · doi:10.1080/15205436.2010.516864 · Cited by 176
“To calculate intercoder reliability, we used Hayes's and Krippendorf's alpha macro (Hayes & Krippendorf, 2007 ).”
MethodRippe, R. C. A., Noppe, G., Windhorst, D. A., Tiemeier, H. W., van Rossum, E. F. C., Jaddoe, V. W. V., Verhulst, F. C., Bakermans‐Kranenburg, M. J., van IJzendoorn, M. H., & van den Akker, E. L. T. (2015) · Psychoneuroendocrinology · doi:10.1016/j.psyneuen.2015.12.016 · Cited by 173
Cheng, C., Wang, H., Sigerson, L., & Chau, C.-L. (2019) · Psychological Bulletin · doi:10.1037/bul0000198 · Cited by 197
“Krippendorff’s alpha ( ) was computed using the SPSS macro KALPHA (Hayes & Krippendorff, 2007) to assess coding agreement.”
MethodReal-time mutual gaze perception enhances collaborative learning and collaboration quality
Schneider, B., & Pea, R. D. (2013) · International Journal of Computer-Supported Collaborative Learning · doi:10.1007/s11412-013-9181-4 · Cited by 183
“An alpha higher than 0.8 is considered as a reliable agreement between judges (Hayes & Krippendorff 2007).”
BackgroundInfluentialCORRELATES OF VIOLENT POLITICAL EXTREMISM IN THE UNITED STATES*
Lafree, G. D., Jensen, M. A., JAMES, P. A., & Safer-Lichtenstein, A. (2018) · Criminology · doi:10.1111/1745-9125.12169 · Cited by 176
“…10 percent of the individuals in the data to allow for iterative reliability tests of the coding instrument for each stage of the coding process, and they used the Krippendorf’s alpha procedure to test for inter-rater reliability across the double-coded cases (Hayes and Krippendorf, 2007).”
MethodWang, H., & Cheng, C. (2022) · JMIR Mental Health · doi:10.2196/23700 · Cited by 187
“92 [62], indicating a high level of interreviewer reliability.”
BackgroundSample size in psychological research over the past 30 years
Marszalek, J. M., Barber, C., Kohlhart, J., & Cooper, B. H. (2011) · Perceptual and Motor Skills · doi:10.2466/03.11.pms.112.2.331-348 · Cited by 167
“Agreement was measured with Krippendorff’s alpha, α K , using the KALPHA SPSS (now PASW) macro (Hayes & Krippendorff, 2007), treating sample size as a ratio level measurement and performing 1,000 bootstrapping samples.”
Evolution of library and information science, 1965–2005: Content analysis of journal articles
Tuomaala, O., Järvelin, K., & Vakkari, P. (2014) · Journal of the Association for Information Science and Technology · doi:10.1002/asi.23034 · Cited by 129
“As Hayes and Krippendorff (2007) have noted, this relatively rarely used algorithm is one of the most sophisticated methods for calculation of reliability.”
MethodCheap talk? Strategy presentations as a form of chief executive officer impression management
Whittington, R., Yakis‐Douglas, B., & Ahn, K. (2015) · Strategic Management Journal · doi:10.1002/smj.2482 · Cited by 171
Davison, K. K., Kronsteiner-Gicevic, S., Aftosmes‐Tobio, A., Ganter, C., Simon, C. L., Newlan, S., & Manganello, J. A. (2016) · American Journal of Public Health · doi:10.2105/ajph.2016.303391 · Cited by 170
Overcoming Resistance Through Narratives: Findings from a Meta-Analytic Review
Ratcliff, C. L., & Sun, Y. (2020) · Human Communication Research · doi:10.1093/hcr/hqz017 · Cited by 170
Circle of Security–Parenting: A randomized controlled trial in Head Start
Cassidy, J., Brett, B. E., Gross, J. T., Stern, J. A., Martin, D. R., Mohr, J. J., & Woodhouse, S. S. (2017) · Development and Psychopathology · doi:10.1017/s0954579417000244 · Cited by 167
“The Krippendorff a values (Hayes & Krippendorff, 2007) for the three task variables ranged from 0.94 to 1.00.”
MethodInfluentialCoder Reliability and Misclassification in the Human Coding of Party Manifestos
Mikhaylov, S., Laver, M., & Benoit, K. R. (2012) · Political Analysis · doi:10.1093/pan/mpr047 · Cited by 250
Repeatability and reproducibility of MRI-based radiomic features in cervical cancer
Fiset, S., Welch, M. L., Weiss, J., Pintilie, M., Conway, J. L., Milosevic, M., Fyles, A., Traverso, A., Jaffray, D. A., Metser, U., et al. (2019) · Radiotherapy and Oncology · doi:10.1016/j.radonc.2019.03.001 · Cited by 167
Watted, A., & Barak, M. (2017) · The Internet and Higher Education · doi:10.1016/j.iheduc.2017.12.001 · Cited by 165
Schaltegger, S., Christ, K. L., Wenzig, J., & Burritt, R. L. (2022) · International Journal of Management Reviews · doi:10.1111/ijmr.12288 · Cited by 244
Gill, D. L., Dickinson, S., & Arno Scharl, A. (2008) · Journal of Communication Management · doi:10.1108/13632540810899425 · Cited by 163
Dangerous minds? Effects of uncivil online comments on aggressive cognitions, emotions, and behavior
Rösner, L., Winter, S., & Krämer, N. C. (2016) · Computers in Human Behavior · doi:10.1016/j.chb.2016.01.022 · Cited by 191
“Additionally we calculated Krippendorff's Alpha as a more conservative index (accounting for agreement expected by chance) with the SPSS extension by Hayes and Krippendorff (2007) for nominal data (attack1: a ¼ 0; attack2: a ¼ .79; attack3: a ¼ 1; profanity: a ¼ 1; shouting: a ¼ no variation;…”
MethodImplementing a General Framework for Assessing Interrater Agreement in Stata
Klein, D. (2018) · The Stata Journal Promoting communications on statistics and Stata · doi:10.1177/1536867x1801800408 · Cited by 173
“The kalpha command (Klein 2014) adds circular and bipolar weights for partial agreement and implements a bootstrap algorithm, described by Hayes and Krippendorff (2007), to obtain confidence intervals.”
Ranalli, J., Link, S., & Chukharev‐Hudilainen, E. (2016) · Educational Psychology · doi:10.1080/01443410.2015.1136407 · Cited by 207
“At the end of training and calibration, we had co-rated a set of 360 errors and were consistently achieving an agreement rate of greater than .7 using Krippendorff’s alpha 4 (Hayes & Krippendorff, 2007).”
MethodCommunity Attitudes Toward Wildlife and Protected Areas in Ethiopia
Tessema, M. E., Lilieholm, R. J., Ashenafi, Z. T., & Leader‐Williams, N. (2010) · Society & Natural Resources · doi:10.1080/08941920903177867 · Cited by 159
“The SPSS version of KALPHA MACRO software was used to compute Krippendorff's alpha to determine intercoder reliability (Hayes and Krippendorff 2007 ).”
MethodFunctional implications of age differences in motor system connectivity
Langan, J. (2010) · Frontiers in Systems Neuroscience · doi:10.3389/fnsys.2010.00017 · Cited by 168
Towards Exploiting Background Knowledge for Building Conversation Systems
Moghe, N., Arora, S., Banerjee, S., & Khapra, M. M. (2018) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d18-1255 · Cited by 162
SemEval 2016 Task 11: Complex Word Identification
Paetzold, G. H., & Specia, L. (2016) · International Workshop on Semantic Evaluation (SemEval) · doi:10.18653/v1/s16-1085 · Cited by 165
“We calculated the Krippendorff’s Alpha agreement coefficient (Hayes and Krippendorff, 2007) for each set of 10 sentences that were annotated by 20 volunteers.”
MethodDavis, J. L., Love, T. P., & Fares, P. (2019) · Social Psychology Quarterly · doi:10.1177/0190272519851025 · Cited by 205
Attrition in developmental psychology
Nicholson, J. S., Deboeck, P. R., & Howard, W. (2015) · International Journal of Behavioral Development · doi:10.1177/0165025415618275 · Cited by 158
“Inter-rater reliability was assessed using Krippendorff’s a (Hayes & Krippendorff, 2007); discrepancies between coders were resolved by the first author as a master coder.”
MethodMobile Persuasion: Can Screen Size and Presentation Mode Make a Difference to Trust?
Kim, K. J., & Sundar, S. S. (2015) · Human Communication Research · doi:10.1111/hcre.12064 · Cited by 155
“The Krippendorff’s alpha (Hayes & Krippendorff, 2007) for the number of heuristic and systematic thoughts were .84, indicating strong intercoder agreement.”
MethodA framework of actions for strong sustainability
Neto, G. C. D. O., Pinto, L. F. R., Amorim, M. P. C., Giannetti, B. F., & Almeida, C. M. V. B. (2018) · Journal of Cleaner Production · doi:10.1016/j.jclepro.2018.06.067 · Cited by 155
Hybrid Activism: Social Movement Mobilization in a Multimovement Environment
Heaney, M. T., & Rojas, F. G. (2014) · American Journal of Sociology · doi:10.1086/674897 · Cited by 154
“The most broadly accepted measure of intercoder agreement is Krippendorff’s a ðHayes and Krippendorff 2007Þ.”
BackgroundGender-Role Portrayals in Television Advertising Across the Globe
Matthes, J., Prieler, M., & Adam, K. (2016) · Sex Roles · doi:10.1007/s11199-016-0617-y · Cited by 175
“We performed three reliability tests using Krippendorff’s Alpha (Hayes and Krippendorff 2007; n=10 each, from 2 to 30 coders).”
MethodThe Importance of the Raw Idea in Innovation: Testing the Sow's Ear Hypothesis
Kornish, L. J., & Ulrich, K. T. (2013) · Journal of Marketing Research · doi:10.1509/jmr.12.0401 · Cited by 153
“24 these experts, and Krippendorff’s alpha (Hayes and Krippendorff, 2007), which is 0.”
BackgroundKahlor, L. A., & Rosenthal, S. (2009) · Science Communication · doi:10.1177/1075547008328798 · Cited by 151
What a Waste! Exploring the Human Reality of Food Waste from the Store Manager's Perspective
Gruber, V., Holweg, C., & Teller, C. (2015) · Journal of Public Policy & Marketing · doi:10.1509/jppm.14.095 · Cited by 160
Svenningsson, J., Höst, G., Hultén, M., & Hallström, J. (2022) · International Journal of Technology and Design Education · doi:10.1007/s10798-021-09657-7 · Cited by 200
Kim, S., & Liu, B. F. (2012) · Journal of Public Relations Research · doi:10.1080/1062726x.2012.626136 · Cited by 150
“80 using SPSS macros for Kripendorff's alpha (Hayes & Krippendorff, 2007 ), which is appropriate for measurement level variables from nominal to ratio and accounts for chance agreement, making it a more conservative estimate of reliability (Lombard, Snyder-Duch, & Bracken, 2002 ).”
MethodSocial Media Messages in an Emerging Health Crisis: Tweeting Bird Flu
Vos, S. C., & Buckner, M. M. (2015) · Journal of Health Communication · doi:10.1080/10810730.2015.1064495 · Cited by 155
“Krippendorff’s alpha was used to measure intercoder reliability, using the SPSS macro created by Hayes and Krippendorff (2007).”
MethodCell phone-induced ostracism threatens fundamental needs
Hales, A. H., Dvir, M., Wesselmann, E., Kruger, D., & Finkenauer, C. (2018) · Journal of Social Psychology · doi:10.1080/00224545.2018.1439877 · Cited by 131
“We conducted an exploratory/posthoc coding of the responses to probe possible of relationship to the conversation partner (inter-rater reliability = .99; Hayes & Krippendorff, 2007), and whether the topic of the conversation involved a hardship or challenge (reliability = .77).”
MethodRead All About It: The Politicization of “Fake News” on Twitter
Brummette, J., DiStaso, M. W., Vafeiadis, M., & Messner, M. (2018) · Journalism & Mass Communication Quarterly · doi:10.1177/1077699018769906 · Cited by 149
“The inter-coder reliability coefficients (Hayes & Krippendorff, 2007) for the variables were as follows: political ideology in tweet (.83), context of the term (.79), valence (.84), type of social media user (.92), user’s political affiliation (.96), gender (.98).”
MethodThe Signaling Role of IT Features in Influencing Trust and Participation in Online Communities
Benlian, A., & Heß, T. (2011) · International Journal of Electronic Commerce · doi:10.2753/jec1086-4415150401 · Cited by 156
“5 percent of the full sample), because it is generally accepted to be the most relevant measure of agreement among multiple coders [50].”
BackgroundBorst, I., Möser, C., & Ferguson, J. (2017) · New Media & Society · doi:10.1177/1461444817694599 · Cited by 167
“The reliability of the human-coded quantitative analyses was measured according to Krippendorff’s alpha (Hayes and Krippendorff, 2007).”
MethodGraphic Warning Labels in Cigarette Advertisements
Strasser, A. A., Tang, K. Z., Römer, D., Jepson, C., & Cappella, J. N. (2012) · American Journal of Preventive Medicine · doi:10.1016/j.amepre.2012.02.026 · Cited by 147
The sound of confidence and doubt
Jiang, X., & Pell, M. D. (2017) · Speech Communication · doi:10.1016/j.specom.2017.01.011 · Cited by 146
“…[.66, .75] for list 2; α= .61, CI [.56, .67] for list 3; α= .71, CI [.65, 75] for list 4; α= .70, CI [.65, .75] for list 5; α= .75, CI [.71, .79] for list 6) based on 10,0 0 0 bootstrap samples to produce confidence intervals at the 95% level ( Hayes and Krippendorff, 2007 ). and structures, etc.).”
Nanni, C., Versari, A., Chauvie, S., Bertone, E., Bianchi, A., Rensi, M., Bellò, M., Gallamini, A., Patriarca, F. F., Gay, F., et al. (2017) · European Journal of Nuclear Medicine and Molecular Imaging · doi:10.1007/s00259-017-3909-8 · Cited by 146
“The method used for inter-observer variability included the percentage of agreement and the Krippendorff’s alpha [15].”
MethodDuijnhouwer, H., Prins, F. J., & Stokking, K. M. (2011) · Learning and Instruction · doi:10.1016/j.learninstruc.2011.10.003 · Cited by 172
“Inter-rater agreement (Krippendorff’s alpha; Hayes & Krippendorff, 2007) was sufficient (ak> 0.65) for all aspects.”
BackgroundInfluentialAvraham, E. (2018) · Journal of Business Research · doi:10.1016/j.jbusres.2018.02.036 · Cited by 145
Zamagni, E., Nanni, C., Gay, F., Pezzi, A., Patriarca, F. F., Bellò, M., Rambaldi, I., Tacchetti, P., Hillengaß, J., Gamberi, B., et al. (2015) · Leukemia · doi:10.1038/leu.2015.291 · Cited by 144
Hackers vs. Testers: A Comparison of Software Vulnerability Discovery Processes
Votipka, D., Stevens, R., Redmiles, E. M., Hu, J., & Mazurek, M. L. (2018) · IEEE Symposium on Security and Privacy (S&P) · doi:10.1109/sp.2018.00003 · Cited by 146
“We use Krippendorff’s Alpha (α) to measure inter-coder reliability as it accounts for chance agreements [93].”
MethodThe Behaviour Change Technique Ontology: Transforming the Behaviour Change Technique Taxonomy v1
Marques, M. M., Wright, A. J., Corker, E., Johnston, M., West, R., Hastings, J., Zhang, L. M., & Michie, S. (2023) · Wellcome Open Research · doi:10.12688/wellcomeopenres.19363.1 · Cited by 235
Gender Stereotypes in Science Education Resources: A Visual Content Analysis
Kerkhoven, A. H., Russo, P., Land‐Zandstra, A. M., Saxena, A., & Rodenburg, F. J. (2016) · PLoS ONE · doi:10.1371/journal.pone.0165037 · Cited by 143
Vandor, P., & Franke, N. (2016) · Journal of Business Venturing · doi:10.1016/j.jbusvent.2016.03.003 · Cited by 152
Political Journalists’ Normalization of Twitter
Molyneux, L., & Mourão, R. R. (2017) · Journalism Studies · doi:10.1080/1461670x.2017.1370978 · Cited by 142
Chen, M. A., & Hwang, G. (2019) · British Journal of Educational Technology · doi:10.1111/bjet.12887 · Cited by 203
How journalists engage in branding on Twitter: individual, organizational, and institutional levels
Molyneux, L., Holton, A. E., & Lewis, S. C. (2017) · Information Communication & Society · doi:10.1080/1369118x.2017.1314532 · Cited by 141
“Overall raw agreement ranged from 94% to 100%, and Krippendorf’s alpha for all variables ranged from.84 to 1.00, well within conventional levels of acceptable intercoder reliability (see Hayes & Krippendorff, 2007).”
BackgroundInfluentialBecker, W., & Schmid, O. (2020) · Business Research · doi:10.1007/s40685-020-00124-y · Cited by 121
Reflection in the Shadow of Blame: When Do Politicians Appoint Commissions of Inquiry?
Sulitzeanu‐Kenan, R. (2010) · British Journal of Political Science · doi:10.1017/s0007123410000049 · Cited by 143
Sharp, G., Tiggemann, M., & Mattiske, J. K. (2016) · Aesthetic Surgery Journal · doi:10.1093/asj/sjv270 · Cited by 140
Hays, D. G., Wood, C., Dahl, H., & Kirk‐Jenkins, A. (2016) · Journal of Counseling & Development · doi:10.1002/jcad.12074 · Cited by 139
Why do people lie online? “Because everyone lies on the internet”
Drouin, M. A., Miller, D. A., Wehle, S. M. J., & Hernandez, E. (2016) · Computers in Human Behavior · doi:10.1016/j.chb.2016.06.052 · Cited by 144
“Inter-rater reliability, as measured by Krippendorff’s alpha (Hayes & Krippendorff, 2007), was high ¼ 0.81.”
MethodInfluentialHong, S., Jahng, M. R., Lee, N., & Wise, K. R. (2019) · Computers in Human Behavior · doi:10.1016/j.chb.2019.106159 · Cited by 139
Data Richness Trade-Offs Between Face-to-Face, Online Audiovisual, and Online Text-Only Focus Groups
Abrams, K. M., Wang, Z., Song, Y. J., & Galindo, S. (2014) · Social Science Computer Review · doi:10.1177/0894439313519733 · Cited by 164
Generating Fact Checking Explanations
Atanasova, P., Simonsen, J. G., Lioma, C., & Augenstein, I. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.656 · Cited by 165
Elmelund‐Præstekær, C. (2010) · European Political Science Review · doi:10.1017/s1755773909990269 · Cited by 137
“Two trained coders recoded a subset of the data (randomly drawn from every party), and the coders’ average tone of each included unit was compared using Krippendorff’s alpha algorithm (Hayes and Krippendorff, 2007).”
MethodAssociation between residential greenness and birth weight: Systematic review and meta-analysis
Dzhambov, A. M., Dimitrova, D., & Dimitrakova, E. D. (2014) · Urban forestry & urban greening · doi:10.1016/j.ufug.2014.09.004 · Cited by 146
Test-Retest and Interrater Reliability of the Functional Movement Screen
Shultz, R., Anderson, S. C., Matheson, G. O., Marcello, B., & Besier, T. F. (2013) · Journal of Athletic Training · doi:10.4085/1062-6050-48.2.11 · Cited by 136
“The K a was developed to establish a reliability measure that works well with more than 2 raters and different types of data (interval, ordinal) and can correctly handle missing data.(16) The total FMS scores were categorized as interval data, whereas the individual tasks were categorized as ordinal data.”
MethodRhun, É. L., Devos, P., Boulanger, T., Smits, M., Brandsma, D., Rudà, R., Furtner, J., Hempel, J.-M., Postma, T. J., Roth, P., et al. (2019) · Neuro-Oncology · doi:10.1093/neuonc/noz024 · Cited by 136
The evolution of wind energy policies in China (1995–2014): An analysis based on policy instruments
Liao, Z. (2015) · Renewable and Sustainable Energy Reviews · doi:10.1016/j.rser.2015.11.097 · Cited by 135
Mapping of internal audit research: a post-Enron structured literature review
Kotb, A. M., Elbardan, H., & Halabi, H. (2020) · Accounting Auditing & Accountability Journal · doi:10.1108/aaaj-07-2018-3581 · Cited by 135
Experiences of U.S. military couples during the post-deployment transition
Knobloch, L. K., & Theiss, J. A. (2012) · Journal of Social and Personal Relationships · doi:10.1177/0265407511431186 · Cited by 136
“…to other content analytic reliability measures because it (a) assesses agreement between any number of observers, (b) is independent of the distribution of categories, (c) generalizes across any level of measurement, and (d) possesses a computable sampling distribution (Hayes & Krippendorff, 2007).”
BackgroundMultichannel integration quality: A systematic review and agenda for future research
Hossain, T. M. T., Akter, S., Kattiyapornpong, U., & Dwivedi, Y. K. (2019) · Journal of Retailing and Consumer Services · doi:10.1016/j.jretconser.2019.03.019 · Cited by 154
Public value and public sector accounting research: a structured literature review
Bracci, E., Papi, L., Bigoni, M., Gagliardo, E. D., & Bruns, H.-J. (2019) · Journal of Public Budgeting Accounting & Financial Management · doi:10.1108/jpbafm-07-2018-0077 · Cited by 150
Tong, C., Gill, H., Li, J., Valenzuela, S., & Rojas, H. (2020) · Mass Communication & Society · doi:10.1080/15205436.2020.1789661 · Cited by 140
Intellectual capital and SMEs’ performance: A structured literature review
Demartini, C., & Beretta, V. (2019) · Journal of Small Business Management · doi:10.1080/00472778.2019.1659680 · Cited by 151
“Since coding is a form of content analysis, the reliability measure we selected was Krippendorff ’ s alpha (Hayes & Krippendorff, 2007; Krippendorff, 2004).”
MethodShu, H., & Gu, X. (2018) · The Internet and Higher Education · doi:10.1016/j.iheduc.2018.05.003 · Cited by 145
On Crowdsourcing Relevance Magnitudes for Information Retrieval Evaluation
Maddalena, E., Mizzaro, S., Scholer, F., & Turpin, A. H. (2017) · ACM Transactions on Information Systems · doi:10.1145/3002172 · Cited by 127
Wirtz, J. G., & Zimbres, T. M. (2018) · Journal of Public Relations Research · doi:10.1080/1062726x.2018.1455146 · Cited by 128
“To establish intercoder reliability, Krippendorf’s alpha was calculated for each variable using the Hayes and Krippendorff (2007) SPSS macro.”
MethodAccounting for modern slavery: an analysis of Australian listed company disclosures
Christ, K. L., Rao, K. K., & Burritt, R. L. (2019) · Accounting Auditing & Accountability Journal · doi:10.1108/aaaj-11-2017-3242 · Cited by 140
Wozniak, A., Lück, J., & Weßler, H. (2014) · Environmental Communication · doi:10.1080/17524032.2014.981559 · Cited by 127
“Krippendorff’s alpha employs a covariance-based correction for chance (Hayes & Krippendorff, 2007).”
MethodSong, H., Tolochko, P., Eberl, J.-M., Eisele, O., Greussing, E., Heidenreich, T., Lind, F., Galyga, S., & Boomgaarden, H. G. (2020) · Political Communication · doi:10.1080/10584609.2020.1723752 · Cited by 123
“In such cases, researchers should adhere to rigorous methodological standards to the degree expected for traditional manual content analysis (e.g., Hayes & Krippendor ff , 2007; Krippendor ff , 2013) in preparing a manually annotated validation data set.”
InfluentialSteiner, D. F., Nagpal, K., Sayres, R., Foote, D., Wedin, B., Pearce, A., Cai, C. J., Winter, S. R., Symonds, M., Yatziv, L., et al. (2020) · JAMA Network Open · doi:10.1001/jamanetworkopen.2020.23267 · Cited by 133
The Blame Frame: Media Attribution of Culpability About the MMR–Autism Vaccination Scare
Holton, A. E., McKeever, B. W., Clarke, C. E., & Smith, M. J. (2012) · Health Communication · doi:10.1080/10410236.2011.633158 · Cited by 127
“It is a robust measure of reliability that can be used regardless of the number of coders, levels of measurement, sample size, and presence of missing data (Hayes & Krippendorff, 2007).”
BackgroundNili, A., Tate, M., Barros, A., & Johnstone, D. (2020) · International Journal of Information Management · doi:10.1016/j.ijinfomgt.2020.102154 · Cited by 157
“( Hayes & Krippendor ff , 2007; Zhao et al., 2013).”
BackgroundMethodResultInfluentialBuilding dynamic service analytics capabilities for the digital marketplace
Akter, S., Motamarri, S., Hani, U., Shams, S. M. R., Fernando, M., Babu, M. M., & Shen, K. N. (2020) · Journal of Business Research · doi:10.1016/j.jbusres.2020.06.016 · Cited by 136
Broderick, J. E., Keefe, F. J., Bruckenthal, P., Junghaenel, D. U., Schneider, S., Schwartz, J. E., Kaell, A. T., Caldwell, D. S., McKee, D., Reed, S., et al. (2014) · Pain · doi:10.1016/j.pain.2014.05.024 · Cited by 118
“79) and Krippendorff's [23] ordinal alpha = .”
Metadata
- Authors
- Hayes, A. F., & Krippendorff, K.
- Affiliations
- The Ohio State University (US)
- University of Pennsylvania (US)
- Venue
- Communication Methods and Measures
- Publisher
- Informa UK Limited
- Published
- 2007-04
- Volume
- 1
- Issue
- 1
- Pages
- 77-89
- ISSN
- 1931-2458 (ISSN-L)
- 1931-2466
- OpenAlex
- W2061504941
- Cited by
- 4,225
- References
- 30
- Citations by year
- 2026: 284 · 2025: 278 · 2024: 280 · 2023: 289 · 2022: 331 · 2021: 379 · 2020: 369 · 2019: 296 · 2018: 260 · 2017: 300 · 2016: 248 · 2015: 205 · 2014: 206 · 2013: 150 · 2012: 107
- FWCI
- 50.07
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Communication Methods and Measures (publishedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Reliability and Agreement in Measurement
- Social Sciences › General Social Sciences › Computational and Text Analysis Methods
- Social Sciences › Decision Sciences › Management Science and Operations Research › Psychometric Methodologies and Testing
- Keywords
- Krippendorff's alpha, inter-rater reliability, content analysis, coding reliability, reliability coefficients, observer agreement, SAS macro, missing data
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
