The DOI matches its Crossref record; 30 of the 30 references it cites were found in a registry or an index.
Answering the Call for a Standard Reliability Measure for Coding Data
Abstract
In content analysis and similar methods, data are typically generated by trained human observers who record or transcribe textual, pictorial, or audible matter in terms suitable for analysis. Conclusions from such data can be trusted only after demonstrating their reliability. Unfortunately, the content analysis literature is full of proposals for so-called reliability coefficients, leaving investigators easily confused, not knowing which to choose. After describing the criteria for a good measure of reliability, we propose Krippendorff's alpha as the standard reliability measure. It is general in that it can be used regardless of the number of observers, levels of measurement, sample sizes, and presence or absence of missing data. To facilitate the adoption of this recommendation, we describe a freely available macro written for SPSS and SAS to calculate Krippendorff's alpha and illustrate its use with a simple example.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“We calculate Krippendorff's alpha (Hayes and Krippendorff, 2007) to measure inter-rater reliability (see Table 3).”
From LLMs cannot find reasoning errors, but can correct them given the error location (2024) · cites this paper as Hayes and Krippendorff, 2007
In the citing paper
“Based on Krippendorff's alpha (α = 0.92), the level of agreement between coders was high and well above the threshold of 0.70 (Hayes & Krippendorff, 2007).”
From Collaboration, Coordination, and Cooperation Among Organizations: Establishing the Distinctive Meanings of These Terms Through a Systematic Literature Review (2020) · cites this paper as Hayes & Krippendorff, 2007
In the citing paper
“we calculated the inter-coder reliability based on 25% of all data units and derived a kappa value of .96, which reflects excellent reliability.”
In the citing paper
“To assess the inter-annotator agreement of stage 1, we report Krippendorf's α (Hayes and Krippendorff, 2007) for each dimension for the 135 articles after training and before any discussion/adjudication: economic (0.44), foreign (0.68), social (0.39).”
From Political Ideology and Polarization: A Multi-dimensional Approach (2022) · cites this paper as Hayes and Krippendorff, 2007
In the citing paper
“This family of metrics does not correct for chance (Cohen, 1960;Hayes and Krippendorff, 2007) and overstates discriminative ability (Guerdan et al., 2025;Han et al., 2025;Collot et al., 2025).”
From Reliability without Validity: A Systematic, Large-Scale Evaluation of LLM-as-a-Judge Models Across Agreement, Consistency, and Bias (2026) · cites this paper as Hayes and Krippendorff, 2007
Citations
Cites
Coefficient Alpha and the Internal Structure of Tests
Cronbach, L. J. (1951) · Psychometrika · doi:10.1007/bf02310555 · Cited by 43,931
A Coefficient of Agreement for Nominal Scales
Cohen, J. (1960) · Educational and Psychological Measurement · doi:10.1177/001316446002000104 · Cited by 42,398
An Introduction to the Bootstrap
Efron, B., & Tibshirani, R. J. (1994) · doi:10.1201/9780429246593 · Cited by 40,500
Nonparametric Statistics for the Behavioral Sciences
Guitton, H., & Siegel, S. (1958) · Revue économique · doi:10.2307/3498751 · Cited by 35,279
Content Analysis: An Introduction to its Methodology.
Shelley, M., & Krippendorff, K. (1984) · Journal of the American Statistical Association · doi:10.2307/2288384 · Cited by 24,346
Self-citationStatistical Methods for Rates and Proportions
Fleiss, J. L., Levin, B., & Paik, M. C. (2003) · Wiley Series in Probability and Statistics · doi:10.1002/0471445428 · Cited by 18,646
Measuring nominal scale agreement among many raters.
Fleiss, J. L. (1971) · Psychological Bulletin · doi:10.1037/h0031619 · Cited by 8,733
Content Analysis for the Social Sciences and Humanities.
Parker, E. B., & Holsti, O. R. (1970) · American Sociological Review · doi:10.2307/2093233 · Cited by 4,553
Content Analysis in Mass Communication: Assessment and Reporting of Intercoder Reliability
Lombard, M., Snyder-Duch, J., & Bracken, C. C. (2002) · Human Communication Research · doi:10.1111/j.1468-2958.2002.tb00826.x · Cited by 3,068
An Introduction to the Bootstrap
Grimshaw, S. D., Efron, B., & Tibshirani, R. J. (1995) · Technometrics · doi:10.2307/1269918 · Cited by 2,281
Research works printed but not linked: 1
- 10.1111/j.1468-2958.2004.tb00738.x · DOI · Paper page
Cited by
Sorted by citations
Items 201–300 of 1,013
Nanni, C., Zamagni, E., Versari, A., Chauvie, S., Bianchi, A., Rensi, M., Bellò, M., Rambaldi, I., Gallamini, A., Patriarca, F. F., et al. (2015) · European Journal of Nuclear Medicine and Molecular Imaging · doi:10.1007/s00259-015-3200-9 · Cited by 124
Short video marketing: what, when and how short-branded videos facilitate consumer engagement
Dong, X., Liu, H., Xi, N., Liao, J., & Yang, Z. G. O. (2023) · Internet Research · doi:10.1108/intr-02-2022-0121 · Cited by 124
Thompson, A. R., & O’Loughlin, V. D. (2014) · Anatomical Sciences Education · doi:10.1002/ase.1507 · Cited by 135
“This statistic is preferred to other options due to its flexibility to be used with all types of data (nominal/dichotomous, ordinal, and interval/ratio), ability to handle multiple observers, and capacity to work with missing data (Hayes and Krippendorff, 2007).”
MethodMistakes and How to Avoid Mistakes in Using Intercoder Reliability Indices
Feng, G. C. (2014) · Methodology · doi:10.1027/1614-2241/a000086 · Cited by 132
Schuler, S., Fanta, D., Rosenkränzer, F., & Rieß, W. (2017) · Journal of Geography in Higher Education · doi:10.1080/03098265.2017.1339264 · Cited by 153
“After a few codings and some adaption of the coding scheme, we reached an acceptable interrater reliability (Krippendorff ’s α > .80; Hayes & Krippendorff, 2007).”
MethodThe role of artificial intelligence in shaping the future of Agile fashion industry
Babu, M. M., Akter, S., Rahman, M., Billah, M. M., & Hack‐Polay, D. (2022) · Production Planning & Control · doi:10.1080/09537287.2022.2060858 · Cited by 152
“In order to embrace further rigour in the thematic analysis, the study used Krippendorff ’ s alpha (or, Kalpha) as a reliability measure, which is 0.84 exceeding the threshold value of 0.70 (Hayes and Krippendorff 2007).)”
InfluentialDeterminants of Maternal Health Care Utilization in Nigeria: a multilevel approach.
Ononokpono, D. N., & Odimegwu, C. O. (2014) · Europe PMC (PubMed Central) · doi:10.11694/pamj.supp.2014.17.1.3596 · Cited by 121
Impression management and formation on Facebook: A lens model approach
Hall, J. A., Pennington, N., & Lueders, A. (2013) · New Media & Society · doi:10.1177/1461444813495166 · Cited by 141
Four dimensions of criticism against gender-fair language
Vergoossen, H. P., Renström, E. A., Lindqvist, A., & Gustafsson Sendén, M. (2020) · Sex Roles · doi:10.1007/s11199-019-01108-x · Cited by 121
“The inter-coder reliability for coding into the original categories was indexed with Krippendorff ’ s alpha (Hayes and Krippendorff 2007) and was .89 for all categories taken together (range = .57 – 1).”
Wirtz, J. G., Sparks, J. V., & Zimbres, T. M. (2017) · International Journal of Advertising · doi:10.1080/02650487.2017.1334996 · Cited by 121
Privacy concerns in E-commerce: A taxonomy and a future research agenda
Bandara, R. J., Fernando, M., & Akter, S. (2019) · Electronic Markets · doi:10.1007/s12525-019-00375-6 · Cited by 133
Product specific values and personal values together better explains green purchase
Bhardwaj, S., Sreen, N., Das, M., Chitnis, A., & Kumar, S. (2023) · Journal of Retailing and Consumer Services · doi:10.1016/j.jretconser.2023.103434 · Cited by 118
McKay, F. H., Slykerman, S., & Dunn, M. (2018) · JMIR mhealth and uhealth · doi:10.2196/11130 · Cited by 117
“This allows for rating of ordinal data, can be used with an unlimited number of raters, and has been found to be superior to Cohen kappa [45,46].”
Hut-Mossel, L., Ahaus, K., Welker, G. A., & Gans, R. O. B. (2021) · PLoS ONE · doi:10.1371/journal.pone.0248677 · Cited by 142
Wei, J., Wei, Y., & Western, A. W. (2016) · Global Environmental Change · doi:10.1016/j.gloenvcha.2016.12.005 · Cited by 124
The social structure of mortgage discrimination
Steil, J., Albright, L., Rugh, J. S., & Massey, D. S. (2017) · Housing Studies · doi:10.1080/02673037.2017.1390076 · Cited by 116
“Intercoder reliability was cross-checked utilizing the ReCal2 Online Utility to compute Krippendorff ’s α on a 10% sample of the 220 texts yielding an inter-rater reliability coefficient of 0.86 (Freelon, 2013; Hayes & Krippendorff, 2007).”
MethodInfluentialTeacher–student interaction and lower secondary school students’ situational engagement
Pöysä, S., Vasalampi, K., Muotka, J. S., Lerkkanen, M., Poikkeus, A., & Nurmi, J. (2018) · British Journal of Educational Psychology · doi:10.1111/bjep.12244 · Cited by 154
Gueler, M. S., & Schneider, S. (2021) · Journal of Business Research · doi:10.1016/j.jbusres.2021.04.061 · Cited by 161
McCluskey, M., & Hmielowski, J. D. (2012) · Journalism · doi:10.1177/1464884911421696 · Cited by 120
“Reliability testing was performed on 120 articles (19% of the 623 articles that constituted the population) using Krippendorff’s alpha, which takes chance agreement into account (see Hayes and Krippendorff, 2007).”
MethodMa, T. (., & Atkin, D. J. (2016) · Telematics and Informatics · doi:10.1016/j.tele.2016.09.009 · Cited by 117
“The Krippendorff’s alpha (KALPHA) statistics (Hayes and Krippendorff, 2007; Hallgren, 2012) rendered a result of Kalpha estimates = 0.84, which indicated high inter-coder reliability (Hallgren, 2012).”
MethodInfluentialMore news from the Euro front: How the press has been framing the Euro crisis in five EU countries
Joris, W., Puustinen, L., & d’Haenens, L. (2018) · International Communication Gazette · doi:10.1177/1748048518754375 · Cited by 116
“Training resulted in high scores of intercoder reliability (Krippendorff’s Alpha) carried out on a sample of 10% of the articles under study (Hayes and Krippendorff, 2007).”
BackgroundLee, N. M., & VanDyke, M. S. (2015) · Science Communication · doi:10.1177/1075547015588600 · Cited by 115
“Krippendorff’s alpha (Hayes & Krippendorff, 2007) for the Facebook categories was above .80 for all categories except “behavior” (.61) and “agency responses” (.79).”
BackgroundSaiewitz, A., & Kida, T. (2017) · Accounting Organizations and Society · doi:10.1016/j.aos.2017.10.002 · Cited by 118
“3 This confirmation bias regarding client responses to audit inquiries echoes earlier findings by Hirst and Koonce (1996), who also found that auditors primarily search for corroborating evidence.”
ResultFanti, S., Minozzi, S., Morigi, J. J., Giesel, F., Ceci, F., Uprimny, C., Hofman, M. S., Eiber, M., Schwarzenbock, S., Castellucci, P., et al. (2017) · European Journal of Nuclear Medicine and Molecular Imaging · doi:10.1007/s00259-017-3725-1 · Cited by 113
Neto, G. C. D. O., Leite, R. R., Shibao, F. Y., & Lucato, W. C. (2016) · Journal of Cleaner Production · doi:10.1016/j.jclepro.2016.08.150 · Cited by 114
Gyllstad, H., & Wolter, B. (2015) · Language Learning · doi:10.1111/lang.12143 · Cited by 113
Dixon, T. L. (2015) · Communication Research · doi:10.1177/0093650215579223 · Cited by 116
“There has been a very intense debate about the assessment of reliability in content analysis (Hayes & Krippendorff, 2007; Krippendorff, 1980, 2004a, 2004b; Wimmer & Dominick, 2005).”
BackgroundThe uncanny valley: No need for any further judgments when an avatar looks eerie
Shin, M., Kim, S. J., & Biocca, F. A. (2019) · Computers in Human Behavior · doi:10.1016/j.chb.2019.01.016 · Cited by 113
The Impact of Forced Answering and Reactance on Answering Behavior in Online Surveys
Sischka, P., Décieux, J. P., Mergener, A., Neufang, K. M., & Schmidt, A. F. (2020) · Social science computer review · doi:10.1177/0894439320907067 · Cited by 114
“About 57.8 % ( n ¼ 251) of the respondents in the NFA condition, who reached the open-ended question gave an answer (i.e., typed signs into the text box), while 97.9 % ( n ¼ 419) of the respondents in the FA condition gave an answer (2.1 % , n ¼ 9 dropped out). vs We coded the open-ended answers in two categories (sem…”
BackgroundInfluentialLike Trainer, Like Bot? Inheritance of Bias in Algorithmic Content Moderation
Binns, R., Veale, M., Kleek, M. G. V., & Shadbolt, N. (2017) · UCL Discovery (University College London) · Cited by 113
High-speed idea filtering with the bag of lemons
Klein, M., & García, A. C. B. (2015) · Decision Support Systems · doi:10.1016/j.dss.2015.06.005 · Cited by 112
The value relevance of risk disclosure in annual reports: Evidence from MENA emerging markets
Moumen, N., Othman, H. B., & Hussainey, K. (2015) · Research in International Business and Finance · doi:10.1016/j.ribaf.2015.02.004 · Cited by 117
Predictors of food decision making: A systematic interdisciplinary mapping (SIM) review
Symmank, C., Mai, R., Hoffmann, S., Stok, F. M., Renner, B., Lien, N., & Rohm, H. (2016) · Appetite · doi:10.1016/j.appet.2016.11.023 · Cited by 111
“Across the eleven stem-categories, mean Krippendorff's alpha indicated acceptable agreement (a ¼ 0.74; see Hayes & Krippendorff, 2007).”
MethodInfluentialInteractional Stancetaking in Online Forums
Kiesling, S. F., Pavalanathan, U., Fitzpatrick, J., Han, X., & Eisenstein, J. (2018) · Computational Linguistics · doi:10.1162/coli_a_00334 · Cited by 120
“Interrater agreement was computed for the 33 threads annotated in phases 1 and 2, using Krippendorff’s α (Hayes and Krippendorff 2007).”
MethodInstitution-wide peer mentoring: Benefits for mentors
Beltman, S., & Schaeben, M. (2012) · Student Success · doi:10.5204/intjfyhe.v3i2.124 · Cited by 110
Service innovation through social robot engagement to improve dementia care quality
Chu, M.-T., Khosla, R., Khaksar, S. M. S., & Nguyen, K. (2016) · Assistive Technology · doi:10.1080/10400435.2016.1171807 · Cited by 118
“Krippendorff’s Alpha (KALPHA) was also used to confirm the inter-coder reliability (Hayes & Krippendorff, 2007).”
BackgroundMethodEmotion in strategic management: A review and future research agenda
Brundin, E. L., Liu, F., & Cyron, T. (2021) · Long Range Planning · doi:10.1016/j.lrp.2021.102144 · Cited by 132
Another brick in the Facebook wall – How personality traits relate to the content of status updates
Winter, S., Neubaum, G., Eimler, S. C., Gordon, V., Theil, J., Herrmann, J., Meinert, J., & Krämer, N. C. (2014) · Computers in Human Behavior · doi:10.1016/j.chb.2014.01.048 · Cited by 117
“We calculated the intercoder reliability for all five raters using Krippendorff’s a (Hayes & Krippendorff, 2007) which reflected an agreement of a = .65.”
MethodScharkow, M. (2011) · Quality & Quantity · doi:10.1007/s11135-011-9545-7 · Cited by 109
A review of treatment adherence measurement methods.
Schoenwald, S. K., & Garland, A. F. (2012) · Psychological Assessment · doi:10.1037/a0029715 · Cited by 108
“Inter-rater reliability amongst coders was calculated using Krippendorff’s Alpha (Hayes & Krippendorff, 2007).”
MethodFinding Frames: Comparing Two Methods of Frame Analysis
David, C. C., Atun, J. M., Legara, E. F., & Monterola, C. P. (2011) · Communication Methods and Measures · doi:10.1080/19312458.2011.624873 · Cited by 108
“Multiple rounds of coding were conducted by two sets of undergraduate student research assistants to achieve intercoder agreement, measured by two measures of reliability Cohen's kappa coefficient (κ) 2 and Krippendorff's alpha (α) (final intercoder agreement measures are in Appendix A) ( Hayes & Krippendorff, 2007 ;…”
MethodInfluentialCOVID-19, a tale of two pandemics: novel coronavirus and fake news messaging
Atehortua, N., & Patino, S. (2020) · Health Promotion International · doi:10.1093/heapro/daaa140 · Cited by 107
“925 using Hayes’ SPSS macro file (Hayes and Krippendorff, 2007) in IBM SPSS Statistics R version 26.”
MethodWhat they can do versus how much they care
Kim, S., & Rader, S. (2010) · Journal of Communication Management · doi:10.1108/13632541011017816 · Cited by 108
“97 using Krippendorff’s alpha (Hayes and Krippendorff, 2007).”
YouEDU: Addressing Confusion in MOOC Discussion Forums by Recommending Instructional Video Clips.
Agrawal, A., Venkatraman, J., Leonard, S., & Paepcke, A. (2015) · Educational Data Mining · Cited by 107
Nostalgia motivates pursuit of important goals by increasing meaning in life
Sedikides, C., Cheung, W., Wildschut, T., Hepper, E. G., Baldursson, E. B., & Pedersen, B. T. (2017) · European Journal of Social Psychology · doi:10.1002/ejsp.2318 · Cited by 117
Hintz, E. A. (2023) · Health Communication · doi:10.1080/10410236.2022.2081046 · Cited by 106
Development of critical spatial thinking through GIS learning
Kim, M., & Bednarz, R. S. (2013) · Journal of Geography in Higher Education · doi:10.1080/03098265.2013.769091 · Cited by 106
“Krippendorff’s a is one of the most generalized measures for inter-rater reliability because it can be used for any scale of measurement, any number of graders, and any data-set with or without missing values (Hayes & Krippendorff, 2007).”
MethodContent Analysis, Quantitative
Coe, K., & Scacco, J. M. (2017) · The International Encyclopedia of Communication Research Methods · doi:10.1002/9781118901731.iecrm0045 · Cited by 106
Hartmann, T., Krakowiak, K. M., & Tsay‐Vogel, M. (2014) · Communication Monographs · doi:10.1080/03637751.2014.922206 · Cited by 105
“Similarly, Heintz-Knowles et al. (2001) observed that most virtual victims appeared comparatively unaffected by the aggressive acts committed against them.”
BackgroundHall, J. A., & Pennington, N. (2013) · Computers in Human Behavior · doi:10.1016/j.chb.2013.01.001 · Cited by 127
Social Media Policies: Implications for Contemporary Notions of Corporate Social Responsibility
Stohl, C., Etter, M. A., Banghart, S., & Woo, D. (2015) · Journal of Business Ethics · doi:10.1007/s10551-015-2743-9 · Cited by 105
“89 was achieved (Hayes and Krippendorff 2007).”
Heuristic Versus Systematic Processing of Specialist Versus Generalist Sources in Online Media
Koh, Y. J., & Sundar, S. S. (2010) · Human Communication Research · doi:10.1111/j.1468-2958.2010.01370.x · Cited by 104
Cynics All Around? The Impact of Election News on Political Cynicism in Comparative Perspective
Schuck, A. R. T., Boomgaarden, H. G., & de Vreese, C. H. (2013) · Journal of Communication · doi:10.1111/jcom.12023 · Cited by 104
“First, in response to recent calls for a standard measure of intercoder reliability we report Krippendorff’s alpha scores for our content analytic indicators (Hayes & Krippendorff, 2007).”
BackgroundFree from demotivation in EFL writing: the use of online flipped writing instruction
Wu, W.-C. V., Yang, J. C., Hsieh, J. C., & Yamamoto, T. (2019) · Computer Assisted Language Learning · doi:10.1080/09588221.2019.1567556 · Cited by 129
“With the inter-rater reliability measured with Krippendorff’s alpha at .85, the evaluation results yielded good reliability (Hayes & Krippendorff, 2007).”
MethodWilkins‐Yel, K. G., Arnold, A., Bekki, J., Natarajan, M., Bernstein, B. L., & Randall, A. K. (2022) · Sex Roles · doi:10.1007/s11199-021-01262-1 · Cited by 123
“To establish interrater reliability between the coding members of the research team, Krippendorff’s alpha ( α ) was calculated based on coding a subset of three transcripts using the KALPHA SPSS macro (Hayes & Krippendorff, 2007).”
MethodDifferential Callosal Contributions to Bimanual Control in Young and Older Adults
Fling, B. W., Walsh, C. M., Bangert, A. S., Reuter‐Lorenz, P. A., Welsh, R. C., & Seidler, R. D. (2010) · Journal of Cognitive Neuroscience · doi:10.1162/jocn.2010.21600 · Cited by 104
“The 180-msec phase delay was chosen because it is among the most difficult patterns to maintain (Semjen & Ivry, 2001).”
MethodEntitled vengeance: A meta‐analysis relating narcissism to provoked aggression
Rasmussen, K. R. (2015) · Aggressive Behavior · doi:10.1002/ab.21632 · Cited by 104
Jiang, M. Y., Jong, M. S., Lau, W. W. F., Chai, C. S., Liu, K. S.-X., & Park, M. (2020) · Computer Assisted Language Learning · doi:10.1080/09588221.2020.1789171 · Cited by 118
“The coding process was conducted by two of the co-authors of the present work and the inter-coder reliability was estimated via Krippendorff ’ s a (Hayes & Krippendorff, 2007).”
MethodContributing to Theory and Knowledge in Quantitative Communication Science
Slater, M. D., & Gleason, L. S. (2012) · Communication Methods and Measures · doi:10.1080/19312458.2012.732626 · Cited by 103
“We assessed 50% of the sample for reliability using Krippendorff’s alpha (Hayes & Krippendorff, 2007) with two coders.”
MethodKapler, I. V., Weston, T., & Wiseheart, M. (2015) · Learning and Instruction · doi:10.1016/j.learninstruc.2014.11.001 · Cited by 101
Grani, G., Lamartina, L., Ascoli, V., Bosco, D., Nardi, F., D’Ambrosio, F., Rubini, A., Giacomelli, L., Biffoni, M., Filetti, S., et al. (2016) · Endocrine · doi:10.1007/s12020-016-1148-6 · Cited by 102
A Meta-Analysis of Uncertainty and Information Management in Illness Contexts
Kuang, K., & Wilson, S. R. (2017) · Journal of Communication · doi:10.1111/jcom.12299 · Cited by 102
Sorting through search results: A content analysis of HPV vaccine information online
Madden, K., Nan, X., Briones, R., & Waks, L. (2011) · Vaccine · doi:10.1016/j.vaccine.2011.10.025 · Cited by 101
The ‘Eudaimonic Experience’: A Scoping Review of the Concept in Digital Games Research
Daneels, R., Bowman, N. D., Possler, D., & Mekler, E. D. (2021) · Media and Communication · doi:10.17645/mac.v9i2.3824 · Cited by 101
“For completeness, we report Krippendorff’s alpha statistics with 10,000 boot-strapped samples (see Hayes & Krippendorff, 2007), noting that for Pair 2 we see a ‘paradox of reliability’ in that a low overall frequency of inclusion codes ( ∼ 9”
MethodNeal, T. M. S., Lienert, P., Denne, E., & Singh, J. P. (2022) · Law and Human Behavior · doi:10.1037/lhb0000482 · Cited by 113
A feedback-based model for CSR assessment and materiality analysis
Calabrese, A., Costa, R. M. D. S., & Rosati, F. (2015) · Accounting Forum · doi:10.1016/j.accfor.2015.06.002 · Cited by 100
“Krippendorff’s alpha is considered the most uitable index for reporting overall reliability in content analysis (Hayes & Krippendorff, 2007; Krippendorff, 2004; Lombard t al., 2002; Neuendorf, 2002).”
Vettehen, P. H., Nuijten, K., & Peeters, A. (2008) · Communication Research · doi:10.1177/0093650208315960 · Cited by 102
“After the coding process was finished, Krippendorff’s alphas were calculated for each sensationalist feature and for each pair of coders as a measure of intercoder reliability (Hayes & Krippendorff, 2007).”
MethodReligion and Crime: A Systematic Review and Assessment of Next Steps
Adamczyk, A., Freilich, J. D., & Kim, C. (2017) · Sociology of Religion · doi:10.1093/socrel/srx012 · Cited by 110
How Political Parties Use Group-Based Appeals: Evidence from Britain 1964–2015
Thau, M. (2017) · Political Studies · doi:10.1177/0032321717744495 · Cited by 100
“…was similar to that of other major projects based on human coding of political texts (Baumgartner et al., 2006; Volkens et al., 2013), and inter-coder reliability tests indicate that group-based appeals have been identified reliably (Krippendorff’s alpha = 0.86; see Hayes and Krippendorff, 2007).”
ResultInfluentialSchneider, B., Sharma, K., Cuendet, S., Zufferey, G., Dillenbourg, P., & Pea, R. D. (2018) · International Journal of Computer-Supported Collaborative Learning · doi:10.1007/s11412-018-9281-2 · Cited by 99
Learning Sentiment from Students’ Feedback for Real-Time Interventions in Classrooms
Altrabsheh, N., Cocea, M., & Fallahkhair, S. (2014) · Lecture notes in computer science · doi:10.1007/978-3-319-11298-5_5 · Cited by 98
“625 and Krippendorff’s alpha [12] was 0.”
Supervised sentiment analysis in multilingual environments
Vilares, D., Alonso, M. Á., & Gómez‐Rodríguez, C. (2017) · Information Processing & Management · doi:10.1016/j.ipm.2017.01.004 · Cited by 98
“For inter-annotator agreement we relied on Krippendorff’s alpha coefficient (Hayes and Krippendorff, 2007), obtaining an agreement from 0.”
MethodInfluentialMedia multitasking performance: Role of message relevance and formatting cues in online environments
Srivastava, J. (2013) · Computers in Human Behavior · doi:10.1016/j.chb.2012.12.023 · Cited by 96
“For the free recall of story information, the intercoder reliability was found to be .99 using Krippendorf’s alpha for interval level coding (Hayes & Krippendorff, 2007).”
MethodInfluentialReid, R. C., Dhuffar, M. K., Parhami, I., & FONG, T. W. (2012) · Journal of Psychiatric Practice · doi:10.1097/01.pra.0000416016.37968.eb · Cited by 97
Psychological Assessments in Legal Contexts: Are Courts Keeping “Junk Science” Out of the Courtroom?
Neal, T. M. S., Slobogin, C., Saks, M. J., Faigman, D. L., & Geisinger, K. F. (2019) · Gothic.net · doi:10.1177/1529100619888860 · Cited by 115
“We used a robust and general reliability statistic called Krippendorff’s α (Kα), which conservatively generates reliability estimates for judgments with any number of raters, at any level of measurement, across sample sizes, and with or without missing data (Hayes & Krippendorff, 2007).”
MethodKumar, K., Kumari, R., Poonia, A., & Kumar, R. (2021) · Journal of financial reporting & accounting · doi:10.1108/jfra-01-2021-0023 · Cited by 97
Cognitive behavioral therapy for chronic pain is effective, but for whom?
Broderick, J. E., Keefe, F. J., Schneider, S., Junghaenel, D. U., Bruckenthal, P., Schwartz, J. E., Kaell, A. T., Caldwell, D. S., McKee, D., & Gould, E. (2016) · Pain · doi:10.1097/j.pain.0000000000000626 · Cited by 94
“79) and Krippendorff's [26] ordinal alpha = 0.”
Dylko, I. B., Beam, M. A., Landreville, K. D., & Geidner, N. W. (2011) · New Media & Society · doi:10.1177/1461444811428899 · Cited by 95
Delivering Behaviour Change Interventions: Development of a Mode of Delivery Ontology
Marques, M. M., Carey, R., Norris, E., Evans, F., Finnerty, A. N., Hastings, J., Jenkins, E., Johnston, M., West, R., & Michie, S. (2020) · Wellcome Open Research · doi:10.12688/wellcomeopenres.15906.2 · Cited by 113
Curtó-Pagès, F., Rivera, E. O., Durán, M. C., & Jané‐Llopis, E. (2021) · Sustainability · doi:10.3390/su13031178 · Cited by 105
“Krippendorff’s alpha was used to assess the inter-coder reliability of the coding process [94], in 2019 yielding values above 0.”
MethodThe sociology of scientific validity: How professional networks shape judgement in peer review
Teplitskiy, M., Acuna, D., Elamrani-Raoult, A., Körding, K., & Evans, J. (2018) · Research Policy · doi:10.1016/j.respol.2018.06.014 · Cited by 113
Historicizing New Media: A Content Analysis of Twitter
Humphreys, L., Gill, P., Krishnamurthy, B., & Newbury, E. (2013) · Journal of Communication · doi:10.1111/jcom.12030 · Cited by 94
Does Media Violence Predict Societal Violence? It Depends on What You Look at and When
Ferguson, C. J. (2014) · Journal of Communication · doi:10.1111/jcom.12129 · Cited by 94
“Interrater reliability was calculated using the Krippendorff formula (Hayes & Krippendorff, 2007).”
MethodMeditation interventions efficiently reduce cortisol levels of at-risk samples: A meta-analysis
Koncz, A., Demetrovics, Z., & Takacs, Z. K. (2021) · Health Psychology Review · doi:10.1080/17437199.2020.1760727 · Cited by 94
“The Krippendorff’s alpha values of interrater reliability (Krippendorff, 1980) were calculated with the KALPHA macro for SPSS (Hayes & Krippendorff, 2007).”
MethodSelf-mocking crisis strategy on social media: Focusing on Alibaba chairman Jack Ma in China
Kim, S., Zhang, X. A., & Zhang, B. W. (2016) · Public Relations Review · doi:10.1016/j.pubrev.2016.10.004 · Cited by 93
Learning From News on Different Media Platforms: An Eye-Tracking Experiment
Kruikemeier, S., Lecheler, S. K., & Boyer, M. M. (2017) · Political Communication · doi:10.1080/10584609.2017.1388310 · Cited by 93
“Krippendorff’s alpha and Fretwurst’s lotus were used to assess reliability (Fretwurst, 2015; Hayes & Krippendorff, 2007).”
MethodMeeting planners' online reviews of destination hotels: A twofold content analysis approach
Boo, S., & Busser, J. A. (2017) · Tourism Management · doi:10.1016/j.tourman.2017.11.014 · Cited by 93
Vitolla, F., Raimo, N., Rubino, M., & Garegnani, G. M. (2021) · Journal of International Management · doi:10.1016/j.intman.2021.100823 · Cited by 104
Social Identity Threat Motivates Science-Discrediting Online Comments
Nauroth, P., Gollwitzer, M., Bender, J., & Rothmund, T. (2015) · PLoS ONE · doi:10.1371/journal.pone.0117476 · Cited by 91
“The inter-rater reliability, assessed by Krippendorff’s α [48], was satisfactory in all categories, all αs> .71 [49].”
MethodPlaying Princess: Preschool Girls’ Interpretations of Gender Stereotypes in Disney Princess Media
Golden, J. C., & Jacoby, J. W. (2017) · Sex Roles · doi:10.1007/s11199-017-0773-8 · Cited by 92
The Behaviour Change Technique Ontology: Transforming the Behaviour Change Technique Taxonomy v1
Marques, M. M., Wright, A. J., Corker, E., Johnston, M., West, R., Hastings, J., Zhang, L., & Michie, S. (2023) · Wellcome Open Research · doi:10.12688/wellcomeopenres.19363.2 · Cited by 214
Identifying robust and sensitive frequency bands for interrogating neural oscillations
Shackman, A. J., McMenamin, B. W., Maxwell, J. S., Greischar, L. L., & Davidson, R. J. (2010) · NeuroImage · doi:10.1016/j.neuroimage.2010.03.037 · Cited by 92
Riding the Stagecoach to Hell: A Qualitative Analysis of Racial Discrimination in Mortgage Lending
Massey, D. S., Rugh, J. S., Steil, J. P., & Albright, L. (2016) · City & Community · doi:10.1111/cico.12179 · Cited by 150
Checking the Fact-Checkers in 2008: Predicting Political Ad Scrutiny and Assessing Consistency
Amazeen, M. A. (2014) · Journal of Political Marketing · doi:10.1080/15377857.2014.959691 · Cited by 96
The media accountability of independent regulatory agencies
Maggetti, M. (2011) · European Political Science Review · doi:10.1017/s1755773911000208 · Cited by 94
“Before the analysis, the consistency of the coding procedure was examined through the recoding of a random sample of articles by another researcher to establish intercoder reliability with Krippendorff’s Alpha (Krippendorff, 2004), using the SPSS macro developed by Hayes (Hayes and Krippendorff, 2007).”
MethodService innovation of cold chain logistics service providers: A multiple-case study in China
Dai, J., Che, W., Lim, J. J., & Shou, Y. (2019) · Industrial Marketing Management · doi:10.1016/j.indmarman.2019.08.002 · Cited by 91
Butijn, B.-J., Tamburri, D. A., & van den Heuvel, W. (2020) · ACM Computing Surveys · doi:10.1145/3369052 · Cited by 124
Effects of attitudes and behaviours on learning mathematics with computer tools
Reed, H. C., Drijvers, P. H. M., & Kirschner, P. A. (2009) · Computers & Education · doi:10.1016/j.compedu.2009.11.012 · Cited by 91
“Rating accuracy was confirmed for two cases by a second researcher, with a Krippendorff’s alpha coefficient for ordinal data (Hayes & Krippendorff, 2007) of .85 indicating good inter-rater reliability.”
MethodOlson, J. S., Wang, D., Olson, G. M., & Zhang, J. (2017) · ACM Transactions on Computer-Human Interaction · doi:10.1145/3038919 · Cited by 101
Gerrath, M. H. E. E., & Usrey, B. (2020) · International Journal of Research in Marketing · doi:10.1016/j.ijresmar.2020.09.010 · Cited by 95
“Overall, the inter-rater reliability for review valence and incentivization acceptance motives was satisfactory ( K-alpha valence = 0.88, K-alpha motives = 0.81; Hayes & Krippendorff, 2007).”
Influential
Metadata
- Authors
- Hayes, A. F., & Krippendorff, K.
- Affiliations
- The Ohio State University (US)
- University of Pennsylvania (US)
- Venue
- Communication Methods and Measures
- Publisher
- Informa UK Limited
- Published
- 2007-04
- Volume
- 1
- Issue
- 1
- Pages
- 77-89
- ISSN
- 1931-2458 (ISSN-L)
- 1931-2466
- OpenAlex
- W2061504941
- Cited by
- 4,225
- References
- 30
- Citations by year
- 2026: 284 · 2025: 278 · 2024: 280 · 2023: 289 · 2022: 331 · 2021: 379 · 2020: 369 · 2019: 296 · 2018: 260 · 2017: 300 · 2016: 248 · 2015: 205 · 2014: 206 · 2013: 150 · 2012: 107
- FWCI
- 50.07
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Communication Methods and Measures (publishedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Reliability and Agreement in Measurement
- Social Sciences › General Social Sciences › Computational and Text Analysis Methods
- Social Sciences › Decision Sciences › Management Science and Operations Research › Psychometric Methodologies and Testing
- Keywords
- Krippendorff's alpha, inter-rater reliability, content analysis, coding reliability, reliability coefficients, observer agreement, SAS macro, missing data
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
