The DOI matches its Crossref record; 30 of the 30 references it cites were found in a registry or an index.
Answering the Call for a Standard Reliability Measure for Coding Data
Abstract
In content analysis and similar methods, data are typically generated by trained human observers who record or transcribe textual, pictorial, or audible matter in terms suitable for analysis. Conclusions from such data can be trusted only after demonstrating their reliability. Unfortunately, the content analysis literature is full of proposals for so-called reliability coefficients, leaving investigators easily confused, not knowing which to choose. After describing the criteria for a good measure of reliability, we propose Krippendorff's alpha as the standard reliability measure. It is general in that it can be used regardless of the number of observers, levels of measurement, sample sizes, and presence or absence of missing data. To facilitate the adoption of this recommendation, we describe a freely available macro written for SPSS and SAS to calculate Krippendorff's alpha and illustrate its use with a simple example.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“We calculate Krippendorff's alpha (Hayes and Krippendorff, 2007) to measure inter-rater reliability (see Table 3).”
From LLMs cannot find reasoning errors, but can correct them given the error location (2024) · cites this paper as Hayes and Krippendorff, 2007
In the citing paper
“Based on Krippendorff's alpha (α = 0.92), the level of agreement between coders was high and well above the threshold of 0.70 (Hayes & Krippendorff, 2007).”
From Collaboration, Coordination, and Cooperation Among Organizations: Establishing the Distinctive Meanings of These Terms Through a Systematic Literature Review (2020) · cites this paper as Hayes & Krippendorff, 2007
In the citing paper
“we calculated the inter-coder reliability based on 25% of all data units and derived a kappa value of .96, which reflects excellent reliability.”
In the citing paper
“To assess the inter-annotator agreement of stage 1, we report Krippendorf's α (Hayes and Krippendorff, 2007) for each dimension for the 135 articles after training and before any discussion/adjudication: economic (0.44), foreign (0.68), social (0.39).”
From Political Ideology and Polarization: A Multi-dimensional Approach (2022) · cites this paper as Hayes and Krippendorff, 2007
In the citing paper
“This family of metrics does not correct for chance (Cohen, 1960;Hayes and Krippendorff, 2007) and overstates discriminative ability (Guerdan et al., 2025;Han et al., 2025;Collot et al., 2025).”
From Reliability without Validity: A Systematic, Large-Scale Evaluation of LLM-as-a-Judge Models Across Agreement, Consistency, and Bias (2026) · cites this paper as Hayes and Krippendorff, 2007
Citations
Cites
Coefficient Alpha and the Internal Structure of Tests
Cronbach, L. J. (1951) · Psychometrika · doi:10.1007/bf02310555 · Cited by 43,931
A Coefficient of Agreement for Nominal Scales
Cohen, J. (1960) · Educational and Psychological Measurement · doi:10.1177/001316446002000104 · Cited by 42,398
An Introduction to the Bootstrap
Efron, B., & Tibshirani, R. J. (1994) · doi:10.1201/9780429246593 · Cited by 40,500
Nonparametric Statistics for the Behavioral Sciences
Guitton, H., & Siegel, S. (1958) · Revue économique · doi:10.2307/3498751 · Cited by 35,279
Content Analysis: An Introduction to its Methodology.
Shelley, M., & Krippendorff, K. (1984) · Journal of the American Statistical Association · doi:10.2307/2288384 · Cited by 24,351
Self-citationStatistical Methods for Rates and Proportions
Fleiss, J. L., Levin, B., & Paik, M. C. (2003) · Wiley Series in Probability and Statistics · doi:10.1002/0471445428 · Cited by 18,646
Measuring nominal scale agreement among many raters.
Fleiss, J. L. (1971) · Psychological Bulletin · doi:10.1037/h0031619 · Cited by 8,733
Content Analysis for the Social Sciences and Humanities.
Parker, E. B., & Holsti, O. R. (1970) · American Sociological Review · doi:10.2307/2093233 · Cited by 4,553
Content Analysis in Mass Communication: Assessment and Reporting of Intercoder Reliability
Lombard, M., Snyder-Duch, J., & Bracken, C. C. (2002) · Human Communication Research · doi:10.1111/j.1468-2958.2002.tb00826.x · Cited by 3,068
An Introduction to the Bootstrap
Grimshaw, S. D., Efron, B., & Tibshirani, R. J. (1995) · Technometrics · doi:10.2307/1269918 · Cited by 2,281
Research works printed but not linked: 1
- 10.1111/j.1468-2958.2004.tb00738.x · DOI · Paper page
Cited by
Sorted by citations
Items 401–500 of 1,013
Progress feedback effects on students' writing mastery goal, self-efficacy beliefs, and performance
Duijnhouwer, H., Prins, F. J., & Stokking, K. M. (2010) · Educational Research and Evaluation · doi:10.1080/13803611003711393 · Cited by 83
“For the ordinal and nominal variables we calculated Krippendorff's alpha (see Table 1 ), because it allows for missing categories (Hayes & Krippendorff, 2007 ).”
MethodLarsson, A. O. (2014) · Information Communication & Society · doi:10.1080/1369118x.2014.967269 · Cited by 71
Wu, Z., & Gros-Louis, J. J. (2014) · Journal of Child Language · doi:10.1017/s0305000914000221 · Cited by 78
Dillard, J. P., Li, R., Meczkowski, E. J., Yang, C. Y., & Shen, L. (2016) · Communication Research · doi:10.1177/0093650216631097 · Cited by 72
“The latter may be assessed by a statistic such as Krippendorff’s alpha (αK; Hayes & Krippendorff, 2007), which would estimate the number of exact matches between dynamic and static values.”
MethodCollins, E. A., Cody, A. M., McDonald, S. E., Nicotera, N., Ascione, F. R., & Williams, J. H. (2017) · Violence Against Women · doi:10.1177/1077801217697266 · Cited by 71
“The CAT tool enables users to upload coded ATLAS.ti documents and compare reliability between raters; Krippendorff’s Alpha (KALPHA) was selected because it is robust in correcting for chance during the coding process involving two or more coders (Hayes & Krippendorff, 2007).”
‘Mum, can I play on the internet?’
Cornish, L. S. (2014) · International Journal of Advertising · doi:10.2501/ija-33-3-437-473 · Cited by 71
12-step facilitation for the dually diagnosed: A randomized clinical trial
Bogenschutz, M. P., Rice, S. L., Tonigan, J. S., Vogel, H. S., Nowinski, J., Hume, D., & Arenella, P. B. (2013) · Journal of Substance Abuse Treatment · doi:10.1016/j.jsat.2013.12.009 · Cited by 70
“Inter-rater reliability of fidelity monitoring was calculated with Krippendorf’s alpha using the KALPHA macro for SPSS (Hayes & Krippendorff, 2007).”
MethodIsrael, A. L., Wong‐Parodi, G., Webler, T., & Stern, P. C. (2015) · Energy Research & Social Science · doi:10.1016/j.erss.2015.05.002 · Cited by 70
Explaining the Relationship Between Media Exposure and Early Adolescents’ Body Image Perceptions
Eyal, K., & Te’eni-Harari, T. (2013) · Journal of Media Psychology Theories Methods and Applications · doi:10.1027/1864-1105/a000094 · Cited by 72
Pettigrew, J., Graham, J. A., Miller-Day, M. A., Hecht, M. L., Krieger, J. L. R., & Shin, Y. (2014) · Prevention Science · doi:10.1007/s11121-014-0459-1 · Cited by 70
“An acceptable level of inter-coder reliability (see Hayes & Krippendorff 2007) was set at 0.80 prior to the outset of coding and was checked four times during the coding process by randomly selecting videos for double coding resulting in the following levels of agreement: 0.94, 0.93, 0.84, and 0.92.”
InfluentialThe effect of abductive reasoning on concept selection decisions
Dong, A., Lovallo, D. P., & Mounarath, R. (2015) · Design Studies · doi:10.1016/j.destud.2014.12.004 · Cited by 70
The Social Network Position of Lead Users
Kratzer, J., Lettl, C. U., Franke, N., & Gloor, P. A. (2015) · Journal of Product Innovation Management · doi:10.1111/jpim.12291 · Cited by 70
“72 (Hayes and Krippendorff, 2007).”
Chen, N.-T. N., Dong, F., Ball‐Rokeach, S. J., Parks, M., & Huang, J. (2012) · New Media & Society · doi:10.1177/1461444811435640 · Cited by 70
“An SPSS macro written by Hayes and Krippendorff (2007) was downloaded to calculate Krippendorff’s Alpha in SPSS Statistics 17.”
MethodWinstone, N., & Boud, D. J. (2018) · Higher Education Research & Development · doi:10.1080/07294360.2018.1532985 · Cited by 89
“Inter-rater reliability was tested using Krippendorff’s alpha (Hayes & Krippendorff, 2007); codes assigned by the two coders were identical for 93% of responses (Krippendorff’s alpha = .95; 95% CI on α [.87, 1.00]).”
MethodInfluentialThe Role of Identification in Giving Sense to Unethical Organizational Behavior
Ploeger, N. A., & Bisel, R. S. (2013) · Management Communication Quarterly · doi:10.1177/0893318912469770 · Cited by 74
“Intercoder reliability was again calculated at the completion of coding and was also acceptable, α = .73 (Hayes & Krippendorff, 2007).”
A Missed Opportunity?: NOAA’s Use of Social Media to Communicate Climate Science
Lee, N. M., VanDyke, M. S., & Cummins, R. G. (2017) · Environmental Communication · doi:10.1080/17524032.2016.1269825 · Cited by 69
“Following recommendations for reliability assessment (Hayes & Krippendorff, 2007), intercoder reliability was assessed using Krippendorff’s alpha.”
MethodA semantics-based measure of emoji similarity
Wijeratne, S., Balasuriya, L., Sheth, A., & Doran, D. (2017) · IEEE/WIC/ACM International Conference on Web Intelligence · doi:10.1145/3106426.3106490 · Cited by 68
Rosenkränzer, F., Hörsch, C., Schuler, S., & Rieß, W. (2017) · International Journal of Science Education · doi:10.1080/09500693.2017.1362603 · Cited by 79
“After a few codings and some adaption of the coding scheme, we reached an acceptable interrater reliability (Krippendorff’s alpha over .80 [Hayes & Krippendorff, 2007]).”
MethodMa, R., & Lam, P. T. I. (2019) · Cities · doi:10.1016/j.cities.2019.03.009 · Cited by 69
Cassia, A. R., Costa, I., da Silva, V. H. C., & Neto, G. C. D. O. (2020) · Technology Analysis and Strategic Management · doi:10.1080/09537325.2020.1714026 · Cited by 69
“The analysis of the articles was conducted in 12 months, and the data collection was developed and conferred independently by three researchers in order to minimise the errors (Hayes and Krippendor ff 2007).”
MethodApplying the cognitive theory of multimedia learning: an analysis of medical animations
Yue, C. L., Kim, J., Ogawa, R. S., Stark, E., & Kim, S. (2013) · Medical Education · doi:10.1111/medu.12090 · Cited by 79
Kim, S., Park, J.-H., & Wertz, E. K. (2010) · Public Relations Review · doi:10.1016/j.pubrev.2010.04.001 · Cited by 69
“Inter-coder reliability ranged from .87 to .97 using Krippendorff’s alpha (Hayes & Krippendorff, 2007).”
MethodÇakır, N. A., Gass, A., Foster, A. N., & Lee, F. J. (2017) · Computers & Education · doi:10.1016/j.compedu.2017.02.002 · Cited by 69
Sznitman, S. R., & Lewis, N. (2015) · International Journal of Drug Policy · doi:10.1016/j.drugpo.2015.01.010 · Cited by 68
Risks and Rewards of Crowdsourcing Marketplaces
Chandler, J., Paolacci, G., & Mueller, P. A. (2013) · Springer eBooks · doi:10.1007/978-1-4614-8806-4_30 · Cited by 73
“Reliability of data with this property can be measured using Krippendorff’s alpha (for SAS and SPSS macros see Hayes & Krippendorff, 2007).”
MethodVariation of SNOMED CT Coding of Clinical Research Concepts among Coding Experts
Andrews, J. E., Richesson, R., & Krischer, J. P. (2007) · Journal of the American Medical Informatics Association · doi:10.1197/jamia.m2372 · Cited by 68
“Krippendorff’s alpha was then figured using the SPSS macro designed by Hayes and Krippendorff.(57) Table 2 shows the results of this analysis, as well as those for the measurement of agreement between pairs of coders (the table also includes results for the other analyses described below).”
MethodAbrantes, A. C. M., Passos, A., Cunha, M. P. E., & Santos, C. M. (2017) · Journal of Business Research · doi:10.1016/j.jbusres.2017.11.005 · Cited by 75
Disturbed Experience of Time in Depression—Evidence from Content Analysis
Vogel, D. H. V., Krämer, K., Schoofs, T., Kupke, C., & Vogeley, K. T. (2018) · Frontiers in Human Neuroscience · doi:10.3389/fnhum.2018.00066 · Cited by 79
“Intercoder agreement was calculated by comparing codes from the coded material from the initial analysis, both inter-coders using Krippendorff ’s alpha (Krippendorff, 2004; Hayes and Krippendorff, 2007).”
MethodMabon, L., & Shih, W.-Y. (2018) · World Development · doi:10.1016/j.worlddev.2018.02.035 · Cited by 68
Viruses Going Viral: Impact of Fear-Arousing Sensationalist Social Media Messages on User Engagement
Ali, K. I., Zain-ul-abdin, K., Li, C., Johns, L. N., Ali, A., & Carcioppolo, N. (2019) · Science Communication · doi:10.1177/1075547019846124 · Cited by 77
Three Decades of Reliability in Communication Content Analyses
Lovejoy, J., Watson, B. R., Lacy, S. R., & Riffe, D. (2016) · Journalism & Mass Communication Quarterly · doi:10.1177/1077699016644558 · Cited by 97
“Krippendorff’s first edition was published in 1980, but because texts are not universally adopted immediately, 1985 was selected as the starting point. Reliability in content analyses was examined through 2014 to provide data on the most current practices. A coding protocol and variable definitions (available on the J…”
BackgroundBelland, B. R., Walker, A. E., & Kim, N. J. (2017) · Review of Educational Research · doi:10.3102/0034654317723009 · Cited by 90
“…of use, were further analyzed using the q test bootstrapping method to examine the probability that the statistics were actually lower than .667 ( Hayes & Krippendorff, 2007). q test results for assessment level coding shows that the chance of obtaining an alpha value below .67 was 3.13%; in…”
MethodInfluentialIdentification of drivers, benefits, and challenges of ISO 50001 through case study content analysis
Fuchs, H., Aghajanzadeh, A., & Therkelsen, P. L. (2020) · Energy Policy · doi:10.1016/j.enpol.2020.111443 · Cited by 67
Automatic Triage of 12‐Lead ECGs Using Deep Convolutional Neural Networks
van de Leur, R. R., Blom, L. J., Gavves, E., Hof, I. E., van der Heijden, J. F., Clappers, N., Doevendans, P. A., Hassink, R. J., & van Es, R. (2020) · Journal of the American Heart Association · doi:10.1161/jaha.119.015138 · Cited by 70
“36,37 40 The first metric estimates the probability to correctly distinguish between all pairs of 2 patients from different categories, where a value of 0.5 denotes random performance and 1 perfect performance.”
BackgroundInfluentialNorris, E., Marques, M. M., Finnerty, A. N., Wright, A. J., West, R., Hastings, J., Williams, P., Carey, R., Kelly, M. P., Johnston, M., et al. (2020) · Wellcome Open Research · doi:10.12688/wellcomeopenres.15904.1 · Cited by 67
Geležėlytė, O., Roberts, N. P., Kvedaraitė, M., Bisson, J. I., Brewin, C. R., Cloître, M., Kairytė, A., Karatzias, T., Shevlin, M. E., & Kazlauskas, E. (2022) · European Journal of Psychotraumatology · doi:10.1080/20008198.2022.2037905 · Cited by 75
The Third-Person Effect in Advertising: A Meta-Analysis
Eisend, M. (2017) · Journal of Advertising · doi:10.1080/00913367.2017.1292481 · Cited by 68
Transparency and the rhetorical use of citations to Robert Yin in case study research
Massaro, M., Dumay, J., & Bagnoli, C. (2019) · Meditari Accountancy Research · doi:10.1108/medar-08-2017-0202 · Cited by 86
Lung Cancer Messages on Twitter: Content Analysis and Evaluation
Sutton, J. N., Vos, S. C., Olson, M. K., Woods, C. L., Cohen, E. L., Gibson, C. B., Phillips, N. E., Studts, J. L., Eberth, J. M., & Butts, C. T. (2017) · Journal of the American College of Radiology · doi:10.1016/j.jacr.2017.09.043 · Cited by 66
Yang, K., & Girgus, J. S. (2018) · Sex Roles · doi:10.1007/s11199-018-0980-y · Cited by 75
“…of the two coders showed acceptable levels of agreement for moderator variables: sample type (Kalpha = .78), scale type (Kalpha = 1.00) country (Kalpha = 1.00), age (Kalpha = .82), sample size for males (Kalpha = .97), and sample size for females (Kalpha = .96) (Hayes and Krippendorff 2007).”
BackgroundInfluentialWu, I., Hsieh, P.-J., & Wu, S.-M. (2022) · The Internet and Higher Education · doi:10.1016/j.iheduc.2022.100871 · Cited by 73
Interrater reliability estimators tested against true interrater reliabilities
Zhao, X., Feng, G. C., Ao, S. H., & Liu, P. L. (2022) · BMC Medical Research Methodology · doi:10.1186/s12874-022-01707-5 · Cited by 93
“The indices were advertised to be “standard” and “global” for “general purpose” [12, 14, 42, 71].”
BackgroundHwang, C. G., Lee, Y., & Diddi, S. (2015) · International Journal of Fashion Design Technology and Education · doi:10.1080/17543266.2014.996917 · Cited by 66
“Main and subthemes were reevaluated throughout the review and coding process (Creswell, 2009; Hayes & Krippendorff, 2007).”
MethodEnhancing student knowledge acquisition from online learning conversations
Eryilmaz, E., van der Pol, J., Ryan, T., Clark, P. M., & Mary, J. (2013) · International Journal of Computer-Supported Collaborative Learning · doi:10.1007/s11412-012-9163-y · Cited by 69
“We checked inter-rater reliability by means of Krippendorff ’ s alpha (Hayes and Krippendorff 2007).”
MethodFrom output to outcome measures in the public sector: a structured literature review
Mas, F. D., Massaro, M., Lombardi, R., & Garlatti, A. (2019) · International journal of organizational analysis · doi:10.1108/ijoa-09-2018-1523 · Cited by 66
Kumar, K., Kumari, R., & Kumar, R. (2021) · Business and Society Review · doi:10.1111/basr.12247 · Cited by 72
Access to Finance: Challenges Faced by Micro, Small, and Medium Enterprises in India
Rajamani, K., Jan, N. A., Subramani, A. K., & Raj, A. N. (2022) · Engineering Economics · doi:10.5755/j01.ee.33.1.27998 · Cited by 65
A taxonomy of anti-vaccination arguments from a systematic literature review and text modelling
Fasce, A., Schmid, P., Holford, D. L., Bates, L., Gurevych, I., & Lewandowsky, S. (2023) · Nature Human Behaviour · doi:10.1038/s41562-023-01644-3 · Cited by 72
Biomedicalization and the public sphere: Newspaper coverage of health and medicine, 1960s–2000s
Hallin, D. C., Brandt, M. R., & Briggs, C. L. (2013) · Social Science & Medicine · doi:10.1016/j.socscimed.2013.07.030 · Cited by 72
“Intercoder reliability as measured by Krippendorff’s a (ordinal), calculated using the SPSS macro described in Hayes and Krippendorff (2007) were .77 each for Investor and Professional/ Scientific, .84 for Patient/Consumer, and .85 for Citizen/Policymaker.”
MethodAngelini, A., Andersen, C. B., Bartoloni, G., Black, F., Bishop, P. W., Doran, H. M., Fedrigo, M., Fries, J. W. U., Goddard, M. J., Goebel, H., et al. (2011) · The Journal of Heart and Lung Transplantation · doi:10.1016/j.healun.2011.05.011 · Cited by 65
“Because Fleiss’s kappa statistics were developed for nominal scales (categories), we also calculated Krippendorff’s alpha statistics, which are appropriate for ordinal data (rank order) but are not category-specific.(9) For both methods, we calculated the bootstrap 95% confidence interval with the percentile method an…”
MethodIdeal Victims and Monstrous Offenders: How the News Media Represent Sexual Predators
DiBennardo, R. (2018) · Socius Sociological Research for a Dynamic World · doi:10.1177/2378023118802512 · Cited by 65
Winschel, J., & Stawinoga, M. (2019) · Management Review Quarterly · doi:10.1007/s11301-019-00154-9 · Cited by 72
“Next, Krippendorff’s α, as a measure of intercoder reliability, was calculated for the two pre-test data sets (Hayes and Krippendorff 2007; Krippendorff 2013), resulting in a score of 0.986.”
MethodInfluentialAntoine, J.-Y., Villaneau, J., & Lefeuvre, A. (2014) · Proceedings of the 14th Conference of the European Chapter of the Association for Computational Linguistics · doi:10.3115/v1/e14-1058 · Cited by 64
“In particular, opposite arguments have been proposed concerning Cohen’s κ (Di Eugenio and Glass 2004, Arstein and Poesio 2008, Hayes, 2007).”
BackgroundEvolution of newspaper coverage of water issues in Australia during 1843–2011
Wei, J., Wei, Y., Western, A. W., Skinner, D., & Lyle, C. (2014) · AMBIO · doi:10.1007/s13280-014-0571-2 · Cited by 64
Kapitány, R., & Nielsen, M. (2016) · Religion Brain & Behavior · doi:10.1080/2153599x.2016.1141792 · Cited by 66
“See Hayes & Krippendorff, 2007).”
Organizational Boundary Regulation Through Social Media Policies
Banghart, S., Etter, M. A., & Stohl, C. (2018) · Management Communication Quarterly · doi:10.1177/0893318918766405 · Cited by 66
“In the final training session, each analyst coded 10 guidelines, and intercoder reliability of .89 was achieved (Hayes & Krippendorff, 2007).”
MethodAttached to dating apps: Attachment orientations and preferences for dating apps
Chin, K. A., Edelstein, R. S., & Vernon, P. A. (2018) · Mobile Media & Communication · doi:10.1177/2050157918770696 · Cited by 81
Tanaka, T., Matsumoto, T., Hayashi, S., Takagi, S., & Kawabata, H. (2019) · Timing & Time Perception · doi:10.1163/22134468-20191150 · Cited by 61
Çakar, K., & Aykol, Ş. (2020) · Cornell Hospitality Quarterly · doi:10.1177/1938965520971281 · Cited by 64
“…suggested to increase the trustworthiness of case study research such as Krippendorff Kappa or Cohen’s Cappa Analysis (Forman & Damschroder, 2007; Hayes & Krippendorff, 2007; Landis & Koch, 1977), using alpha (α) as a reliability coefficient to measure agreement among observers or coders…”
MethodDoes populism go viral? How Italian leaders engage citizens through social media
Bracciale, R., Andretta, M., & Martella, A. (2021) · Information Communication & Society · doi:10.1080/1369118x.2021.1874472 · Cited by 85
“The Krippendorff α calculated for each variable coded by the two coders ranged from 0.68 to 0.82, considered highly satisfactory values according to field norms (Hayes & Krippendorff, 2007).”
MethodInfluentialB2B eWOM on Alibaba: Signaling through online reviews in platform-based social exchange
Tóth, Z., Mrad, M., Itani, O. S., Luo, J., & Liu, M. J. (2022) · Industrial Marketing Management · doi:10.1016/j.indmarman.2022.04.019 · Cited by 69
Reddit in communication research: current status, future directions and best practices
Hintz, E. A., & Betts, T. (2022) · Annals of the International Communication Association · doi:10.1080/23808985.2022.2064325 · Cited by 73
Grande, M., Meffert, E., Schoenberger, E., Jung, S., Frauenrath, T., Huber, W., Hußmann, K., Moormann, M., & Heim, S. (2012) · NeuroImage · doi:10.1016/j.neuroimage.2012.03.087 · Cited by 63
“Inter-rater reliabilitymeasured byKrippendorff's alpha (Krippendorff, 2004; Hayes and Krippendorff, 2007) was high (α≥0.”
MethodInfluentialHow prevention curricula are taught under real‐world conditions
Miller-Day, M. A., Pettigrew, J., Hecht, M. L., Shin, Y., Graham, J., & Krieger, J. L. R. (2013) · Health Education · doi:10.1108/09654281311329259 · Cited by 72
“80 intercoder reliaiblity using Krippendorf’s a coefficient (Hayes and Krippendorff, 2007).”
MethodJensen, J. D. (2011) · Journal of Communication · doi:10.1111/j.1460-2466.2011.01549.x · Cited by 64
Mayasi, Y. M., Helenius, J., McManus, D. D., Goddeau, R., Jun-O’Connell, A. H., Moonis, M., & Henninger, N. (2017) · Journal of Neurology Neurosurgery & Psychiatry · doi:10.1136/jnnp-2016-315457 · Cited by 64
Jahng, M. R., & Lee, N. (2018) · Science Communication · doi:10.1177/1075547017751948 · Cited by 64
“Intercoder reliability was calculated using the SPSS macro for Krippendorf’s alpha (Hayes & Krippendorff, 2007).”
MethodKoenigstorfer, J., Bocarro, J. N., Byers, T., Edwards, M. B., Jones, G. J., & Preuß, H. (2019) · Leisure Studies · doi:10.1080/02614367.2019.1662830 · Cited by 67
Fairness and Decision-making in Collaborative Shift Scheduling Systems
Uhde, A., Schlicker, N., Wallach, D. P., & Hassenzahl, M. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376656 · Cited by 64
“86, [23]) and remaining disagreements were resolved in a follow-up discussion.”
BackgroundA Review of Free iPhone Applications Designed to Target Anxiety and Worry
Kertz, S. J., MacLaren Kelly, J., Stevens, K. T., Schrock, M., & Danitz, S. B. (2017) · Journal of Technology in Behavioral Science · doi:10.1007/s41347-016-0006-y · Cited by 56
Assumptions behind Intercoder Reliability Indices
Zhao, X., Liu, J. S., & Deng, K. (2013) · Annals of the International Communication Association · doi:10.1080/23808985.2013.11679142 · Cited by 113
“Because %-agreement fails to take into account chance agreements, it is often considered “the most primitive” (Cohen, 1960, p. 38) and “fl awed” (Hayes & Krippendorff, 2007, p. 80) indicator of reliability, leading to decades-long efforts to “account for” and “remove” chance agreements…”
BackgroundMethodYanovitzky, I., & Weber, M. S. (2018) · Evidence & Policy · doi:10.1332/174426418x15378680726175 · Cited by 69
Hidden competence: women’s mathematical participation in public and private classroom spaces
Ernest, J. B., Reinholz, D. L., & Shah, N. (2019) · Educational Studies in Mathematics · doi:10.1007/s10649-019-09910-w · Cited by 63
Internet addiction in Gulf countries: A systematic review and meta-analysis
Al-Khani, A. M., Saquib, J., Rajab, A. M., Khalifa, M. A., Almazrou, A., & Saquib, N. (2021) · Journal of Behavioral Addictions · doi:10.1556/2006.2021.00057 · Cited by 70
Colclasure, B. C., Marlier, A., Durham, M. F., Brooks, T. D., & Kerr, M. (2021) · Education Sciences · doi:10.3390/educsci11090556 · Cited by 73
Vandevelde, S., Keer, H. V., Schellings, G. L. M., & van Hout‐Wolters, B. H. A. M. (2015) · Learning and Individual Differences · doi:10.1016/j.lindif.2015.08.027 · Cited by 63
Less Is More: Teachers' Influence during Peer Collaboration.
Lin, T.-J., Jadallah, M., Anderson, R. C., Baker, A. R., Kim, N.-J., Kim, I.-H., Kuo, L.-J., Miller, B. W., Dong, T., & Wu, X. (2014) · Journal of Educational Psychology · doi:10.1037/a0037758 · Cited by 67
Mapping queries to the Linking Open Data cloud: A case study using DBpedia
Meij, E., Bron, M., Laura Hollink, A., Huurnink, B., & de Rijke, M. (2011) · Journal of Web Semantics · doi:10.1016/j.websem.2011.04.001 · Cited by 63
“Moreover, it does not require that each annotator annotate each document [67,68].”
BackgroundComparing holistic and analytic scoring methods: issues of validity and reliability
Harsch, C., & Martin, G. (2012) · Assessment in Education Principles Policy and Practice · doi:10.1080/0969594x.2012.742422 · Cited by 63
Interaction Effect of Leadership and Communication Media on Feedback Positivity in Virtual Teams
Kahai, S. S., Huang, R., & Jestice, R. J. (2012) · Group & Organization Management · doi:10.1177/1059601112462061 · Cited by 63
“The inter-rater reliability for allocations made by the experts, as indicated by Krippendorff’s alpha (Hayes & Krippendorff, 2007), was 0.80.”
MethodInfluentialRaes, A., Schellens, T., Wever, B. D., & Benoit, D. F. (2016) · Computers in Human Behavior · doi:10.1016/j.chb.2015.12.064 · Cited by 70
Harlow, S., & Benbrook, A. (2019) · Information, Communication & Society · doi:10.1080/1369118x.2017.1386705 · Cited by 60
“After two coder training sessions, acceptable inter-coder reliability levels were reached (Hayes & Krippendorff, 2007), with alphas ranging from 0.7 (what kind of celebrity wrote the tweet) and 0.76 (what kind of celebrity was mentioned in the tweet) to 1 (was the tweet written by a celebrity and…”
MethodInfluentialIs Pornography Consumption a Risk Factor for Condomless Sex?
Tokunaga, R. S., Wright, P. J., & Vangeel, L. (2020) · Human Communication Research · doi:10.1093/hcr/hqaa005 · Cited by 69
Deng, T., Ekachai, D. “., & Pokrywczynski, J. V. (2020) · Health Communication · doi:10.1080/10410236.2020.1859725 · Cited by 62
“Intercoder reliability was calculated using Krippendorff’s alpha test in IBM SPSS with 10,000 bootstrap samples (Hayes & Krippendorff, 2007).”
MethodSingh, R. K., Agrawal, S., Sahu, A., & Kazançoğlu, Y. (2021) · The TQM Journal · doi:10.1108/tqm-02-2021-0051 · Cited by 66
Qualitative analysis of visual risk communication on twitter during the Covid-19 pandemic
Sleigh, J., Amann, J., Schneider, M., & Vayena, E. (2021) · BMC Public Health · doi:10.1186/s12889-021-10851-4 · Cited by 62
“The revised codebook was then applied to a subset of 60 tweets by two researchers (JS, JA) independently to establish intercoder reliability [27, 28].”
MethodGutiérrez, W. R., Barber, J. P., & Villar, C. (2022) · European Management Journal · doi:10.1016/j.emj.2022.10.009 · Cited by 62
Zhang, C., & Meadows, C. W. (2012) · International journal of communication · Cited by 62
Do dimensional psychopathology measures relate to creative achievement or divergent thinking?
Zabelina, D. L., Condon, D. M., & Beeman, M. J. (2014) · Frontiers in Psychology · doi:10.3389/fpsyg.2014.01029 · Cited by 73
Smith, F., Carlsson, E. K., Kokkinakis, D., Forsberg, M., Kodeda, K., Sawatzky, R. G., Friberg, F., & Öhlén, J. (2013) · Patient Education and Counseling · doi:10.1016/j.pec.2013.10.009 · Cited by 62
“Inter-rater reliability A translation of the SAM + CAM instrument was used and interrater reliability was examined by computing Krippendorff’s alpha coefficient [43,44] on the subscale level and on the item level.”
MethodSáinz, M., Meneses, J., Pérez, B. S. L., & Fàbregues, S. (2014) · Sex Roles · doi:10.1007/s11199-014-0424-2 · Cited by 62
“As was the case in the present study, this macro makes it possible to compute Krippendorff's alpha reliability estimate for subjective judgments made at any level of measurement, including nominal and ordinal data (Hayes and Krippendorff 2007).”
MethodSchneider, B., Strait, M. K., Muller, L., Elfenbein, S., Shaer, O., & Shen, C. (2012) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/2207676.2208720 · Cited by 62
Merchie, E., & Keer, H. V. (2016) · Contemporary Educational Psychology · doi:10.1016/j.cedpsych.2016.05.005 · Cited by 78
Tenbrink, T. (2020) · Cambridge University Press eBooks · doi:10.1017/9781108525176 · Cited by 66
Plummer, R., Velaniškis, J., de Grosbois, D., Kreutzwiser, R. D., & de Loë, R. (2010) · Environmental Science & Policy · doi:10.1016/j.envsci.2010.05.004 · Cited by 57
When Self Is the Source: Effects of Media Customization on Message Processing
Kang, H., & Sundar, S. S. (2016) · Media Psychology · doi:10.1080/15213269.2015.1121829 · Cited by 62
“The inter-coder agreement for two coders was .94 (Krippendorff ’ s α ; Hayes & Krippendorff, 2007).”
MethodDimensions of Entrepreneurial Success: A Multilevel Study on Stakeholders of Micro-Enterprises
Razmus, W., & Łaguna, M. (2018) · Frontiers in Psychology · doi:10.3389/fpsyg.2018.00791 · Cited by 61
“Agreement among judges was assessed using Krippendorff ’s alpha coefficient, whose values range from 0 to 1; the closer to 1 they are, the higher the inter-rater agreement (Hayes and Krippendorff, 2007).”
MethodResultMcGarrigle, L., Boulton, E., & Todd, C. J. (2020) · BMC Geriatrics · doi:10.1186/s12877-020-01880-6 · Cited by 62
“Inter-rater reliability between two independent raters was assessed for BCT, credibility, and senior friendliness ratings using Krippendorff’s Alpha (Kalpha) [26], and all interpretations were based on mean rater scores.”
MethodHedges, Tag Questions, Message Processing, and Persuasion
Hosman, L. A., & Siltanen, S. A. (2011) · Journal of Language and Social Psychology · doi:10.1177/0261927x11407169 · Cited by 60
Gros-Louis, J. J., & Miller, J. (2017) · Journal of Child Language · doi:10.1017/s0305000917000472 · Cited by 60
Metadata
- Authors
- Hayes, A. F., & Krippendorff, K.
- Affiliations
- The Ohio State University (US)
- University of Pennsylvania (US)
- Venue
- Communication Methods and Measures
- Publisher
- Informa UK Limited
- Published
- 2007-04
- Volume
- 1
- Issue
- 1
- Pages
- 77-89
- ISSN
- 1931-2458 (ISSN-L)
- 1931-2466
- OpenAlex
- W2061504941
- Cited by
- 4,225
- References
- 30
- Citations by year
- 2026: 284 · 2025: 278 · 2024: 280 · 2023: 289 · 2022: 331 · 2021: 379 · 2020: 369 · 2019: 296 · 2018: 260 · 2017: 300 · 2016: 248 · 2015: 205 · 2014: 206 · 2013: 150 · 2012: 107
- FWCI
- 50.07
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Communication Methods and Measures (publishedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Reliability and Agreement in Measurement
- Social Sciences › General Social Sciences › Computational and Text Analysis Methods
- Social Sciences › Decision Sciences › Management Science and Operations Research › Psychometric Methodologies and Testing
- Keywords
- Krippendorff's alpha, inter-rater reliability, content analysis, coding reliability, reliability coefficients, observer agreement, SAS macro, missing data
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
