The DOI matches its Crossref record; 30 of the 30 references it cites were found in a registry or an index.
Answering the Call for a Standard Reliability Measure for Coding Data
Abstract
In content analysis and similar methods, data are typically generated by trained human observers who record or transcribe textual, pictorial, or audible matter in terms suitable for analysis. Conclusions from such data can be trusted only after demonstrating their reliability. Unfortunately, the content analysis literature is full of proposals for so-called reliability coefficients, leaving investigators easily confused, not knowing which to choose. After describing the criteria for a good measure of reliability, we propose Krippendorff's alpha as the standard reliability measure. It is general in that it can be used regardless of the number of observers, levels of measurement, sample sizes, and presence or absence of missing data. To facilitate the adoption of this recommendation, we describe a freely available macro written for SPSS and SAS to calculate Krippendorff's alpha and illustrate its use with a simple example.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“We calculate Krippendorff's alpha (Hayes and Krippendorff, 2007) to measure inter-rater reliability (see Table 3).”
From LLMs cannot find reasoning errors, but can correct them given the error location (2024) · cites this paper as Hayes and Krippendorff, 2007
In the citing paper
“Based on Krippendorff's alpha (α = 0.92), the level of agreement between coders was high and well above the threshold of 0.70 (Hayes & Krippendorff, 2007).”
From Collaboration, Coordination, and Cooperation Among Organizations: Establishing the Distinctive Meanings of These Terms Through a Systematic Literature Review (2020) · cites this paper as Hayes & Krippendorff, 2007
In the citing paper
“we calculated the inter-coder reliability based on 25% of all data units and derived a kappa value of .96, which reflects excellent reliability.”
In the citing paper
“To assess the inter-annotator agreement of stage 1, we report Krippendorf's α (Hayes and Krippendorff, 2007) for each dimension for the 135 articles after training and before any discussion/adjudication: economic (0.44), foreign (0.68), social (0.39).”
From Political Ideology and Polarization: A Multi-dimensional Approach (2022) · cites this paper as Hayes and Krippendorff, 2007
In the citing paper
“This family of metrics does not correct for chance (Cohen, 1960;Hayes and Krippendorff, 2007) and overstates discriminative ability (Guerdan et al., 2025;Han et al., 2025;Collot et al., 2025).”
From Reliability without Validity: A Systematic, Large-Scale Evaluation of LLM-as-a-Judge Models Across Agreement, Consistency, and Bias (2026) · cites this paper as Hayes and Krippendorff, 2007
Citations
Cites
Coefficient Alpha and the Internal Structure of Tests
Cronbach, L. J. (1951) · Psychometrika · doi:10.1007/bf02310555 · Cited by 43,931
A Coefficient of Agreement for Nominal Scales
Cohen, J. (1960) · Educational and Psychological Measurement · doi:10.1177/001316446002000104 · Cited by 42,398
An Introduction to the Bootstrap
Efron, B., & Tibshirani, R. J. (1994) · doi:10.1201/9780429246593 · Cited by 40,500
Nonparametric Statistics for the Behavioral Sciences
Guitton, H., & Siegel, S. (1958) · Revue économique · doi:10.2307/3498751 · Cited by 35,279
Content Analysis: An Introduction to its Methodology.
Shelley, M., & Krippendorff, K. (1984) · Journal of the American Statistical Association · doi:10.2307/2288384 · Cited by 24,346
Self-citationStatistical Methods for Rates and Proportions
Fleiss, J. L., Levin, B., & Paik, M. C. (2003) · Wiley Series in Probability and Statistics · doi:10.1002/0471445428 · Cited by 18,646
Measuring nominal scale agreement among many raters.
Fleiss, J. L. (1971) · Psychological Bulletin · doi:10.1037/h0031619 · Cited by 8,733
Content Analysis for the Social Sciences and Humanities.
Parker, E. B., & Holsti, O. R. (1970) · American Sociological Review · doi:10.2307/2093233 · Cited by 4,553
Content Analysis in Mass Communication: Assessment and Reporting of Intercoder Reliability
Lombard, M., Snyder-Duch, J., & Bracken, C. C. (2002) · Human Communication Research · doi:10.1111/j.1468-2958.2002.tb00826.x · Cited by 3,068
An Introduction to the Bootstrap
Grimshaw, S. D., Efron, B., & Tibshirani, R. J. (1995) · Technometrics · doi:10.2307/1269918 · Cited by 2,281
Research works printed but not linked: 1
- 10.1111/j.1468-2958.2004.tb00738.x · DOI · Paper page
Cited by
Sorted by citations
Items 501–600 of 1,013
Exploring the intellectual cores of the blockchain-Internet of Things (BIoT)
Tsang, Y.-P., Wu, C.-H., Ip, A. W. H., & Shiau, W.-L. (2021) · Journal of Enterprise Information Management · doi:10.1108/jeim-10-2020-0395 · Cited by 57
A Critical Review of Case Studies on Dissociative Amnesia
Mangiulli, I., Otgaar, H., Jelícic, M., & Merckelbach, H. L. G. J. (2021) · Clinical Psychological Science · doi:10.1177/21677026211018194 · Cited by 63
“We computed interrater reliability statistics using Krippendorff’s α (K α ; Hayes & Krippendorff, 2007).”
On the choice of measures of reliability and validity in the content-analysis of texts
Олейник, А., Popova, I., Kirdina, S., & Shatalova, T. (2013) · Quality & Quantity · doi:10.1007/s11135-013-9919-0 · Cited by 64
Statistical affect detection in collaborative chat
Brooks, M., Kuksenok, K., Torkildson, M. K., Perry, D., Robinson, J. J., Scott, T. J., Anicello, O., Zukowski, A., Harris, P. G., & Aragon, C. (2013) · ACM Conference on Computer Supported Cooperative Work (CSCW) · doi:10.1145/2441776.2441813 · Cited by 60
“Although Cohen’s kappa has known weaknesses [13], it is also widely understood, so we developed a way to compute a version of kappa over our coded data.”
BackgroundMethodTsandilas, T. (2018) · ACM Transactions on Computer-Human Interaction · doi:10.1145/3182168 · Cited by 60
Sarris, J. J., Byrne, G. J., Stough, C. K. K., Bousman, C., Mischoulon, D., Murphy, J., MacDonald, P., Adams, L., Nazareth, S. M., Oliver, G., et al. (2018) · Journal of Affective Disorders · doi:10.1016/j.jad.2018.11.092 · Cited by 60
What does safety commitment mean to leaders? A multi-method investigation
Fruhen, L. S., Griffin, M. A., & Andrei, D. M. (2018) · Journal of Safety Research · doi:10.1016/j.jsr.2018.12.011 · Cited by 68
“Sufficient inter-rater agreement of α = 0.79 (95% CI LL 0.72 to CI UL 0.86) was achieved (computed with a syntax by Hayes & Krippendorff, 2007).”
MethodInfluentialRetractedOhashi, R., Martignoni, G., Hartmann, A., Caliò, A., Segala, D., Stöhr, C., Wach, S., Erlmeier, F., Weichert, W., Autenrieth, M. E., et al. (2019) · Archiv für Pathologische Anatomie und Physiologie und für Klinische Medicin · doi:10.1007/s00428-019-02710-w · Cited by 59
How to Choose a Measure of Resilience: An Organizing Framework for Resilience Measurement
Fisher, D. M., & Law, R. D. (2020) · Applied Psychology · doi:10.1111/apps.12243 · Cited by 74
Gossip drives vicarious learning and facilitates social connection
Jolly, E., & Chang, L. J. (2021) · Current Biology · doi:10.1016/j.cub.2021.03.090 · Cited by 67
Edelmann, F., Wachter, R., Duvinage, A., Mueller, S., Fegers‐Wustrow, I., Schwarz, S., Christle, J. W., Pieske‐Kraigher, E., Seyfarth, M., Knapp, M., et al. (2025) · Nature Medicine · doi:10.1038/s41591-024-03342-7 · Cited by 64
Drowning in the shallows: an Australian study of the PhD experience of wellbeing
Beasy, K. M., Emery, S., & Crawford, J. (2019) · Teaching in Higher Education · doi:10.1080/13562517.2019.1669014 · Cited by 97
“Two of the research team independently inductively coded a sample of responses to gain a picture of emergent themes within participants’ experiences and to enable preliminary reliability within the analysis procedure (Hayes and Krippendorff 2007).”
MethodPerformance assessment in full-scale simulators – A case of maritime pilotage operations
Ernstsen, J., & Nazir, S. (2020) · Safety Science · doi:10.1016/j.ssci.2020.104775 · Cited by 63
England, L. J., Tong, V. T., Koblitz, A., Kish-Doto, J., Lynch, M. M., & Southwell, B. G. (2016) · Preventive Medicine Reports · doi:10.1016/j.pmedr.2016.09.002 · Cited by 58
“We calculated Krippendorff's alpha (Hayes and Krippendorff, 2007) for each category and found adequate agreement with alpha exceeding 0.”
MethodInfluentialWinkel, A. F., Telzak, B., Shaw, J., Hollond, C., Magro, J., Nicholson, J., & Quinn, G. P. (2021) · Journal of General Internal Medicine · doi:10.1007/s11606-021-06836-z · Cited by 66
Awuah, B., Yazdifar, H., & Elbardan, H. (2023) · Journal of Accounting & Organizational Change · doi:10.1108/jaoc-10-2022-0155 · Cited by 59
Towards LLM-based Fact Verification on News Claims with a Hierarchical Step-by-Step Prompting Method
Zhang, X., & Gao, W. (2023) · International Joint Conference on Natural Language Processing and Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics (IJCNLP-AACL) · doi:10.18653/v1/2023.ijcnlp-main.64 · Cited by 130
Displays of dabbing marijuana extracts on YouTube
Krauss, M. J., Sowles, S. J., Mylvaganam, S., Zewdie, K., Bierut, L. J., & Cavazos‐Rehg, P. A. (2015) · Drug and Alcohol Dependence · doi:10.1016/j.drugalcdep.2015.08.020 · Cited by 58
“Regarding reliability, the median Krippendorff's alpha across codes was 0.75 (range 0.47 to 1.0; the low of 0.47 was for the use of some other method of dabbing, which had low prevalence in the sample) (Hayes and Krippendorff, 2007).”
MethodInfluentialA matter of context: A comparison of two types of contextualized personality measures
Holtrop, D., Born, M. P., de Vries, A., & de Vries, R. E. (2014) · Personality and Individual Differences · doi:10.1016/j.paid.2014.04.029 · Cited by 68
Retrieval Consistency in the Presence of Query Variations
Bailey, P., Moffat, A., Scholer, F., & Thomas, P. (2017) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3077136.3080839 · Cited by 58
“We assess how similar the relative ordering of systems is using one-way Intraclass Correlation [4], Kendall’s Coecient of Concordance [21] (commonly wrien as Kendall’sW ), and Krippendor’s α [17].”
MethodLeadership under Fire: How Governments Manage Crisis Communication
Liu, B. F., Iles, I. A., & Herovic, E. (2019) · Communication Studies · doi:10.1080/10510974.2019.1683593 · Cited by 66
“An inter-coder reliability coefficient, Krippendorff’s alpha, was computed using the KALPHA macro in the statistical software SPSS (Hayes & Krippendorff, 2007).”
MethodSustainability Accounting and Reporting in the Public Sector: Towards Public Value Co-Creation?
Tommasetti, A., Mussari, R., Maione, G., & Sorrentino, D. (2020) · Sustainability · doi:10.3390/su12051909 · Cited by 57
“’s S”, Scott’s Pi, Fleiss’ K, Cohen’s κ), the Krippendorff’s α (K-alpha) [70,71] has been employed in this article, since it represents the most suitable parameter to evaluate the methodology reliability in the SLRs [37].”
MethodThe Attractiveness of Adaptive Heritage Reuse: A Theoretical Framework
Tu, H. (2020) · Sustainability · doi:10.3390/su12062372 · Cited by 68
A Comparison of Three Approaches for Measuring Negative Cognitions for Psychological Reactance
Reynolds‐Tylus, T., Bigsby, E., & Quick, B. L. (2020) · Communication Methods and Measures · doi:10.1080/19312458.2020.1810647 · Cited by 58
Assessing Data Quality: An Approach and An Application
McMann, K. M., Pemstein, D., Seim, B., Teorell, J., & Lindberg, S. I. (2021) · Political Analysis · doi:10.1017/pan.2021.27 · Cited by 57
“Most political scientists are concerned about the quality of the measures they use, yet few scholars rigorously evaluate a chosen measure’s quality prior to incorporating it in substantive research (Herrera and Kapur 2007). This tendency stems not from a lack of tools: the social sciences are replete with methods for…”
Baez‐Navarro, X., Bockstal, M. V., Nawawi, D., Broeckx, G., Colpaert, C., Doebar, S. C., Hogenes, M. C. H., Koop, E. A., Lambein, K., Peeters, D. J. E., et al. (2023) · Modern Pathology · doi:10.1016/j.modpat.2022.100009 · Cited by 59
Guo, X., Yang, Z., Sun, J., & Zhang, Y. (2023) · Expert Systems with Applications · doi:10.1016/j.eswa.2023.122398 · Cited by 60
Web-based Collaborative Inquiry to Bridge Gaps in Secondary Science Education
Raes, A., Schellens, T., & Wever, B. D. (2013) · Journal of the Learning Sciences · doi:10.1080/10508406.2013.836656 · Cited by 57
“Regarding all items, Krippendorff’s alpha ranged from 0.65 to 1, which indicates good to excellent interrater agreement (Hayes & Krippendorff, 2007).”
MethodInfluentialUsing video-cases to assess student reflection: Development and validation of an instrument
Koole, S., Dornan, T., Aper, L., Wever, B. D., Scherpbier, A. J. J. A., Valcke, M., Cohen‐Schotanus, J., & Derese, A. (2012) · BMC Medical Education · doi:10.1186/1472-6920-12-22 · Cited by 61
“Hayes AF, Krippendorff K: Answering the Call for a Standard Reliability Measure for Coding Data.”
MethodInfluentialRevisiting Instructor Misbehaviors: A Revised Typology and Development of a Measure
Goodboy, A. K., & Myers, S. A. (2014) · Communication Education · doi:10.1080/03634523.2014.978798 · Cited by 57
“Based on recommendations by Hayes and Krippendorff (2007), intercoder reliability was calculated and achieved with a Krippendorff’s alpha of 0.94.”
MethodInfluentialDealing with conflicting information from multiple nonlinear texts: Effects of prior attitudes
van Strien, J. L. H., Brand‐Gruwel, S., & Boshuizen, H. P. A. (2013) · Computers in Human Behavior · doi:10.1016/j.chb.2013.11.021 · Cited by 57
“In order to deal with the ordinal nature of the coding scheme, Krippendorff’s alpha (Hayes & Krippendorff, 2007) was calculated as a measure of interrater reliability.”
MethodFraming the Immigration Policy Agenda
Dekker, R., & Scholten, P. W. A. (2017) · The International Journal of Press/Politics · doi:10.1177/1940161216688323 · Cited by 64
“We calculated the Krippendorff’s alpha by use of the SPSS macro developed by Hayes and Krippendorff (2007).”
An analysis of humor orientation on Facebook: A lens model approach
Pennington, N., & Hall, J. A. (2014) · Humor - International Journal of Humor Research · doi:10.1515/humor-2013-0053 · Cited by 57
“Reliability was measured using Krippendorf’s alpha MACRO for SPSS (Hayes and Krippendorf 2007).”
MethodQuinlan, S., Shephard, M. P., & Paterson, L. (2015) · Electoral Studies · doi:10.1016/j.electstud.2015.02.009 · Cited by 56
“To ensure reliability and replication, intra-coder and inter-coder tests (for e.g.: Neuendorf, 2002; Hayes and Krippendorff, 2007; Krippendorff, 2013) involving two other investigators were conducted on all variables with the results blind of the original coder and of the intra-rater test.”
MethodShi, X., Liu, W., & Zhang, J. (2021) · International Journal of Logistics Research and Applications · doi:10.1080/13675567.2021.1988909 · Cited by 56
“0, embedding with a KALPHA macro developed by Hayes and Krippendorff (2007).”
BackgroundMethodMasip, J., & Herrero, C. (2014) · Journal of Communication · doi:10.1111/jcom.12135 · Cited by 61
Efficacy Information in Media Coverage of Infectious Disease Risks
Evensen, D. T. N., & Clarke, C. E. (2011) · Science Communication · doi:10.1177/1075547011421020 · Cited by 56
“We used Krippendorff’s alpha to calculate reliability coefficients, as it can be used regardless of the number of observers, levels of measurement, sample sizes, and the presence of missing data (Hayes & Krippendorff, 2007).”
MethodKing, A. J. (2014) · Health Communication · doi:10.1080/10410236.2013.878778 · Cited by 55
“Krippendorff’s alpha was used to determine intercoder reliability, which is a measure that has been refined and put forward as a standard of communication research (Hayes & Krippendorff, 2007; Krippendorff, 2011).”
MethodBain, P. G., Vaes, J., Kashima, Y., Haslam, N., & Guan, Y. (2011) · Journal of Cross-Cultural Psychology · doi:10.1177/0022022111419029 · Cited by 56
“Inter-rater reliability was excellent overall (Krippendorff’s alpha = .86, Hayes & Krippendorff, 2007).”
MethodHennessy, M. B., McCowan, B., Jiang, J., & Capitanio, J. P. (2014) · Frontiers in Behavioral Neuroscience · doi:10.3389/fnbeh.2014.00309 · Cited by 54
“Behavioral management staff are trained to achieve excellent inter-observer reliability [Krippendorf α coefficient = 1; p (not reaching α-min 0.90) = 0.0000] (Hayes and Krippendorff, 2007).”
BackgroundInfluentialMould‐Millman, N., Rominski, S. D., Bogus, J., Ginde, A. A., Zakariah, A. N., Boatemaah, C. A., Yancey, A. H., Akoriyea, S. K., & Campbell, T. (2015) · Global Health Science and Practice · doi:10.9745/ghsp-d-15-00170 · Cited by 56
“66 to 1, suggesting good to excellent interrater reliability.(22,23) Intrarater and test-retest reliability were non-applicable because the EMS educational intervention (an integral part of the survey mission) would skew respondents’ subsequent answers and introduce error into tests of reliability.”
Background“I Shared the News Today, oh Boy”
Larsson, A. O. (2016) · Journalism Studies · doi:10.1080/1461670x.2016.1154797 · Cited by 56
IT innovation adoption by enterprises: Knowledge discovery through text analytics
Basole, R. C., Seuss, C. D., & Rouse, W. B. (2012) · Decision Support Systems · doi:10.1016/j.dss.2012.10.029 · Cited by 61
Edrich, T., Stopfkuchen-Evans, M. F., Scheiermann, P., Heim, M. H., Chan, W., Stone, M. B., Dankl, D., Aichner, J., Hinzmann, D., Song, P., et al. (2016) · Anesthesia & Analgesia · doi:10.1213/ane.0000000000001383 · Cited by 56
Reliability in content analysis: The case of semantic feature norms classification
Bolognesi, M., Pilgram, R., & van den Heerik, R. A. M. (2016) · Behavior Research Methods · doi:10.3758/s13428-016-0838-6 · Cited by 58
“This measurement, however, does not take into account that the chance that agreement is reached decreases when the number of categories increases (Hayes & Krippendorff, 2007:80), nor that the chance that agreement is reached is affected by overrepresentation of categories (Artstein & Poesio, 2008;…”
BackgroundHossain, M. M., Islam, M. T., Momin, M. A., Nahar, S., & Alam, M. S. (2018) · Journal of Business Ethics · doi:10.1007/s10551-018-3874-6 · Cited by 61
Nagaraj, T. S., Jeyapaul, R., Vimal, K. E. K., & Mathiyazhagan, K. (2019) · Production Planning & Control · doi:10.1080/09537287.2019.1612109 · Cited by 56
“The reliability test was accepted by Krippendorff’s alpha (a¼ 0.794) (Hayes and Krippendorff 2007).”
MethodInfluentialIs a Picture Worth A Thousand Words? An Experiment Comparing Observer-Based Skin Tone Measures
Campbell, M. E., Keith, V. M., Gonlin, V., & Carter‐Sowell, A. R. (2020) · Race and Social Problems · doi:10.1007/s12552-020-09294-0 · Cited by 54
Zacher, M. L., Bayly, M., Brennan, E., Dono, J., Miller, C. L., Durkin, S. J., Scollo, M., & Wakefield, M. A. (2014) · Addiction · doi:10.1111/add.12466 · Cited by 55
Paige, S. R., Stellefson, M., Chaney, B. H., & Alber, J. M. (2015) · American Journal of Health Education · doi:10.1080/19325037.2015.1044586 · Cited by 55
“All data analyses were conducted with SPSS 22.0.61 Krippendorff’s alpha (a) was calculated using a KALPHA macro compatible with SPSS.55 Krippendorff’s a for each variable was above the recommended cutoff value for acceptable intercoder reliability: category of health information on COPD (a ¼ .70), embedded visual comm…”
MethodInfluentialHarsch, C., & Martin, G. (2012) · Assessing Writing · doi:10.1016/j.asw.2012.06.003 · Cited by 55
“As for an acceptable level of agreement, there is no consensus in the literature about what value constitutes a satisfactory level of agreement for any given index (for an overview discussing a range of reliability indices, see e.g. Hayes & Krippendorff, 2007; Stemler, 2004).”
BackgroundRichter, J. N., Hochner, B., & Kuba, M. J. (2015) · Journal of Experimental Biology · doi:10.1242/jeb.115915 · Cited by 55
“The animals showed flexibility in movement control by adapting to the constraint and using an appropriate movement to get to the food reward.”
BackgroundMethodInfluentialKowalewski, T. M., Comstock, B. A., Sweet, R. M., Schaffhausen, C. R., Menhadji, A. D., Averch, T. D., Box, G. N., Brand, T. C., Ferrandino, M. N., Kaouk, J. H., et al. (2016) · The Journal of Urology · doi:10.1016/j.juro.2016.01.005 · Cited by 55
“compared to the mean expert ratings graphically, overall and separately by task, as well as through the use of Cronbach’s alpha statistic.(16,18) Pearson’s R correlation statistic was also used to evaluate the correlation between”
MethodThe Functional Movement Screen
Beardsley, C., & Contreras, B. M. (2014) · Strength and conditioning journal · doi:10.1519/ssc.0000000000000074 · Cited by 55
“Most commonly, the intraclass correlation coefficient was used (63), although Kappa (60) was used twice and the Krippendorff alpha (K alpha) (31) was used once.”
MethodRelational capital disclosure, corporate reporting and company performance: Evidence from Europe
Martini, S., Corvino, A., Doni, F., & Rigolini, A. (2016) · Journal of Intellectual Capital · doi:10.1108/jic-07-2015-0065 · Cited by 62
Student teachers' prior knowledge as prerequisite to learn how to assess pupils' learning strategies
Glogger‐Frey, I., Deutscher, M., & Renkl, A. (2018) · Teaching and Teacher Education · doi:10.1016/j.tate.2018.01.012 · Cited by 63
Room for Improvement: Girls’ and Boys’ Home Environments are Still Gendered
MacPhee, D., & Prendergast, S. (2018) · Sex Roles · doi:10.1007/s11199-018-0936-2 · Cited by 55
“To assess intercoder reliability, we calculated Krippendorff ’s alpha (Kalpha; Hayes and Krippendorff 2007) for each category tally (i.”
MethodPractice variability promotes an external focus of attention and enhances motor skill learning
Chua, L.-K., Dimapilis, M. K., Iwatsuki, T., Abdollahipour, R., Lewthwaite, R., & Wulf, G. (2019) · Human Movement Science · doi:10.1016/j.humov.2019.02.015 · Cited by 55
Kim, N. J., Belland, B. R., Lefler, M., Andreasen, L., Walker, A. E., & Axelrod, D. B. (2019) · Educational Psychology Review · doi:10.1007/s10648-019-09502-3 · Cited by 60
“To identify the level of consistency in coding between the three coders, Krippendorff ’ s alpha was used to assess agreement among the initial codes, as the metric addresses chance agreement and is robust across assessment levels and number of observers (Hayes and Krippendorff 2007) (see Fig.”
MethodOpen Banking: The Emergence of New Digital Business Models
Ramdani, B., Rothwell, B., & Boukrami, E. (2020) · International Journal of Innovation and Technology Management · doi:10.1142/s0219877020500339 · Cited by 59
Hendriks, H., van den Putte, B., & de Bruijn, G. (2013) · Prevention Science · doi:10.1007/s11121-013-0418-2 · Cited by 53
“39 (Hayes and Krippendorff 2007).”
BackgroundMethodInfluentialEffects of bin proximity and informational prompts on recycling and contamination
Rosenthal, S., & Linder, N. (2021) · Resources Conservation and Recycling · doi:10.1016/j.resconrec.2021.105430 · Cited by 54
On the Evaluation of Commit Message Generation Models: An Experimental Study
Tao, W., Wang, Y., Shi, E., Lun, D., Han, S., Zhang, H., Dong-mei, Z., & Wen-qiang, Z. (2021) · IEEE International Conference on Software Maintenance (ICSM) · doi:10.26226/morressier.613b5419842293c031b5b634 · Cited by 62
Success factors in sustainable textile product innovation: An empirical investigation
Rese, A., Baier, D., & Rausch, T. M. (2021) · Journal of Cleaner Production · doi:10.1016/j.jclepro.2021.129829 · Cited by 64
Content Analysis of Video Games: Challenges and Potential Solutions
Schmierbach, M. (2009) · Communication Methods and Measures · doi:10.1080/19312450802458950 · Cited by 61
“Coder training is also central to the issue of reliability, an important topic that receives significant attention in the broader methodological research (Potter & Levine-Donnerstein, 1999; Lombard, Snyder-Duch, & Bracken, 2002; Krippendorff, 2004b; Hayes & Krippendorff, 2007).”
BackgroundTower, Am I Cleared to Land?: Problematic Communication in Aviation Discourse
Howard, J. W. (2008) · Human Communication Research · doi:10.1111/j.1468-2958.2008.00325.x · Cited by 56
Consumers’ health-related perceptions of bread – Implications for labeling and health communication
Sandvik, P., Nydahl, M. C., Kihlberg, I., & Marklinder, I. (2017) · Appetite · doi:10.1016/j.appet.2017.11.092 · Cited by 55
Sampaio, R. C., & Lycarião, D. (2018) · Revista de Sociologia e Política · doi:10.1590/1678-987318266602 · Cited by 54
Calderón, C. A., de la Vega, G., & Herrero, D. B. (2020) · Social Sciences · doi:10.3390/socsci9110188 · Cited by 73
“According to the degree of agreement between both coders, we used Krippendorff’s alpha to test the reliability, as this is the most recommended measure (Hayes and Krippendorff 2007).”
MethodKim, H. (2015) · International Journal of Health Policy and Management · doi:10.15171/ijhpm.2015.157 · Cited by 53
The future of marketing analytics in the sharing economy
Akter, S., Hani, U., Dwivedi, Y. K., & Sharma, A. (2022) · Industrial Marketing Management · doi:10.1016/j.indmarman.2022.04.008 · Cited by 61
Jiang, M. Y., Jong, M. S., Lau, W. W. F., Chai, C. S., & Wu, N. (2022) · Journal of Computer Assisted Learning · doi:10.1111/jcal.12732 · Cited by 60
Are they all the same? An exploratory, categorical analysis of drinking game types
LaBrie, J. W., Ehret, P. J., & Hummer, J. F. (2013) · Addictive Behaviors · doi:10.1016/j.addbeh.2012.12.002 · Cited by 53
The impacts of the Turkish government’s conspiratorial framing of the Gezi Park protests
Nefes, T. S. (2017) · Social movement studies · doi:10.1080/14742837.2017.1319269 · Cited by 53
“In order to establish the reliability of the content analysis (Hayes & Krippendorff, 2007; Lombard, Snyder-Duch, & Bracken, 2002), I invited three colleagues to separately code randomly chosen parts of the material.”
MethodWhat drives risk disclosure in Islamic and conventional banks? An international comparison
Grassa, R., Moumen, N., & Hussainey, K. (2020) · International Journal of Finance & Economics · doi:10.1002/ijfe.2122 · Cited by 56
“We rely on Krippendorff's alpha test, which is the most appropriate test of inter-rater reliability (Hayes & Krippendorff, 2007; Krippendorff, 2010).”
MethodLegal Consciousness: A Systematic Review of its Conceptualization and Measurement Methods1
Horák, F., Lacko, D., & Klocek, A. (2020) · Anuario de Psicología Jurídica · doi:10.5093/apj2021a2 · Cited by 62
“Krippendorff’s alpha (Hayes & Krippendorff, 2007) within the 100 shared articles demonstrated high satisfactory inter- rater reliability (α = .806).”
BackgroundDo ownership structures affect risk disclosure in Islamic banks? International evidence
Grassa, R., Moumen, N., & Hussainey, K. (2020) · Journal of financial reporting & accounting · doi:10.1108/jfra-02-2020-0036 · Cited by 58
van Lacum, E., Ossevoort, M. A., & Goedhart, M. J. (2014) · CBE—Life Sciences Education · doi:10.1187/cbe.13-06-0110 · Cited by 63
“Krippendorff’s alpha (Hayes and Krippendorff, 2007) was 0.”
BackgroundInfluentialCommunication and decision-making in mental health: A systematic review focusing on Bipolar disorder
Fisher, A. C., Manicavasagar, V. L., Kiln, F., & Juraskova, I. (2016) · Patient Education and Counseling · doi:10.1016/j.pec.2016.02.011 · Cited by 53
A practical perspective on repatriate knowledge transfer
Burmeister, A., & Deller, J. (2016) · Journal of Global Mobility The Home of Expatriate Management Research · doi:10.1108/jgm-09-2015-0041 · Cited by 52
Support, reluctance, and production in child abuse investigative interviews.
Blasbalg, U., Hershkowitz, I., & Karni‐Visel, Y. (2018) · Psychology Public Policy and Law · doi:10.1037/law0000183 · Cited by 51
“The K alpha interrater index (Hayes & Krippendorff, 2007) was computed for support, questioning, reluctance, and the production of forensically relevant information, resulting in reliability values of .81, .89, .84, and .83, respectively.”
MethodEliciting Motivation Knowledge from Log Files Towards Motivation Diagnosis for Adaptive Systems
Cocea, M., & Weibelzahl, S. (2007) · Lecture notes in computer science · doi:10.1007/978-3-540-73078-1_23 · Cited by 52
Grinstead, J., Baron, A., Vega‐Mendoza, M., la Mora, J. D., Cantú-Sánchez, M., & Flores, B. (2012) · Journal of Speech Language and Hearing Research · doi:10.1044/1092-4388(2012/11-0289 · Cited by 52
Miller, M. K., Peake, J. S., & Boulton, B. (2010) · Politics & Gender · doi:10.1017/s1743923x10000036 · Cited by 51
Kronsteiner-Gicevic, S., Aftosmes‐Tobio, A., Manganello, J. A., Ganter, C., Simon, C. L., Newlan, S., & Davison, K. K. (2016) · Obesity Reviews · doi:10.1111/obr.12416 · Cited by 51
Li, J., & Noortgate, W. V. D. (2019) · Sociological Methods & Research · doi:10.1177/0049124119882468 · Cited by 53
Tourism brochures: Linking message strategies, tactics and brand destination attributes
Brito, P. Q., & Pratas, J. M. (2014) · Tourism Management · doi:10.1016/j.tourman.2014.10.013 · Cited by 52
Edrich, T., Seethala, R. R., Olenchock, B. A., Mizuguchi, A., Rivero, J. M., Beutler, S. S., Fox, J. A., Liu, X., & Frendl, G. (2013) · Journal of Cardiothoracic and Vascular Anesthesia · doi:10.1053/j.jvca.2013.07.011 · Cited by 51
Coordinating online health communities for cognitive and affective value creation
Oerle, S. V., Mahr, D., & Lievens, A. (2016) · Journal of service management · doi:10.1108/josm-09-2015-0264 · Cited by 54
Braun, S. S., & Davidson, A. J. (2016) · Sex Roles · doi:10.1007/s11199-016-0693-z · Cited by 51
Nguyen, T. T. D. T., Garncarz, T., Ng, F., Dabbish, L., & Dow, S. P. (2017) · ACM Conference on Computer Supported Cooperative Work (CSCW) · doi:10.1145/2998181.2998319 · Cited by 56
“For each article, as we employed 3 raters to rate on an ordinal scale from 1 to 7, we computed Krip-pendorff's α [15] to access interrater reliability.”
MethodAnalyzing consumer goal structure in online group buying: A means-end chain approach
Xiao, L., Guo, Z., & D’Ambra, J. D. (2017) · Information & Management · doi:10.1016/j.im.2017.03.001 · Cited by 53
Don’t You Dare Push Me: How Persuasive Social Media Tactics Shape Customer Engagement
Weiger, W. H., Hammerschmidt, M., & Wetzel, H. A. (2018) · Journal of the Association for Consumer Research · doi:10.1086/698713 · Cited by 55
Chernyak, N., Kang, C., & Kushnir, T. (2019) · Developmental Psychology · doi:10.1037/dev0000670 · Cited by 52
Keller, L., Stötter, J., Oberrauch, A., Kuthe, A., Körfgen, A., & Hüfner, K. (2019) · GAIA - Ecological Perspectives for Science and Society · doi:10.14512/gaia.28.1.10 · Cited by 51
Provoost, S., Ruwaard, J., van Breda, W., Riper, H. M., & Bosse, T. (2019) · Frontiers in Psychology · doi:10.3389/fpsyg.2019.01065 · Cited by 54
“Krippendorff’s alpha can also be used to assess IRR on items with a categorical scale, but contrary to Cohen’s Kappa is able to deal with missing values (Hayes and Krippendorff, 2007).”
MethodThe State of Social Media Policies in Higher Education
Pomerantz, J., Hank, C., & Sugimoto, C. R. (2015) · PLoS ONE · doi:10.1371/journal.pone.0127485 · Cited by 51
The False Positive Problem of Automatic Bot Detection in Social Science ResearchPreprint
Rauchfleisch, A., & Kaiser, J. (2020) · SSRN Electronic Journal · doi:10.2139/ssrn.3565233 · Cited by 53
Reality television and the muscular male ideal
Dallesasse, S. L., & Kluck, A. S. (2013) · Body Image · doi:10.1016/j.bodyim.2013.02.004 · Cited by 52
A Pilot Study of Opinion Summarization in Conversations
Wang, D., & Liu, Y. (2011) · Cited by 50
Metadata
- Authors
- Hayes, A. F., & Krippendorff, K.
- Affiliations
- The Ohio State University (US)
- University of Pennsylvania (US)
- Venue
- Communication Methods and Measures
- Publisher
- Informa UK Limited
- Published
- 2007-04
- Volume
- 1
- Issue
- 1
- Pages
- 77-89
- ISSN
- 1931-2458 (ISSN-L)
- 1931-2466
- OpenAlex
- W2061504941
- Cited by
- 4,225
- References
- 30
- Citations by year
- 2026: 284 · 2025: 278 · 2024: 280 · 2023: 289 · 2022: 331 · 2021: 379 · 2020: 369 · 2019: 296 · 2018: 260 · 2017: 300 · 2016: 248 · 2015: 205 · 2014: 206 · 2013: 150 · 2012: 107
- FWCI
- 50.07
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Communication Methods and Measures (publishedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Reliability and Agreement in Measurement
- Social Sciences › General Social Sciences › Computational and Text Analysis Methods
- Social Sciences › Decision Sciences › Management Science and Operations Research › Psychometric Methodologies and Testing
- Keywords
- Krippendorff's alpha, inter-rater reliability, content analysis, coding reliability, reliability coefficients, observer agreement, SAS macro, missing data
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
