The DOI matches its Crossref record; 30 of the 30 references it cites were found in a registry or an index.
Answering the Call for a Standard Reliability Measure for Coding Data
Abstract
In content analysis and similar methods, data are typically generated by trained human observers who record or transcribe textual, pictorial, or audible matter in terms suitable for analysis. Conclusions from such data can be trusted only after demonstrating their reliability. Unfortunately, the content analysis literature is full of proposals for so-called reliability coefficients, leaving investigators easily confused, not knowing which to choose. After describing the criteria for a good measure of reliability, we propose Krippendorff's alpha as the standard reliability measure. It is general in that it can be used regardless of the number of observers, levels of measurement, sample sizes, and presence or absence of missing data. To facilitate the adoption of this recommendation, we describe a freely available macro written for SPSS and SAS to calculate Krippendorff's alpha and illustrate its use with a simple example.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“We calculate Krippendorff's alpha (Hayes and Krippendorff, 2007) to measure inter-rater reliability (see Table 3).”
From LLMs cannot find reasoning errors, but can correct them given the error location (2024) · cites this paper as Hayes and Krippendorff, 2007
In the citing paper
“Based on Krippendorff's alpha (α = 0.92), the level of agreement between coders was high and well above the threshold of 0.70 (Hayes & Krippendorff, 2007).”
From Collaboration, Coordination, and Cooperation Among Organizations: Establishing the Distinctive Meanings of These Terms Through a Systematic Literature Review (2020) · cites this paper as Hayes & Krippendorff, 2007
In the citing paper
“we calculated the inter-coder reliability based on 25% of all data units and derived a kappa value of .96, which reflects excellent reliability.”
In the citing paper
“To assess the inter-annotator agreement of stage 1, we report Krippendorf's α (Hayes and Krippendorff, 2007) for each dimension for the 135 articles after training and before any discussion/adjudication: economic (0.44), foreign (0.68), social (0.39).”
From Political Ideology and Polarization: A Multi-dimensional Approach (2022) · cites this paper as Hayes and Krippendorff, 2007
In the citing paper
“This family of metrics does not correct for chance (Cohen, 1960;Hayes and Krippendorff, 2007) and overstates discriminative ability (Guerdan et al., 2025;Han et al., 2025;Collot et al., 2025).”
From Reliability without Validity: A Systematic, Large-Scale Evaluation of LLM-as-a-Judge Models Across Agreement, Consistency, and Bias (2026) · cites this paper as Hayes and Krippendorff, 2007
Citations
Cites
Coefficient Alpha and the Internal Structure of Tests
Cronbach, L. J. (1951) · Psychometrika · doi:10.1007/bf02310555 · Cited by 43,931
A Coefficient of Agreement for Nominal Scales
Cohen, J. (1960) · Educational and Psychological Measurement · doi:10.1177/001316446002000104 · Cited by 42,398
An Introduction to the Bootstrap
Efron, B., & Tibshirani, R. J. (1994) · doi:10.1201/9780429246593 · Cited by 40,500
Nonparametric Statistics for the Behavioral Sciences
Guitton, H., & Siegel, S. (1958) · Revue économique · doi:10.2307/3498751 · Cited by 35,279
Content Analysis: An Introduction to its Methodology.
Shelley, M., & Krippendorff, K. (1984) · Journal of the American Statistical Association · doi:10.2307/2288384 · Cited by 24,351
Self-citationStatistical Methods for Rates and Proportions
Fleiss, J. L., Levin, B., & Paik, M. C. (2003) · Wiley Series in Probability and Statistics · doi:10.1002/0471445428 · Cited by 18,646
Measuring nominal scale agreement among many raters.
Fleiss, J. L. (1971) · Psychological Bulletin · doi:10.1037/h0031619 · Cited by 8,733
Content Analysis for the Social Sciences and Humanities.
Parker, E. B., & Holsti, O. R. (1970) · American Sociological Review · doi:10.2307/2093233 · Cited by 4,553
Content Analysis in Mass Communication: Assessment and Reporting of Intercoder Reliability
Lombard, M., Snyder-Duch, J., & Bracken, C. C. (2002) · Human Communication Research · doi:10.1111/j.1468-2958.2002.tb00826.x · Cited by 3,068
An Introduction to the Bootstrap
Grimshaw, S. D., Efron, B., & Tibshirani, R. J. (1995) · Technometrics · doi:10.2307/1269918 · Cited by 2,281
Research works printed but not linked: 1
- 10.1111/j.1468-2958.2004.tb00738.x · DOI · Paper page
Cited by
Sorted by citations
Items 801–900 of 1,013
Leveraging Passage-level Cumulative Gain for Document Ranking
Wu, Z., Mao, J., Liu, Y., Zhan, J., Zheng, Y., Zhang, M., & Ma, S. (2020) · International World Wide Web Conference (WWW) · doi:10.1145/3366423.3380305 · Cited by 43
Hayman, M., Alfrey, K.-L. R., Cannon, S., Alley, S., Rebar, A. L., Williams, S. L., Short, C. E., Altazan, A. D., Comardelle, N., Currie, S., et al. (2021) · JMIR mhealth and uhealth · doi:10.2196/23649 · Cited by 41
“Interrater reliability [25] for app quality resulted in low reliability (mean Kα 0.”
BackgroundMethodPride Parades and LGBT Movements
Peterson, A., Wahlström, M., & Wennerhag, M. (2018) · Routledge eBooks · doi:10.4324/9781315474052 · Cited by 50
A horizon scan exercise for aquatic invasive alien species in Iberian inland waters
Oficialdegui, F. J., Zamora‐Marín, J. M., Guareschi, S., Anastácio, P., Murillo, P. G., Ribeiro, F., Miranda, R., Cobo, F., Gallardo, B., García‐Berthou, E., et al. (2023) · The Science of The Total Environment · doi:10.1016/j.scitotenv.2023.161798 · Cited by 42
Leary, H. (2021) · Digital Commons - USU (Utah State University) · doi:10.26076/c570-0a4e · Cited by 38
When Sex Doesn't Sell: Using Sexualized Images of Women Reduces Support for Ethical Campaigns
Bongiorno, R., Bain, P. G., & Haslam, N. (2013) · PLoS ONE · doi:10.1371/journal.pone.0083311 · Cited by 39
First-level and second-level intermedia agenda-setting among major news websites
Lim, J. (2011) · Asian Journal of Communication · doi:10.1080/01292986.2010.539300 · Cited by 39
“SPSS macro (Hayes & Krippendorff, 2007 ) calculated Krippendorff's α, which was expressed by α=1 - ( D o / D e ), where D o was the observed disagreement and D e was the disagreement expected by chance (Krippendorff, 2004 ).”
MethodRodríguez‐Medina, J., Antón, L. J. M., Carbonero, M. Á., & Bernal, A. O. (2016) · Frontiers in Psychology · doi:10.3389/fpsyg.2016.01986 · Cited by 48
Kaczmarek, L. D., Behnke, M., Kashdan, T. B., Kusiak, A., Marzec, K., Mistrzak, M., & Włodarczyk, M. (2017) · The Journal of Positive Psychology · doi:10.1080/17439760.2017.1326519 · Cited by 38
“The interrater agreement was satisfactory (Krippendorff’s α = 0.70) (Hayes & Krippendorff, 2007).”
MethodInfluentialUNTANGLING A COMPLEX MEDIA SYSTEM
Moe, H., & Larsson, A. O. (2013) · Information Communication & Society · doi:10.1080/1369118x.2013.783607 · Cited by 38
“Intercoder reliability was assessed as each of the two authors recoded a random 20 percent sample of each other’s coding, using Krippendorf’s alpha (Hayes & Krippendorff 2007) to gage reliability.”
MethodRetail Chains’ Corporate Social Responsibility Communication
Utgård, J. (2015) · Journal of Business Ethics · doi:10.1007/s10551-015-2952-2 · Cited by 38
“I therefore calculated the Krippendorff’s alpha, a measure of inter-coder reliability with many desirable characteristics (Hayes and Krippendorff 2007).”
MethodRecent Ocean Literacy Research in United States Public Schools: Results and Implications
Plankis, B. J., & Marrero, M. E. (2010) · DergiPark (Istanbul University) · doi:10.18497/iejee-green.63556 · Cited by 38
Watson, M. K., Pelkey, J., Noyes, C. R., & Rodgers, M. O. (2016) · Journal of Cleaner Production · doi:10.1016/j.jclepro.2016.04.063 · Cited by 38
Occupational noise and ischemic heart disease: A systematic review
Dzhambov, A. M., & Dimitrova, D. (2016) · Noise and Health · doi:10.4103/1463-1741.189241 · Cited by 38
“The agreement on these studies was assessed using Krippendorff ’ s alpha coefficient [21] and it was high ( > 0.76) for the different selection criteria.”
InfluentialBrinkel, J., May, J., Krumkamp, R., Lamshöft, M., Kreuels, B., Owusu‐Dabo, E., Mohammed, A., Marinović, A. B., Dako‐Gyeke, P., Krämer, A., et al. (2017) · Tropical Medicine & International Health · doi:10.1111/tmi.12864 · Cited by 38
“[18] by calculating Krippendorff’s alpha for each category [19].”
BackgroundMethodTowards Sustainable Growth in the Textile Industry: A Case Study of Environmental Policy in China
Xu, C.-K., Cheng, H., & Liao, Z. (2018) · Polish Journal of Environmental Studies · doi:10.15244/pjoes/79720 · Cited by 38
“Content analysis is defined as the systematic interpretation of textual, visual, or audible matter such as newspaper editorials, television news, advertisements, public speeches, and other verbal or nonverbal units of analysis [31-32].”
BackgroundMedia violence exposure and aggression in adolescents: A risk and resilience perspective
Khurana, A., Bleakley, A., Ellithorpe, M. E., Hennessy, M. H., Jamieson, P. E., & Weitz, I. (2018) · Aggressive Behavior · doi:10.1002/ab.21798 · Cited by 38
“Aftermultiple training sessions, the coders achieved inter-coder reliability for identifying the presence of each behavior as calculated by Krippendorff's alpha (Hayes & Krippendorff, 2007) using a separately validated test sample of 59 segments.”
MethodDisturbed time experience during and after psychosis
Vogel, D. H. V., Beeker, T., Haidl, T. K., Kupke, C., Heinze, M., & Vogeley, K. T. (2019) · Schizophrenia Research Cognition · doi:10.1016/j.scog.2019.100136 · Cited by 38
Blömke, L., Volkmann, N., & Kemper, N. (2019) · Meat Science · doi:10.1016/j.meatsci.2019.107934 · Cited by 38
Macaulay, P. J. R., Boulton, M. J., Betts, L. R., Boulton, L., Camerone, E. M., Down, J., Hughes, J., Kirkbride, C., & Kirkham, R. (2019) · Journal of Children and Media · doi:10.1080/17482798.2019.1697716 · Cited by 43
Using the Flipped Classroom to Enhance Adult EFL Learners' Speaking Skills
Yeşilçınar, S. (2019) · PASAA · doi:10.58837/chula.pasaa.58.1.8 · Cited by 38
Smith, D. W., & Smothers, S. M. (2012) · Journal of Visual Impairment & Blindness · doi:10.1177/0145482x1210600905 · Cited by 39
“All analyses were conducted using the statistical formula features of Microsoft Excel, SPSS 18.0, and a macro to compute Krippendorff’s alpha (Hayes & Krippendorff, 2007). f Visual Impairment & Blindness, September 2012 549”
MethodInfluentialJebril, N., de Vreese, C. H., van Dalen, A., & Albæk, E. (2013) · Scandinavian Political Studies · doi:10.1111/1467-9477.12003 · Cited by 38
“Krippendorff’s alpha’s scores were also calculated using Hayes and Krippendorff (2007) and showed the same results.”
Agnew, Ç., & Flin, R. H. (2013) · International Journal of Nursing Studies · doi:10.1016/j.ijnurstu.2013.10.001 · Cited by 38
Fraustino, J. D., & Connolly-Ahern, C. (2015) · Journal of Public Relations Research · doi:10.1080/1062726x.2015.1098543 · Cited by 38
“Reliability was calculated using Krippendorff’s alpha, which provides a more conservative estimate than percent agreement and allows for assessing variables at all levels of measurement and with missing values (Hayes & Krippendorff, 2007; Krippendorff, 2004).”
MethodLookingbill, V., Mohammadi, E., & Cai, Y. (2023) · JAMA Network Open · doi:10.1001/jamanetworkopen.2023.8897 · Cited by 37
Zhang, X., Tao, W.-T., & Kim, S. (2014) · International Journal of Strategic Communication · doi:10.1080/1553118x.2014.886251 · Cited by 34
“Krippendorff’s alpha, which is appropriate for measuring different levels of variables (Hayes & Krippendorff, 2007), was used in the calculation.”
MethodIntroducing a brief measure of cultural and religious identification in American Jewish identity.
Friedlander, M. L., Friedman, M. L., Miller, M. J., Ellis, M. V., Friedlander, L. K., & Mikhaylov, V. G. (2010) · Journal of Counseling Psychology · doi:10.1037/a0019577 · Cited by 37
“Within the expert panel, Krippendorff’s alpha (kalpha) test of agreement for categorical data with multiple judges (Hayes & Krippendorff, 2007) was .43.”
MethodCombining concepts and language models for information access
Meij, E. (2011) · ACM SIGIR Forum · doi:10.1145/1988852.1988873 · Cited by 35
Spentzas, T., Henricksen, J. W., Patters, A. B., & Chaum, E. (2010) · Pediatric Critical Care Medicine · doi:10.1097/pcc.0b013e3181ce755c · Cited by 39
Overexplicit Referent Tracking in L2 English: Strategy, Avoidance, or Myth?
Ryan, J. (2015) · Language Learning · doi:10.1111/lang.12139 · Cited by 37
German Exceptionalism? An Empirical Analysis of 20 Years of Law and Order Legislation
Wenzelburger, G., & Staff, H. (2016) · Politics & Policy · doi:10.1111/polp.12151 · Cited by 37
Cheng, Y. (2016) · Asian Journal of Communication · doi:10.1080/01292986.2015.1130159 · Cited by 37
“A total number of 916 assertions were identified by three independent coders and the reliability between these well-trained coders was 0.87 (Hayes & Krippendorff, 2007).”
BackgroundMethodInfluentialLee, T. H., & Riffe, D. (2017) · Business & Society · doi:10.1177/0007650317696314 · Cited by 37
“Krippendorff’s α was used for the intercoder reliability check (Hayes & Krippendorff, 2007).”
MethodPeterman, K., Daugherty, J. L., Custer, R. L., & Ross, J. (2017) · International Journal of Science Education · doi:10.1080/09500693.2017.1359431 · Cited by 37
“K alpha is ideal for this study because it can be used with teams of more than two researches and for codes that are dichotomous (Hayes & Krippendorff, 2007).”
MethodHelp welcome or not: Understanding consumer shopping motivation in curated fashion retailing
Sebald, A. K., & Jacob, F. (2017) · Journal of Retailing and Consumer Services · doi:10.1016/j.jretconser.2017.10.008 · Cited by 37
Çakar, K. (2018) · International Journal of Tourism Cities · doi:10.1108/ijtc-03-2017-0018 · Cited by 37
(Pre)occupations: A data-driven model of jobs and its consequences for categorization and evaluation
Imhoff, R., Koch, A. S., & Flade, F. (2018) · Journal of Experimental Social Psychology · doi:10.1016/j.jesp.2018.04.001 · Cited by 38
RAMAS: Russian Multimodal Corpus of Dyadic Interaction for Affective Computing
Perepelkina, O. S., Kazimirova, E., & Konstantinova, M. (2018) · Lecture notes in computer science · doi:10.1007/978-3-319-99579-3_52 · Cited by 37
A Look at Culture and Stigma of Suicide: Textual Analysis of Community Theatre Performances
Keller, S., McNeill, V., Honea, J. C., & Miller, L. P. (2019) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph16030352 · Cited by 37
“it is possible to construe defensive-avoidance, a fear control response that occurs when an individual feels unable to control a risk [65], as both a result of self-stigma and a barrier to help-seeking.”
BackgroundMethodReal-Time Mutual Gaze Perception Enhances Collaborative Learning and Collaboration Quality
Schneider, B., & Pea, R. D. (2016) · Educational media and technology yearbook · doi:10.1007/978-3-319-45001-8_7 · Cited by 41
Reliability analysis of fish traits reveals discrepancies among databases
Cano‐Barbacil, C., Radinger, J., & García‐Berthou, E. (2020) · Freshwater Biology · doi:10.1111/fwb.13469 · Cited by 39
“…incomplete or missing data (Krippendorff, 2011; Zapf, Castell, Morawietz, & Karch, 2016); and, in contrast to many other reliability measures, it can be computed for any type of variable (nominal, ordinal, interval, and ratio) and thus enables one to compare them (Hayes & Krippendorff, 2007).”
BackgroundMethodInfluentialAssociations of 3-year-olds’ Block-building Complexity with Later Spatial and Mathematical Skills
Bower, C. A., Odean, R., Verdine, B. N., Medford, J. R., Marzouk, M. A., Golinkoff, R. M., & Hirsh‐Pasek, K. (2020) · Journal of Cognition and Development · doi:10.1080/15248372.2020.1741363 · Cited by 39
Fready, S., Vel, P., & Nyadzayo, M. W. (2022) · Journal of Business and Industrial Marketing · doi:10.1108/jbim-01-2021-0074 · Cited by 37
Wehkamp, J., Aquino, A. R. D., Fuss, S., & Reed, E. (2015) · Forest Policy and Economics · doi:10.1016/j.forpol.2015.05.005 · Cited by 38
Framing Serbia: the effects of news framing on public support for EU enlargement
Lecheler, S. K., & de Vreese, C. H. (2010) · European Political Science Review · doi:10.1017/s1755773909990233 · Cited by 37
Physical Activity Benefits Creativity: Squeezing a Ball for Enhancing Creativity
Kim, J. (2015) · Creativity Research Journal · doi:10.1080/10400419.2015.1087258 · Cited by 41
“Krippendorff’s as (Hayes & Krippendorff, 2007) for fluency, flexibility, and originality were 1.00, .95, and .87, respectively.”
InfluentialReddy, C. D., & Hamann, R. (2016) · Business & Society · doi:10.1177/0007650316629127 · Cited by 39
“We calculated the Krippendorff alpha (Hayes & Krippendorff, 2007) to measure the degree of agreement between these coding sets, with alpha scores between .67 and .80 representing sufficient reliability (Krippendorff, 1980).”
MethodMayweg-Paus, E., & Jucks, R. (2017) · Journal of Language and Social Psychology · doi:10.1177/0261927x17716102 · Cited by 36
“Agreement between raters on where information came from was good with Krippendorff’s α = .84 for information from the discussion and Krippendorff’s α = .79 for information not directly from the discussion (Hayes & Krippendorff, 2007; Krippendorff, 2004).”
MethodMade to Feel Like Less of a Woman: The Experience of Stigma for Mothers Who Do Not Breastfeed
Bresnahan, M., Zhuang, J., Goldbort, J. G., Bogdan‐Lovis, E., Park, S.-Y., & Hitt, R. (2019) · Breastfeeding Medicine · doi:10.1089/bfm.2019.0171 · Cited by 36
Carey, R. L., Polanco, C., & Blackman, H. W. (2021) · Journal of Research on Adolescence · doi:10.1111/jora.12706 · Cited by 42
“K alpha coefficients range from 0 to 1, with 1 indicating perfect agreement among raters (Hayes & Krippendorff, 2007).”
InfluentialSports Betting Advertising: A Systematic Review of Content Analysis Studies
Killick, E. A., & Griffiths, M. D. (2022) · International Journal of Mental Health and Addiction · doi:10.1007/s11469-022-00775-4 · Cited by 36
“Krippendorrf’s alpha (Kalpha) is the most appropriate reliability coefficient typically suggested as a standard reliability measure for content analysis (Hayes & Krippendorff, 2007; Riffe et al., 2005).”
MethodThe burden of stress in head and neck cancer
Devins, G. M., Payne, A. Y. M., Lebel, S., Mah, K., Lee, R. N. F., Irish, J. C., Wong, J. C., & Rodin, G. (2012) · Psycho-Oncology · doi:10.1002/pon.3050 · Cited by 36
“Given its binary response format, responses are not amenable to traditional factor analysis to establish an underlying measurement structure [49].”
BackgroundMethodPiotrowski, J. T. (2014) · Media Psychology · doi:10.1080/15213269.2014.932288 · Cited by 35
“Krippendorff’s alpha (Hayes & Krippendorff, 2007) indicated acceptable intercoder reliability (˛ D .88).”
MethodEducating women about normal female genital appearance variation
Sharp, G., & Tiggemann, M. (2015) · Body Image · doi:10.1016/j.bodyim.2015.11.006 · Cited by 34
Re-distribution and public governance – the politics of higher education in Western Europe
Jungblut, J. (2016) · European Politics and Society · doi:10.1080/23745118.2016.1140395 · Cited by 37
“Krippendorff’s alpha was calculated at .7981 (Hayes & Krippendorff, 2007).”
MethodSexualization in U.S. Latina and White Girls’ Preferred Children’s Television Programs
McDade‐Montez, E., Wallander, J. L., & Cameron, L. D. (2016) · Sex Roles · doi:10.1007/s11199-016-0692-0 · Cited by 36
“The Krippendorff’s alpha, which is particularly suited to content analysis data (Hayes and Krippendorff 2007), was .”
MethodInter-rater reliability of the Shoulder Symptom Modification Procedure in people with shoulder pain
Lewis, J. S., McCreesh, K. M., Barratt, E., Hegedus, E. J., & Sim, J. (2016) · BMJ Open Sport & Exercise Medicine · doi:10.1136/bmjsem-2016-000181 · Cited by 37
“Statistical analysis The inter-rater reliability of clinicians’ assessment of the response to SSMP procedures was calculated by analysing the responses provided by the clinicians (no change, worse, partial improvement, complete improvement) using Krippendorff’s α.(46) This statistic is a reliability coefficient suitab…”
MethodSadeh, F., & Kacker, M. (2017) · Small Business Economics · doi:10.1007/s11187-017-9892-4 · Cited by 38
“We calculated inter-rater reliability of the results as Krippendorff’s alpha of 0.83, based on Hayes and Krippendorff (2007).”
MethodInfluentialBRAUER, M. F., & Zimmermann, M. (2017) · Journal of Management · doi:10.1177/0149206317734901 · Cited by 36
Hecht, M. L., Shin, Y., Pettigrew, J., Miller-Day, M. A., & Krieger, J. L. R. (2018) · Prevention Science · doi:10.1007/s11121-018-0937-y · Cited by 36
MRI LI-RADS Version 2018: Impact of and Reduction in Ancillary Features
van der Pol, C. B., Dhindsa, K., Shergill, R., Zha, N., Ferri, M., Kagoma, Y. K., Lee, S., Satkunasingham, J., Wat, J., & Tsai, S. (2021) · American Journal of Roentgenology · doi:10.2214/ajr.20.23031 · Cited by 36
Lee, Y., Huang, K., Blom, R., Schriner, R., & Ciccarelli, C. (2021) · Cyberpsychology Behavior and Social Networking · doi:10.1089/cyber.2020.0176 · Cited by 36
Sunde, N., & Dror, I. E. (2021) · Forensic Science International Digital Investigation · doi:10.1016/j.fsidi.2021.301175 · Cited by 36
Peer-to-peer accommodation platform affordance: Scale development and empirical investigation
Yan, R., & Gong, X. (2022) · Journal of Business Research · doi:10.1016/j.jbusres.2022.02.032 · Cited by 36
The taste & affect music database: Subjective rating norms for a new set of musical stimuli
Guedes, D., Prada, M., Garrido, M. V., & Lamy, E. (2022) · Behavior Research Methods · doi:10.3758/s13428-022-01862-z · Cited by 43
“The agreement rate was estimated with Krippendorff's alpha test (Hayes & Krippendorff, 2007).”
MethodSemantic-Rich Facial Emotional Expression Recognition
Chen, K., Yang, X., Fan, C., Zhang, W., & Ding, Y. (2022) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2022.3201290 · Cited by 37
“Furthermore, to demonstrate the test validity of the rating experiment, we also evaluate the inter-rater reliability by Krippendorff’s alpha test [57].”
MethodData governance and green technological innovation performance: A curvilinear relationship
Yin, J., & Li, C. (2022) · Journal of Cleaner Production · doi:10.1016/j.jclepro.2022.134441 · Cited by 43
LaMarre, H. L., & Walther, W. (2013) · International Journal of Public Opinion Research · doi:10.1093/ijpor/edt008 · Cited by 36
Correction of Overstatement and Omission in Direct-to-Consumer Prescription Drug Advertising
Aikin, K. J., Betts, K. R., O’Donoghue, A. C., Rupert, D. J., Lee, P. K., Amoozegar, J. B., & Southwell, B. G. (2015) · Journal of Communication · doi:10.1111/jcom.12167 · Cited by 36
“Using Krippendorff ’s alpha as our intercoder reliability measure (Hayes & Krippendorff, 2007), we achieved an alpha of at least .70 for all coded variables presented.”
MethodOpen Video Game Development and Participatory Design
Thominet, L. (2020) · Technical Communication Quarterly · doi:10.1080/10572252.2020.1866679 · Cited by 35
“Interrater reliability was calculated using Krippendorf’s alpha (α) because it is a widely accepted method for calculating reliability in content analyses (Hayes & Krippendorff, 2007).”
MethodTopooco, N., Fowler, L. A., Fitzsimmons‐Craft, E. E., DePietro, B., Vázquez, M. M., Firebaugh, M., Ceglarek, P. J. D., Monterubio, G. E., Newman, M. G., Eisenberg, D., et al. (2022) · Internet Interventions · doi:10.1016/j.invent.2022.100528 · Cited by 37
“Krippendorff's alpha coefficient (Hayes and Krippendorff, 2007) was used to test inter-rater reliability.”
MethodTowards a conceptual systematic review: proposing a methodological framework
Schreiber, F., & Cramer, C. (2022) · Educational Review · doi:10.1080/00131911.2022.2116561 · Cited by 77
“Krippendor ff ’ s alpha (Hayes & Krippendor ff , 2007) should be reported as the coe ffi cient of intercoder agreement, as it can deal with a wide range of given scenarios – such as more than two coders, di ff erent interval scales, or missing data – and it provides a scaled value between 0 (complete…”
Cheng, C., Ying, W., Ebrahimi, O. V., & Wong, K. F. E. (2023) · Health Psychology Review · doi:10.1080/17437199.2023.2175015 · Cited by 44
“Inter-reviewer reliabilities were assessed by Krippendor ff ’ s alpha (Krippendor ff , 2018), which was computed using the KALPHA macro for SPSS (Hayes & Krippendor ff , 2007).”
MethodHasan, F., Foster, M. M., & Cho, H. (2023) · Journal of Health Communication · doi:10.1080/10810730.2023.2235563 · Cited by 52
Koporčić, N., Sjödin, D., Kohtamäki, M., & Parida, V. (2024) · Small Business Economics · doi:10.1007/s11187-024-00897-0 · Cited by 43
Hamad, S., Lai, F., Shad, M. K., Shah, S. Q. A., Jan, A. A., & Ali, S. E. A. (2024) · Business Strategy & Development · doi:10.1002/bsd2.398 · Cited by 41
Cardoni, A., & Киселева, Е. (2024) · Business Strategy and the Environment · doi:10.1002/bse.4038 · Cited by 35
Issue ownership as a determinant of negative campaigning
Elmelund‐Præstekær, C. (2011) · International Political Science Review · doi:10.1177/0192512110382028 · Cited by 35
“Using Krippendorff’s alpha algorithm (Hayes and Krippendorff, 2007), the coders’ RTI values for each letter were compared, yielding an alpha value of .93.”
MethodHildebrand, M., & Noteborn, M. G. C. (2015) · Substance Use & Misuse · doi:10.3109/10826084.2014.998238 · Cited by 35
“Kα is a reliability coefficient developed to estimate reliability regardless of the number of observers, levels of measurement, sample sizes, and the presence or absence of missing data (Hayes & Krippendorff, 2007; Krippendorff, 2004a, b).”
BackgroundMethodYang, Q., Zhang, X., Du, X., Bielefield, A., & Liu, Y. (2016) · Applied Sciences · doi:10.3390/app6020048 · Cited by 35
“To identify competencies from job ads, Sutton (2011) utilizes a content analysis methodology proposed by Krippendorff [12].”
BackgroundMethodSaksobhavivat, N., Shanmuganathan, K., Boscak, A. R., Sliker, C. W., Stein, D. M., Bodanapally, U. K., Archer-Arroyo, K., Miller, L. A., Fleiter, T. R., Alexander, M. T., et al. (2016) · European Radiology · doi:10.1007/s00330-016-4260-3 · Cited by 35
D’Mello, S. K. (2015) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2015.2457413 · Cited by 34
“There are many possible metrics to compute OO reliability , each with different strengths and weaknesses [38].”
BackgroundKetelaars, P. (2016) · Mobilization An International Quarterly · doi:10.17813/1086-671x-21-3-341 · Cited by 35
Managing Uncertainty in Paramedics’ Decision Making
Harenčárová, H. (2016) · Journal of Cognitive Engineering and Decision Making · doi:10.1177/1555343416674814 · Cited by 44
“The Krippendorf’s α for uncertainty, strategies, and substrategies was .779, .693, and .876, respectively.”
MethodDaalmans, S., Kleemans, M., & Sadza, A. J. C. (2017) · Sex Roles · doi:10.1007/s11199-016-0727-6 · Cited by 35
“Intercoder reliabilities were calculated in SPSS using the macro by Hayes for Krippendorff’s alpha (Hayes and Krippendorff 2007).”
MethodNederhand, J., & van Meerkerk, I. (2017) · Policy & Politics · doi:10.1332/030557317x15035697297906 · Cited by 35
“Krippendorff’s alpha test (Hayes and Krippendorff 2007) resulted in an average inter-coder reliability coefficient of .83 (range .77 to .89), which indicates good inter-coder reliability.”
MethodAhmed, S., Cho, J., & Jaidka, K. (2017) · Telematics and Informatics · doi:10.1016/j.tele.2017.09.005 · Cited by 35
Sex Ed to Go: A Content Analysis of Comprehensive Sexual Education Apps
Kalke, K., Ginossar, T., Shah, S. F. A., & West, A. J. (2017) · Health Education & Behavior · doi:10.1177/1090198117749259 · Cited by 35
A critical analysis of inter-coder reliability methods in information systems research
Nili, A., Tate, M., & Barros, A. (2017) · QUT ePrints (Queensland University of Technology) · Cited by 34
Evaluating the prevalence and quality of conference codes of conduct
Foxx, A. J., Barak, R. S., Lichtenberger, T. M., Richardson, L. K., Rodgers, A. J., & Williams, E. W. (2019) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1819409116 · Cited by 36
“, complete matches between all coders) to 0 representing “the absence of reliability” (49).”
BackgroundBredt, S. D. G. T., Torres, J., Diniz, L. B. F., Praça, G. M., de Andrade, A. G. P., Morales, J. C. P., Rosso, T. L. N., & Chagas, M. H. (2020) · Biology of Sport · doi:10.5114/biolsport.2020.93038 · Cited by 45
“The Krip - pendorff alpha (a reliability coefficient for nominal data) [27] values were 0.76 (95% CI 0.73–0.79) and 0.98 (95% CI 0.98–0.99) for between-and within-observer reliability, respectively.”
BackgroundInfluentialLeon, R.-D., Rodríguez, R. R., Gómez-Gasquet, P., & Mula, J. (2020) · Information Processing & Management · doi:10.1016/j.ipm.2020.102237 · Cited by 35
Gesser‐Edelsburg, A., Walter, N., & Green, M. S. (2014) · American Journal of Infection Control · doi:10.1016/j.ajic.2014.04.012 · Cited by 36
Karsay, K., & Matthes, J. (2016) · Communication Research · doi:10.1177/0093650216661434 · Cited by 35
“The reliability was measured with Krippendorff’s alpha, which was satisfactorily, α = .89 (Hayes & Krippendorff, 2007).”
MethodLabrie, N., & Schulz, P. J. (2014) · Argumentation · doi:10.1007/s10503-014-9331-5 · Cited by 33
“8 In order to calculate Krippendorff’s alpha, use was made of a SPSS macro developed and freely distributed by Hayes and Krippendorff (2007). Quantifying Doctors’ Argumentation in General Practice 37”
MethodConstituency Diversity, District Magnitude and Voter Co-ordination
Potter, J. D. (2016) · British Journal of Political Science · doi:10.1017/s0007123415000484 · Cited by 34
Does fake news impact stock returns? Evidence from US and EU stock markets
Arcuri, M. C., Gandolfi, G., & Russo, I. (2023) · Journal of Economics and Business · doi:10.1016/j.jeconbus.2023.106130 · Cited by 47
Hsieh, J. C. (2023) · Education and Information Technologies · doi:10.1007/s10639-023-12135-3 · Cited by 34
“The inter-rater reliability was assessed using Krippendorff’s alpha, yielding a good reliability coefficient of 0.83 (Hayes & Krippendorff, 2007).”
InfluentialExposure to Smoking in Movies and Smoking Initiation Among Black Youth
Cin, S. D., Stoolmiller, M., & Sargent, J. D. (2013) · American Journal of Preventive Medicine · doi:10.1016/j.amepre.2012.12.008 · Cited by 34
Metadata
- Authors
- Hayes, A. F., & Krippendorff, K.
- Affiliations
- The Ohio State University (US)
- University of Pennsylvania (US)
- Venue
- Communication Methods and Measures
- Publisher
- Informa UK Limited
- Published
- 2007-04
- Volume
- 1
- Issue
- 1
- Pages
- 77-89
- ISSN
- 1931-2458 (ISSN-L)
- 1931-2466
- OpenAlex
- W2061504941
- Cited by
- 4,225
- References
- 30
- Citations by year
- 2026: 284 · 2025: 278 · 2024: 280 · 2023: 289 · 2022: 331 · 2021: 379 · 2020: 369 · 2019: 296 · 2018: 260 · 2017: 300 · 2016: 248 · 2015: 205 · 2014: 206 · 2013: 150 · 2012: 107
- FWCI
- 50.07
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Communication Methods and Measures (publishedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Reliability and Agreement in Measurement
- Social Sciences › General Social Sciences › Computational and Text Analysis Methods
- Social Sciences › Decision Sciences › Management Science and Operations Research › Psychometric Methodologies and Testing
- Keywords
- Krippendorff's alpha, inter-rater reliability, content analysis, coding reliability, reliability coefficients, observer agreement, SAS macro, missing data
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
