Cited by 4,225Reliability and Agreement in MeasurementComputational and Text Analysis MethodsPsychometric Methodologies and Testing

The DOI matches its Crossref record; 30 of the 30 references it cites were found in a registry or an index.

Answering the Call for a Standard Reliability Measure for Coding Data

Hayes, Andrew F., Krippendorff, Klaus·2007·Communication Methods and Measures

DOI 10.1080/19312450709336664

Open this paper in CoChat — free account

Abstract

In content analysis and similar methods, data are typically generated by trained human observers who record or transcribe textual, pictorial, or audible matter in terms suitable for analysis. Conclusions from such data can be trusted only after demonstrating their reliability. Unfortunately, the content analysis literature is full of proposals for so-called reliability coefficients, leaving investigators easily confused, not knowing which to choose. After describing the criteria for a good measure of reliability, we propose Krippendorff's alpha as the standard reliability measure. It is general in that it can be used regardless of the number of observers, levels of measurement, sample sizes, and presence or absence of missing data. To facilitate the adoption of this recommendation, we describe a freely available macro written for SPSS and SAS to calculate Krippendorff's alpha and illustrate its use with a simple example.

Integrity signals

Registry

Crossref · 10.1080/19312450709336664

Cites

30 of 30 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Claims the paper makes

This paper’s full text isn’t openly available, so its claims and self-check weren’t run. The registry and citation checks still apply.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Coefficient Alpha and the Internal Structure of Tests
Cronbach, L. J. (1951) · Psychometrika · doi:10.1007/bf02310555 · Cited by 43,931
A Coefficient of Agreement for Nominal Scales
Cohen, J. (1960) · Educational and Psychological Measurement · doi:10.1177/001316446002000104 · Cited by 42,398
An Introduction to the Bootstrap
Efron, B., & Tibshirani, R. J. (1994) · doi:10.1201/9780429246593 · Cited by 40,500
Nonparametric Statistics for the Behavioral Sciences
Guitton, H., & Siegel, S. (1958) · Revue économique · doi:10.2307/3498751 · Cited by 35,279
Content Analysis: An Introduction to its Methodology.
Shelley, M., & Krippendorff, K. (1984) · Journal of the American Statistical Association · doi:10.2307/2288384 · Cited by 24,351
Self-citation
Statistical Methods for Rates and Proportions
Fleiss, J. L., Levin, B., & Paik, M. C. (2003) · Wiley Series in Probability and Statistics · doi:10.1002/0471445428 · Cited by 18,646
Measuring nominal scale agreement among many raters.
Fleiss, J. L. (1971) · Psychological Bulletin · doi:10.1037/h0031619 · Cited by 8,733
Content Analysis for the Social Sciences and Humanities.
Parker, E. B., & Holsti, O. R. (1970) · American Sociological Review · doi:10.2307/2093233 · Cited by 4,553
Content Analysis in Mass Communication: Assessment and Reporting of Intercoder Reliability
Lombard, M., Snyder-Duch, J., & Bracken, C. C. (2002) · Human Communication Research · doi:10.1111/j.1468-2958.2002.tb00826.x · Cited by 3,068
An Introduction to the Bootstrap
Grimshaw, S. D., Efron, B., & Tibshirani, R. J. (1995) · Technometrics · doi:10.2307/1269918 · Cited by 2,281

Show all 30 references

Research works printed but not linked: 1

10.1111/j.1468-2958.2004.tb00738.x · DOI · Paper page

Sorted by citations

Items 901–1,000 of 1,013

Understand users’ comprehension and preferences for composing information visualizations
Yang, H., Li, Y., & Zhou, M. X. (2014) · ACM Transactions on Computer-Human Interaction · doi:10.1145/2541288 · Cited by 34
“The Krippendorff s a ranged from 0.25 to 1 [Hayes and Krippendorff 2007] for different codes, where 0 indicates the absence of reliability, 1 indicates the perfect reliability, and a score higher than 0.7 is considered achieving acceptable reliability in practice.9 By the testing score, the…”
BackgroundInfluential
News Reporting of Opinion Polls: Journalism and Statistical Noise
Bhatti, Y., & Pedersen, R. T. (2015) · International Journal of Public Opinion Research · doi:10.1093/ijpor/edv008 · Cited by 34
Group interactions and time: Using sequential analysis to study group dynamics in project meetings.
Klonek, F. E., Quera, V., Burba, M., & Kauffeld, S. (2016) · Group Dynamics Theory Research and Practice · doi:10.1037/gdn0000052 · Cited by 34
The Reliability of Multi-Valued Coding of Data
Krippendorff, K., & Craggs, R. (2016) · Communication Methods and Measures · doi:10.1080/19312458.2016.1228863 · Cited by 34
“These virtues have been discussed (e.g., Hayes & Krippendorff, 2007; Krippendorff, 2011, 2013) and do not need to be reiterated here.”
BackgroundInfluentialSelf-citation
Breast Cancer Survivors’ Contribution to Psychosocial Adjustment of Newly Diagnosed Breast Cancer Patients in a Computer-Mediated Social Support Group
Moon, T., Chih, M., Shah, D. V., Yoo, W., & Gustafson, D. H. (2017) · Journalism & Mass Communication Quarterly · doi:10.1177/1077699016687724 · Cited by 34
“The results indicated that the coding schemes for emotional support (Krippendorff’s α = .80) and informational support (Krippendorff’s α = .70) were reliable (Hayes & Krippendorff, 2007).”
Method
An inverted personalization effect when learning with multimedia: The case of aversive content
Kühl, T., & Zander, S. (2017) · Computers & Education · doi:10.1016/j.compedu.2017.01.013 · Cited by 34
Local Television News Coverage of the Affordable Care Act: Emphasizing Politics Over Consumer Information
Gollust, S. E., Baum, L. M., Niederdeppe, J., Barry, C. L., & Fowler, E. F. (2017) · American Journal of Public Health · doi:10.2105/ajph.2017.303659 · Cited by 34
“For 1 ordinal variable (tone of coverage), we estimated Krippendorff’s alpha.(15) For some relatively lowfrequency items, IRR statistics fell below 0.”
Method
Clinician Perceptions Related to the Use of the CBT-I Coach Mobile App
Miller, K. E., Kuhn, E. R., Owen, J. E., Taylor, K. L., Yu, J., Weiss, B. J., Crowley, J. J., & Trockel, M. (2017) · Behavioral Sleep Medicine · doi:10.1080/15402002.2017.1403326 · Cited by 33
“Interrater agreement was calculated using Krippendorf’s alpha, with alphas of .80 being considered satisfactory (Hayes & Krippendorff, 2007).”
Method
The role of gesture delay in coda /r/ weakening: An articulatory, auditory and acoustic study
Lawson, E., Stuart‐Smith, J., & Scobbie, J. M. (2018) · The Journal of the Acoustical Society of America · doi:10.1121/1.5027833 · Cited by 34
Flow and structure of time experience – concept, empirical validation and implications for psychopathology
Vogel, D. H. V., Falter‐Wagner, C. M., Schoofs, T., Krämer, K., Kupke, C., & Vogeley, K. T. (2018) · Phenomenology and the Cognitive Sciences · doi:10.1007/s11097-018-9573-z · Cited by 33
Story of a Seed: educational theatre improves students’ comprehension of plant reproduction and attitudes to plants in primary science education
Stagg, B. C., & Verde, M. F. (2018) · Research in Science & Technological Education · doi:10.1080/02635143.2018.1455655 · Cited by 34
“Krippendorff’s ordinal alpha is 0.991 (n = 21), a substantial degree of reliability (Hayes and Krippendorff 2007).”
BackgroundMethodInfluential
An account of the textile waste policy in China (1991–2017)
Xu, C.-K., Cheng, H., Liao, Z., & Hu, H. (2019) · Journal of Cleaner Production · doi:10.1016/j.jclepro.2019.06.283 · Cited by 34
Mixing business with politics: Does corporate social responsibility end where lobbying transparency begins?
Favotto, A., & Kollman, K. L. (2019) · Regulation & Governance · doi:10.1111/rego.12275 · Cited by 34
“Drawing on Hayes and Krippendorff (2007), we have estimated intercoder reliability using Krippendorff α for our overarching disclosure quality index and for each issue area.”
Method
Computer-aided diagnostic system for thyroid nodule sonographic evaluation outperforms the specificity of less experienced examiners
Fresilli, D., Grani, G., Pascali, M. L. D., Alagna, G., Tassone, E., Ramundo, V., Ascoli, V., Bosco, D., Biffoni, M., Bononi, M., et al. (2020) · Journal of Ultrasound · doi:10.1007/s40477-020-00453-y · Cited by 34
“Interobserver agreement was assessed with Krippendorff’s alpha [30] for analyses involving ordinal data and more than two observers and with Cohen’s kappa for analyses involving dichotomous variables and two observers.”
Method
Each post matters: a content analysis of #mentalhealth images on Instagram
Lee, N., Buchanan, K., & Yu, M. (2020) · Journal of Visual Communication in Medicine · doi:10.1080/17453054.2020.1781535 · Cited by 40
Meta-analysis of controlled trials testing horticultural therapy for the improvement of cognitive function
Tu, H., & Chiu, P.-Y. (2020) · Scientific Reports · doi:10.1038/s41598-020-71621-7 · Cited by 34
The economic worth of loyalty programs: An event study analysis
Faramarzi, A., & Bhattacharya, A. (2020) · Journal of Business Research · doi:10.1016/j.jbusres.2020.09.044 · Cited by 34
“The inter-coder reliability in terms of Krippendorff ’ s alpha (Hayes & Krippendorff, 2007) was 90%, which shows high reliability.”
Method
The Influence of Online Health Information Seeking Before a Consultation on Anxiety, Satisfaction, and Information Recall, Mediated by Patient Participation: Field Study
de Looper, M., van Weert, J. C. M., Schouten, B. C., Bolle, S., Belgers, E. H. J., Eddes, E. H., & Smets, E. M. A. (2021) · Journal of Medical Internet Research · doi:10.2196/23670 · Cited by 39
“A total of 10% of the cases (7/63) were double-coded by 2 independent coders to check intercoder reliability (mean κ =0.71; P <.001) [89].”
Influential
Mobile News Learning — Investigating Political Knowledge Gains in a Social Media Newsfeed with Mobile Eye Tracking
Ohme, J., Masłowska, E., & Mothes, C. (2021) · Political Communication · doi:10.1080/10584609.2021.2000082 · Cited by 34
“The intercoder reliability test (Hayes & Krippendorff, 2007) revealed strong agreement between coders (MKalpha = .85, Rangekalpha = .80– 1.00).”
MethodInfluential
The Role of Sustainability Reporting in the Agri-Food Supply Chain
Becker, J. T., & Ellis, J. D. (2017) · Journal of Agriculture and Environmental Sciences · doi:10.15640/jaes.v6n1a2 · Cited by 34
“An intercoder reliability score of 0.913 was obtained; absolute agreement for Krippendorff’s alpha is 1.0, so the score of 0.913 is considered to be a strong level of agreement (Hayes & Krippendorff, 2007).”
Influential
Alcohol and Other Drug Resistance Strategies Employed by Rural Adolescents
Pettigrew, J., Miller-Day, M. A., Krieger, J. L. R., & Hecht, M. L. (2011) · Journal of Applied Communication Research · doi:10.1080/00909882.2011.556139 · Cited by 34
“Next, an additional 10% of total units were analyzed using the agreed upon coding categories and employing Krippendorff’s alpha (Hayes & Krippendorff, 2007) as the index of reliability between two independent observers.”
Method
Enhancing Practice and Achievement in Introductory Programming With a Robot Olympics
Scott, M., Counsell, S., Lauria, S., Swift, S., Tucker, A., Shepperd, M., & Ghinea, G. (2015) · IEEE Transactions on Education · doi:10.1109/te.2014.2382567 · Cited by 35
“Moderation of 12 truncated submissions showed that self-consistency ( ) and faculty agreement with marks ( ) were adequate [38].”
Background
Arousing News Characteristics in Dutch Television News 1990–2004: An Exploration of Competitive Strategies
Vettehen, P. H., Beentjes, J. W. J., Nuijten, C., & Peeters, A. (2010) · Mass Communication & Society · doi:10.1080/15205431003615893 · Cited by 34
“They were computed using the SPSS macro described by Hayes and Krippendorff ( 2007 ).”
Method
The influence of pro-environmental managers’ personal values on environmental disclosure
Luque‐Vílchez, M., Mesa‐Pérez, E., Husillos, J., & Larrínaga, C. (2019) · Sustainability Accounting Management and Policy Journal · doi:10.1108/sampj-01-2018-0016 · Cited by 35
Sculptors, Architects, and Painters Conceive of Depicted Spaces Differently
Cialone, C., Tenbrink, T., & Spiers, H. J. (2017) · Cognitive Science · doi:10.1111/cogs.12510 · Cited by 33
“Krippendorff’s alpha (Krippendorff, 2004) was computed separately for each category (see Appendix S3: 1 and 2 for details), using the SPSS macro described in Hayes and Krippendorff (2007).”
Method
“Motivated To Teach, but Stressed Out by Teacher Education”: A Content Analysis of Self-Reported Sources of Stress and Motivation Among Preservice Teachers
Núñez‐Regueiro, F., Boulley, G. E., Azouaghe, S., Leroy, N., & Núñez-Regueiro, S. (2023) · Journal of Teacher Education · doi:10.1177/00224871231181374 · Cited by 33
“…Krippendorff’s alpha indicator and its bootstrap sampling distribution were also computed, thus identifying categories for which the encoders’ agreement was significantly higher than what could be expected by chance, with a 95% confidence interval (Hayes & Krippendorff, 2007; Krippendorff, 2004).”
Method
The central role of symptom severity and associated characteristics for functional impairment in misophonia
Möllmann, A., Heinrichs, N., Illies, L., Potthast, N., & Kley, H. (2023) · Frontiers in Psychiatry · doi:10.3389/fpsyt.2023.1112472 · Cited by 35
“Analyses were conducted with SPSS Version 26 and the KALPHA macro (43).”
Method
Not in My Face: Challenges and Ethical Considerations in Automatic Face Emotion Recognition Technology
Mattioli, M., & Cabitza, F. (2024) · Machine Learning and Knowledge Extraction · doi:10.3390/make6040109 · Cited by 47
“In contrast, reliability refers to consistency across different times, places, subjects, and experimental conditions [135,136].”
Does Movie or Video Game Violence Predict Societal Violence? It Depends on What You Look at and When
Ferguson, C. J. (2015) · Journal of Communication · doi:10.1111/jcom.12142 · Cited by 33
Too good to care: The effect of skill on hostility and aggression following violent video game play
Matthews, N. L. (2015) · Computers in Human Behavior · doi:10.1016/j.chb.2015.01.059 · Cited by 33
Psychiatric disorders and aggression in the printed media: is there a link? a central European perspective
Nawka, A., Rukavina, T. V., Nawková, L., Jovanović, N., Brborović, O., & Raboch, J. (2012) · BMC Psychiatry · doi:10.1186/1471-244x-12-19 · Cited by 33
“75 indicated high reliability [34-36].”
State Implementation of Language Acquisition Policies and Reading Achievement Among Hispanic Students
López, F., & McEneaney, E. H. (2012) · Educational Policy · doi:10.1177/0895904811417581 · Cited by 35
“A measure of reliability recommended for content analysis and adjustable for ratings at different levels of measurement is Krippendorf’s alpha (Grayson & Rust, 2001; Hayes & Krippendorf, 2007).”
Method
Giving Boys a Shot: The HPV Vaccine’s Portrayal in Canadian Newspapers
Perez, S., Fedoruk, C., Shapiro, G. K., & Rosberger, Z. (2016) · Health Communication · doi:10.1080/10410236.2015.1089466 · Cited by 33
“As in other media analyses (Casciotti et al., 2014; Forster et al., 2010; King et al., 2014), Krippendorff’s alpha, a widely accepted reliability coefficient that assesses interrater agreement, was calculated (Hayes & Krippendorff, 2007; Krippendorff, 2004).”
Method
Measuring the morphological characteristics of thoracolumbar fascia in ultrasound images: an inter-rater reliability study
Coninck, K. D., Hambly, K., Dickinson, J. W., & Passfield, L. (2018) · BMC Musculoskeletal Disorders · doi:10.1186/s12891-018-2088-5 · Cited by 32
Standardizing an approach to the evaluation of implementation science proposals
Crable, E. L., Biancarelli, D. L., Walkey, A. J., Allen, C. G., Proctor, E. K., & Drainoni, M. (2018) · Implementation Science · doi:10.1186/s13012-018-0770-5 · Cited by 36
“70 was deemed a priori as the lowest acceptable level of agreement to establish reliability of the new scoring protocol [10, 11].”
BackgroundMethod
Facilitating the Expression of Emotions by Alleged Victims of Child Abuse During Investigative Interviews Using the Revised NICHD Protocol
Karni‐Visel, Y., Hershkowitz, I., Lamb, M. E., & Blasbalg, U. (2019) · Child Maltreatment · doi:10.1177/1077559519831382 · Cited by 32
Young adults' motivations to feed wild birds and influences on their potential participation in citizen science: An exploratory study
Martin, V. Y., & Greig, E. I. (2019) · Biological Conservation · doi:10.1016/j.biocon.2019.05.009 · Cited by 39
Trapped in a service encounter
Fließ, S., & Volkers, M. (2019) · Journal of service management · doi:10.1108/josm-01-2019-0013 · Cited by 34
Visual‐only assessments of skin lesions on free‐ranging common bottlenose dolphins ( Tursiops truncatus ): Reliability and utility of quantitative tools
Toms, C. N., Stone, T., & Och‐Adams, T. (2020) · Marine Mammal Science · doi:10.1111/mms.12670 · Cited by 32
How Do Populists Visually Represent ‘The People’? A Systematic Comparative Visual Content Analysis of Donald Trump and Bernie Sanders’ Instagram Accounts
Moffitt, B. (2022) · The International Journal of Press/Politics · doi:10.1177/19401612221100418 · Cited by 46
“Strong reliability (>0.8) was achieved with race and gender variables, and suf ﬁ cient reliability for the age variable (>0.67) (Hayes and Krippendorff 2007) (see Supplementary Information ﬁ le).”
BackgroundInfluential
Toward Enhancing 12-Step Facilitation among Young People: A Systematic Qualitative Investigation of Young Adults’ 12-Step Experiences
Labbe, A. K., Slaymaker, V. J., & Kelly, J. F. (2014) · Substance Abuse · doi:10.1080/08897077.2014.950001 · Cited by 33
“Interrater agreement among the raters was calculated utilizing Krippendorff’s alpha, which accommodates multiple raters.(34)”
Method
Participatory design and validation of mobility enhancement robotic wheelchair
Daveler, B. J., Salatin, B., Grindle, G. G., Candiotti, J. L., Wang, H., & Cooper, R. A. (2015) · The Journal of Rehabilitation Research and Development · doi:10.1682/jrrd.2014.11.0278 · Cited by 33
“Second, the CT used for the targeted coding of the focus group transcripts included a statistical test for calculating Krippendorff alpha to determine the interrater reliability [19].”
Method
Correction of misleading information in prescription drug television advertising: The roles of advertisement similarity and time delay
Aikin, K. J., Southwell, B. G., Paquin, R. S., Rupert, D. J., O’Donoghue, A. C., Betts, K. R., & Lee, P. K. (2016) · Research in Social and Administrative Pharmacy · doi:10.1016/j.sapharm.2016.04.004 · Cited by 35
Missing the mark for patient engagement: mHealth literacy strategies and behavior change processes in smoking cessation apps
Paige, S. R., Alber, J. M., Stellefson, M. L., & Krieger, J. L. (2018) · Patient Education and Counseling · doi:10.1016/j.pec.2017.11.006 · Cited by 25
“Krippendorff’s alpha [16] revealed adequate inter-coder reliability for the processes of change ( α = .86) and mHealth Literacy strategies ( α = .87).”
Method
Testing the Ability and Limitations of ChatGPT to Generate Differential Diagnoses from Transcribed Radiologic Findings
Sun, S. H., Huynh, K. N., Cortes, G., Hill, R., Tran, J., Yeh, L., Ngo, A. L., Houshyar, R., Yaghmai, V., & Tran, M. D. (2024) · Radiology · doi:10.1148/radiol.232346 · Cited by 33
Liking Them Thin: Adolescents’ Favorite Television Characters and Body Image
Te’eni-Harari, T., & Eyal, K. (2015) · Journal of Health Communication · doi:10.1080/10810730.2015.1012241 · Cited by 31
Showing versus telling: Comparing online video from newspaper and television websites
Bock, M. A. (2015) · Journalism · doi:10.1177/1464884914568076 · Cited by 32
“Both waves of coding were accomplished by a two-person team, who worked to a standard until they achieved acceptable inter-coder reliability (above 85% measured by Krippendorff’s alpha (Hayes and Krippendorff, 2007)).”
Method
Language Instruction Educational Programs and Academic Achievement of Latino English Learners: Considerations for States with Changing Demographics
López, F., McEneaney, E. H., & Nieswandt, M. (2015) · American Journal of Education · doi:10.1086/680410 · Cited by 32
An Information Literacy Snapshot: Authentic Assessment across the Curriculum
Dance, B., Davis, E., Fagerheim, B. A., Hedrich, A. E., Lundstrom, K., Martin, P. N., & Holliday, W. (2015) · College & Research Libraries · doi:10.5860/crl.76.2.170 · Cited by 38
Relational Maintenance in the Digital Age: Implicit Rules and Multiple Modalities
Eden, J., & Veksler, A. E. (2016) · Communication Quarterly · doi:10.1080/01463373.2015.1103279 · Cited by 31
“Approximately one third of the data in each group, (35% of the total data or 1,014 of the items) were double coded so that inter-coder reliability could be calculated using an SPSS macro (Hayes & Krippendorff, 2007) which returned Krippendroff ’ s alphas for each group of data.”
Method
Inter-rater reliability of the Reaper Oral Mucosa Pressure Injury Scale (ROMPIS): A novel scale for the assessment of the severity of pressure injuries to the mouth and oral mucosa
Reaper, S., Green, C., Gupta, S., & Tiruvoipati, R. (2016) · Australian Critical Care · doi:10.1016/j.aucc.2016.06.003 · Cited by 32
Perceived access to recreational/green areas as an effect modifier of the relationship between health and neighbourhood noise/air quality: Results from the 3rd European Quality of Life Survey (EQLS, 2011–2012)
Dimitrova, D., & Dzhambov, A. M. (2017) · Urban forestry & urban greening · doi:10.1016/j.ufug.2017.02.012 · Cited by 32
Redesigning a course to help students achieve higher-order cognitive thinking skills: from goals and mechanics to student outcomes
Casagrand, J. L., & Semsar, K. (2017) · AJP Advances in Physiology Education · doi:10.1152/advan.00102.2016 · Cited by 33
We're sorry but it's not our fault: Organizational apologies in ambiguous crisis situations
Bentley, J. M., Oostman, K. R., & Shah, S. F. A. (2017) · Journal of Contingencies and Crisis Management · doi:10.1111/1468-5973.12169 · Cited by 32
Narrative Language and Reading Comprehension in Students With Mild Intellectual Disabilities
Barton-Hulsey, A., Sevcik, R. A., & Romski, M. A. (2017) · American Journal on Intellectual and Developmental Disabilities · doi:10.1352/1944-7558-122.5.392 · Cited by 32
“NSS reliability was calculated using Krippendorff’s Alpha (Hayes & Krippendorff, 2007).”
Method
Casual Sexual Scripts on the Screen: A Quantitative Content Analysis
Timmermans, E., & den Bulck, J. V. (2018) · Archives of Sexual Behavior · doi:10.1007/s10508-018-1147-1 · Cited by 32
“Intercoder reliability was computed for this subsample and was measured through Krippendorff’s alpha (kalpha) coefficient (Hayes & Krippendorff, 2007).”
Method
Eye glances towards conflict-relevant cues: the roles of anticipatory competence and driver experience
Stahl, P., Donmez, B., & Jamieson, G. A. (2019) · Accident Analysis & Prevention · doi:10.1016/j.aap.2019.07.031 · Cited by 32
A novel systematic approach for analysing exploratory design ideation
Hay, L., Duffy, A. H. B., Grealy, M., Tahsiri, M., McTeague, C. P., & Vuletic, T. (2019) · Journal of Engineering Design · doi:10.1080/09544828.2019.1662381 · Cited by 32
“Using this information, ICR was quantified using a statistical measure of agreement called Krippendorff’s alpha (Hayes and Krippendorff 2007).”
Method
The Characteristics, Activities and Goals of Environmental Organizations Engaged in Advocacy Within the Australian Environmental Movement
Gulliver, R., Fielding, K. S., & Louis, W. R. (2019) · Environmental Communication · doi:10.1080/17524032.2019.1697326 · Cited by 32
“The final average inter-coder reliability was 0.809 (Krippendorff’s Alpha) and 0.81 (Cohen’s Kappa), which are considered acceptable (Hayes & Krippendorff, 2007).”
MethodInfluential
Migración, miedos y medios en la elección presidencial en Chile (2017)
Dammert, L., & Erlandsen, M. (2020) · Revista CS · doi:10.18046/recs.i31.3730 · Cited by 32
The effect of technology-assisted and project-based learning approaches on students’ attitudes towards mathematics and their academic achievement
Demir, C. G., & Önal, N. (2021) · Education and Information Technologies · doi:10.1007/s10639-020-10398-8 · Cited by 32
Biological motion perception links diverse facets of theory of mind during middle childhood
Warnell, K. R., Anderson, L. C., Velnoskey, K. R., Thompson, J. C., & Redcay, E. (2015) · Journal of Experimental Child Psychology · doi:10.1016/j.jecp.2015.09.003 · Cited by 37
“Based on a subset of 23 participants, interrater reliability was high for both story types (Strange Stories Krippendorff’s alpha = .77; Control Stories Krippendorff’s alpha = .81) (Fleiss, 1981; Hayes & Krippendorff, 2007).”
Background
Conceptualizing Beauty: A Content Analysis of U.S. and French Women’s Fashion Magazine Advertisements
Morris, P. K., & Nichols, K. (2013) · Online Journal of Communication and Media Technologies · doi:10.29333/ojcmt/2410 · Cited by 32
Patients' experience post-lumbar fusion regarding back problems, recovery and expectations in terms of the international classification of functioning, disability and health
Abbott, A. D., Hedlund, R., & Tyni‐Lenné, R. (2010) · Disability and Rehabilitation · doi:10.3109/09638288.2010.533240 · Cited by 32
“Unlike other reliability measures, it can generalise across scales of measurement, can be used with any number of observers with or without missing data, and it satisfies all of the important criteria for a good measure of reliability [22].”
Background
The Impact of Media Coverage and Motivation on Performance-Relevant Information
Elenbaas, M., Boomgaarden, H. G., Schuck, A. R. T., & de Vreese, C. H. (2013) · Political Communication · doi:10.1080/10584609.2012.737411 · Cited by 33
A study of inter-annotator agreement for opinion retrieval
Bermingham, A., & Smeaton, A. F. (2009) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/1571941.1572127 · Cited by 32
“We use Krippendorﬀ’s alpha [2] for measuring inter-annotator agreement.”
Method
Do open‐ended survey questions on migration motives create coder variability problems?
Niedomysl, T., & Malmberg, B. (2008) · Population Space and Place · doi:10.1002/psp.493 · Cited by 32
Changing Their Mind About Confessing to Police: The Role of Contextual Factors in Crime Confession
Deslauriers‐Varin, N., Beauregard, É., & Wong, J. S. (2011) · Police Quarterly · doi:10.1177/1098611110392721 · Cited by 32
“Reliability was computed for this subsample (n = 70) using Krippendorff’s alpha (Hayes & Krippendorff, 2007) for each of the aforementioned variables.”
Method
Crisis in the air: An investigation of AirAsia’s crisis-response effectiveness based on frame alignment
Ohlrogge, F., der Land, S. F. V., & van der Meer, T. G. L. A. (2016) · Public Relations Review · doi:10.1016/j.pubrev.2016.09.002 · Cited by 32
“Framing theory provides researchers with a valuable approach for understanding individuals’ interpretations and their onstructions of meaning relating to relevant events (Gamson & Modigliani, 1989). One of the first scholars to define the oncept was Goffman (1974), who stated that individuals draw on “schemata of inte…”
Background
The Hegemony of Global Politics: News Coverage of Climate Change in a Small Country
Horta, A., Carvalho, A., & Schmidt, L. (2017) · Society & Natural Resources · doi:10.1080/08941920.2017.1295497 · Cited by 32
Use of ICT for acquiring, practicing and assessing algebraic expertise
Bokhove, C. (2011) · Utrecht University Repository (Utrecht University) · Cited by 32
In Focus, Out of Privacy: The Wearer's Perspective on the Privacy Dilemma of Camera Glasses
Bhardwaj, D., Ponticello, A., Tomar, S., Dabrowski, A., & Krombholz, K. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642242 · Cited by 34
“Nonetheless, we also reset all pairs of glasses before handing them out again.”
Usability and inter-rater reliability of the NeuroMotion app: A tool in General Movements Assessments
Svensson, K., Örtqvist, M., Bos, A. F., Eliasson, A., & Sundelin, H. E. K. (2021) · European Journal of Paediatric Neurology · doi:10.1016/j.ejpn.2021.05.006 · Cited by 31
“We believe that K-alpha is best suited given that it can be used for two or more assessors and handle missing data [22].”
BackgroundMethod
The Use of the Exploratory Sequential Approach in Mixed-Method Research: A Case of Contextual Top Leadership Interventions in Construction H&S
Gogo, S., & Musonda, I. (2022) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph19127276 · Cited by 31
“The researcher produced a simple, short set of guidelines to precede every survey section to ensure understanding of the context and answering requirements [41].”
BackgroundMethodInfluential
Reproducible measurable residual disease detection by multiparametric flow cytometry in acute myeloid leukemia
Röhnert, M. A., Krämer, M., Schadt, J., Ensel, P., Thiede, C., Krause, S. W., Bücklein, V., Hoffmann, J., Jaramillo, S., Schlenk, R. F., et al. (2022) · Leukemia · doi:10.1038/s41375-022-01647-5 · Cited by 34
“Statistical analysis To define IRR, Kα, a reliability coefficient ranging from 0 to 1, with 1 representing perfect agreement between multiple raters [41], was calculated.”
Method
From ideas to innovations: The role of individuals in idea implementation
Valtonen, A., Kimpimäki, J., & Malacina, I. (2023) · Creativity and Innovation Management · doi:10.1111/caim.12577 · Cited by 35
An investigation of care-based vs. rule-based morality in frontotemporal dementia, Alzheimer's disease, and healthy controls
Carr, A. R., Paholpak, P., Daianu, M., Fong, S. S., Mather, M. J., Jimenez, E. E., Thompson, P. M., & Mendez, M. F. (2015) · Neuropsychologia · doi:10.1016/j.neuropsychologia.2015.09.033 · Cited by 32
Use of effective and efficient quality control methods to implement psychosocial interventions.
Garland, A. F., & Schoenwald, S. K. (2013) · Clinical Psychology Science and Practice · doi:10.1111/cpsp.12021 · Cited by 31
Reporting Demonstrations: On Episodic and Thematic Coverage of Protest Events in Belgian Television News
Wouters, R. (2015) · Political Communication · doi:10.1080/10584609.2014.958257 · Cited by 31
A systematic review of media parenting in the context of childhood obesity research
Aftosmes‐Tobio, A., Ganter, C., Kronsteiner-Gicevic, S., Newlan, S., Simon, C. L., Davison, K. K., & Manganello, J. A. (2016) · BMC Public Health · doi:10.1186/s12889-016-2981-5 · Cited by 31
“We assessed inter-coder reliability using both simple percent agreement and the Krippendorff ’s alpha (kalpha; α) coefficient [22] for each coding category by double-coding a randomly selected sample of eligible studies (15 %; n = 15).”
Method
The news framing of the ‘Syria fighters’ in Flanders and the Netherlands: Victims or terrorists?
Berbers, A., Joris, W., Boesman, J. L. J., d’Haenens, L., Koeman, J., & Gorp, B. V. (2015) · Ethnicities · doi:10.1177/1468796815603753 · Cited by 31
“A 10% random sample was tested for intercoder reliability (n1⁄4 42) using Krippendorffs’ Alpha (Hayes and Krippendorff, 2007).”
Method
Team-based primary care: The medical assistant perspective
Sheridan, B., Chien, A. T., Peters, A. S., Rosenthal, M. B., Brooks, J. V., & Singer, S. J. (2016) · Health Care Management Review · doi:10.1097/hmr.0000000000000136 · Cited by 31
“Fleiss_ Kappa, an interrater reliability coefficient that allows formore than two coders and corrects for the possibility of matches that occur by chance (Hayes & Krippendorff, 2007), indicated substantial agreement at 0.8.”
BackgroundInfluential
Significant change events in psychodynamic psychotherapy: Is cognition or emotion more important?
McCarthy, K. L., Caputi, P., & Grenyer, B. F. S. (2017) · Psychology and Psychotherapy Theory Research and Practice · doi:10.1111/papt.12116 · Cited by 31
Bad News: The Changing Coverage of National Leaders in Foreign Media of Western Democracies
Balmas, M. (2017) · Mass Communication & Society · doi:10.1080/15205436.2017.1323104 · Cited by 30
“According to Hayes and Krippendorff (2007), with α = .70, the risk of accepting the data as reliable when they are not is quite low (q = 0.0125).”
BackgroundInfluential
Fully contextualized, frequency-based personality measurement: A replication and extension
Robie, C., Risavy, S. D., Holtrop, D., & Born, M. P. (2017) · Journal of Research in Personality · doi:10.1016/j.jrp.2017.05.005 · Cited by 31
Using the Structured Interview of Personality Organization for DSM–5 Level of Personality Functioning Rating Performed by Inexperienced Raters
Preti, E., Pierro, R. D., Costantini, G., Benzi, I. M. A., Panfilis, C. D., & Madeddu, F. (2018) · Journal of Personality Assessment · doi:10.1080/00223891.2018.1448985 · Cited by 31
“We first assessed the reliability of LPFS ratings provided by students (Hayes & Krippendorff, 2007; Krippendorff, 2004; Shrout & Fleiss, 1979).”
Method
Let CONAN tell you a story: Procedural quest generation
Breault, V., Ouellet, S., & Davies, J. (2021) · Entertainment Computing · doi:10.1016/j.entcom.2021.100422 · Cited by 31
The elephant in the record: On the multiplicity of data recording work
Cabitza, F., Locoro, A., Alderighi, C., Rasoini, R., Compagnone, D., & Berjano, P. L. (2019) · Health Informatics Journal · doi:10.1177/1460458218824705 · Cited by 32
“For the sections of the forms that encompass the mutually exclusive type of multiple choice items (rendered as radio buttons on Web forms), we employed a chance-adjusted measure, Krippendorff’s agreement coefficient alpha .”
MethodInfluential
Learner profiles in secondary education: Occurrence and relationship with performance and student characteristics
Rogiers, A., Merchie, E., & Keer, H. V. (2019) · The Journal of Educational Research · doi:10.1080/00220671.2018.1538093 · Cited by 33
“Two independent trained coders double-scored 23% of the scratch papers, resulting in a high interrater reliability (Krippendorff’s a¼ .93 and .91, respectively; Hayes & Krippendorff, 2007).”
Method
Measuring Quality Technology Integration in Science Classrooms
Parker, C. E., Stylinski, C. D., Bonney, C. R., DeLisi, J., Wong, J., & Doty, C. (2019) · Journal of Science Education and Technology · doi:10.1007/s10956-019-09787-7 · Cited by 31
“Krippendorf ’ s alpha was used to determine reliability; Krippendorf ’ s alpha is appropriate for this analysis because it “ generalizes across scales of measurement; can be used with any number of observers, with or without missing data ” (Hayes and Krippendorf 2007, p. 78).”
Method
Governance and Type of Industry as Determinants of Corporate Social Responsibility Disclosures in Latin America
Gómez, N. A., & García, S. M. (2019) · Latin American Business Review · doi:10.1080/10978526.2019.1697185 · Cited by 31
“…and accuracy), reproducibility is the strongest and most reliable, because it allows several coders, under clear coding rules, to simultaneously analyze and check the level of coincidence of the coding process, under a specific measurement unit previously agreed upon (Hayes & Krippendorff, 2007).”
Background
Biomedicalization and Media in Comparative Perspective: Audiences, Frames, and Actors in Norwegian, Spanish, U.K. and U.S. Health News
Hallin, D. C., Figenschou, T. U., & Thorbjørnsrud, K. (2020) · The International Journal of Press/Politics · doi:10.1177/1940161220960415 · Cited by 31
“Intercoder reliability calculated using Krippendorff’s a (Hayes and Krippendorff 2007) averaged .71, and ranged from .63 (for Biomedical Frame) to .85 for Citizen Target Audience.”
Method
Mainstream Protest Reporting in the Contemporary Media Environment: Exploring (In)Stability and Adherence to Protest Paradigm From 1998 to 2017
Gil‐López, T. (2020) · Journalism & Mass Communication Quarterly · doi:10.1177/1077699020984783 · Cited by 31
“After the coding of the reliability sample was ended, Krippendorff’s alpha (Hayes & Krippendorff, 2007) was computed to assess the overall consistency of the content categories.”
Method
Knowledge grounded medical dialogue generation using augmented graphs
Varshney, D., Zafar, A., Behera, N. K., & Ekbal, A. (2023) · Scientific Reports · doi:10.1038/s41598-023-29213-8 · Cited by 33
Ethical leadership and follower unethical pro-organizational behavior: examining the dual influence mechanisms of affective and continuance commitments
Park, J. G., Zhu, W., Kwon, B., & Bang, H. (2023) · The International Journal of Human Resource Management · doi:10.1080/09585192.2023.2191209 · Cited by 36
“Table 4 summarizes the results of a bootstrapping test with 10,000 bootstrap samples (Hayes, 2013; Hayes & Krippendorff, 2007) on the differential mediating roles of two types of organizational commitment (i.e. affective commitment and continuance commitment).”
Background
Preferred Women’s Waist-to-Hip Ratio Variation over the Last 2,500 Years
Bovet, J., & Michel Raymond, A. (2015) · PLoS ONE · doi:10.1371/journal.pone.0123284 · Cited by 31
“Tentatively, Krippendorff’s alpha [32] was used to estimate this value.”
Assessing students’ use of LinkedIn in a business and professional communication course
Slone, A., & Gaffney, A. L. H. (2016) · Communication Teacher · doi:10.1080/17404622.2016.1219043 · Cited by 31
“Using Krippendorff ’ s alpha (Hayes & Krippendorff, 2007) as a measure of reliability, we were consistently reliable: nominal-level data (55 coding decisions), α = .82; one ordinal level criteria (extent of first-person language used), α = .73; ratio-level data (35 coding decisions), α = .95.”
Sixth-grade students’ engagement in academic tasks
Parsons, S. A., Malloy, J. A., Parsons, A. W., Peters‐Burton, E. E., & Burrowbridge, S. C. (2016) · The Journal of Educational Research · doi:10.1080/00220671.2016.1246408 · Cited by 40
“Interrater reliability (aK) of the task openness score was .73, with a 95% confidence interval ranging from .59 to .84, which is an acceptable level of reliability when using this calculation (Hayes & Krippendorff, 2007).”
Method
Entrepreneurial tweaking
Hunt, R. A. (2013) · European Journal of Innovation Management · doi:10.1108/14601061311324511 · Cited by 31
“Both reliability correlations are well above the levels necessary to establish inter-rater validity (Lombard, Snyder-Duch Bracken 2002; Hayes & Krippendorff ,2007).”
Fictive motion in the context of mountaineering
Egorova, E., Tenbrink, T., & Purves, R. S. (2018) · Spatial Cognition and Computation · doi:10.1080/13875868.2018.1431646 · Cited by 30
“An interrater reliability analysis using Krippendorff’s alpha (Hayes & Krippendorff, 2007) was performed to determine consistency of the identified markers.”
Method

Authors: Hayes, A. F., & Krippendorff, K.
Affiliations: The Ohio State University (US)
University of Pennsylvania (US)
ORCID: Hayes, Andrew F.
Krippendorff, Klaus
Venue: Communication Methods and Measures
Publisher: Informa UK Limited
Published: 2007-04
Volume: 1
Issue: 1
Pages: 77-89
ISSN: 1931-2458 (ISSN-L)
1931-2466
DOI: 10.1080/19312450709336664
OpenAlex: W2061504941
Cited by: 4,225
References: 30
Citations by year: 2026: 284 · 2025: 278 · 2024: 280 · 2023: 289 · 2022: 331 · 2021: 379 · 2020: 369 · 2019: 296 · 2018: 260 · 2017: 300 · 2016: 248 · 2015: 205 · 2014: 206 · 2013: 150 · 2012: 107
FWCI: 50.07
Type: journal article
Language: en
Open access: closed
Locations: Communication Methods and Measures (publishedVersion)
Topics: Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Reliability and Agreement in Measurement
Social Sciences › General Social Sciences › Computational and Text Analysis Methods
Social Sciences › Decision Sciences › Management Science and Operations Research › Psychometric Methodologies and Testing
Keywords: Krippendorff's alpha, inter-rater reliability, content analysis, coding reliability, reliability coefficients, observer agreement, SAS macro, missing data

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Answering the Call for a Standard Reliability Measure for Coding Data

Abstract

Overview

Integrity signals

Verification

Claims the paper makes

Claims other papers make about this one

Citations

Cites

Cited by

Related

Metadata

Record

About these checks