The DOI matches its Crossref record; 30 of the 30 references it cites were found in a registry or an index.
Answering the Call for a Standard Reliability Measure for Coding Data
Abstract
In content analysis and similar methods, data are typically generated by trained human observers who record or transcribe textual, pictorial, or audible matter in terms suitable for analysis. Conclusions from such data can be trusted only after demonstrating their reliability. Unfortunately, the content analysis literature is full of proposals for so-called reliability coefficients, leaving investigators easily confused, not knowing which to choose. After describing the criteria for a good measure of reliability, we propose Krippendorff's alpha as the standard reliability measure. It is general in that it can be used regardless of the number of observers, levels of measurement, sample sizes, and presence or absence of missing data. To facilitate the adoption of this recommendation, we describe a freely available macro written for SPSS and SAS to calculate Krippendorff's alpha and illustrate its use with a simple example.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“We calculate Krippendorff's alpha (Hayes and Krippendorff, 2007) to measure inter-rater reliability (see Table 3).”
From LLMs cannot find reasoning errors, but can correct them given the error location (2024) · cites this paper as Hayes and Krippendorff, 2007
In the citing paper
“Based on Krippendorff's alpha (α = 0.92), the level of agreement between coders was high and well above the threshold of 0.70 (Hayes & Krippendorff, 2007).”
From Collaboration, Coordination, and Cooperation Among Organizations: Establishing the Distinctive Meanings of These Terms Through a Systematic Literature Review (2020) · cites this paper as Hayes & Krippendorff, 2007
In the citing paper
“we calculated the inter-coder reliability based on 25% of all data units and derived a kappa value of .96, which reflects excellent reliability.”
In the citing paper
“To assess the inter-annotator agreement of stage 1, we report Krippendorf's α (Hayes and Krippendorff, 2007) for each dimension for the 135 articles after training and before any discussion/adjudication: economic (0.44), foreign (0.68), social (0.39).”
From Political Ideology and Polarization: A Multi-dimensional Approach (2022) · cites this paper as Hayes and Krippendorff, 2007
In the citing paper
“This family of metrics does not correct for chance (Cohen, 1960;Hayes and Krippendorff, 2007) and overstates discriminative ability (Guerdan et al., 2025;Han et al., 2025;Collot et al., 2025).”
From Reliability without Validity: A Systematic, Large-Scale Evaluation of LLM-as-a-Judge Models Across Agreement, Consistency, and Bias (2026) · cites this paper as Hayes and Krippendorff, 2007
Citations
Cites
Coefficient Alpha and the Internal Structure of Tests
Cronbach, L. J. (1951) · Psychometrika · doi:10.1007/bf02310555 · Cited by 43,931
A Coefficient of Agreement for Nominal Scales
Cohen, J. (1960) · Educational and Psychological Measurement · doi:10.1177/001316446002000104 · Cited by 42,398
An Introduction to the Bootstrap
Efron, B., & Tibshirani, R. J. (1994) · doi:10.1201/9780429246593 · Cited by 40,500
Nonparametric Statistics for the Behavioral Sciences
Guitton, H., & Siegel, S. (1958) · Revue économique · doi:10.2307/3498751 · Cited by 35,279
Content Analysis: An Introduction to its Methodology.
Shelley, M., & Krippendorff, K. (1984) · Journal of the American Statistical Association · doi:10.2307/2288384 · Cited by 24,351
Self-citationStatistical Methods for Rates and Proportions
Fleiss, J. L., Levin, B., & Paik, M. C. (2003) · Wiley Series in Probability and Statistics · doi:10.1002/0471445428 · Cited by 18,646
Measuring nominal scale agreement among many raters.
Fleiss, J. L. (1971) · Psychological Bulletin · doi:10.1037/h0031619 · Cited by 8,733
Content Analysis for the Social Sciences and Humanities.
Parker, E. B., & Holsti, O. R. (1970) · American Sociological Review · doi:10.2307/2093233 · Cited by 4,553
Content Analysis in Mass Communication: Assessment and Reporting of Intercoder Reliability
Lombard, M., Snyder-Duch, J., & Bracken, C. C. (2002) · Human Communication Research · doi:10.1111/j.1468-2958.2002.tb00826.x · Cited by 3,068
An Introduction to the Bootstrap
Grimshaw, S. D., Efron, B., & Tibshirani, R. J. (1995) · Technometrics · doi:10.2307/1269918 · Cited by 2,281
Research works printed but not linked: 1
- 10.1111/j.1468-2958.2004.tb00738.x · DOI · Paper page
Cited by
Sorted by citations
Items 701–800 of 1,013
Knobloch‐Westerwick, S., & Hoplamazian, G. J. (2011) · Communication Research · doi:10.1177/0093650211425040 · Cited by 44
“Analysis utilizing Hayes and Krippendorff’s (2007) SPSS macro for calculating intercoder reliability revealed the coders were highly consistent in determining selective reading for the nine magazine issues in 5-second intervals (Krippendorff’s alpha = .”
MethodStrategies of Successful Synthesis Solutions: Mapping, Mechanisms, and More
Bodé, N. E., & Flynn, A. B. (2016) · Journal of Chemical Education · doi:10.1021/acs.jchemed.5b00900 · Cited by 45
The Iterative Reweighted Mixed-Norm Estimate for Spatio-Temporal MEG/EEG Source Reconstruction
Strohmeier, D., Bekhti, Y., Haueisen, J., & Gramfort, A. (2016) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2016.2553445 · Cited by 44
“To evaluate the stability of the reconstructed support, we compute Krippendorff’s alpha [56].”
MethodZiegler, W., Staiger, A., Schölderle, T., & Vogel, M. (2017) · Journal of Speech Language and Hearing Research · doi:10.1044/2017_jslhr-s-16-0336 · Cited by 44
MOOC Success Factors: Proposal of an Analysis Framework
Azevedo, J., & Marques, M. (2017) · Journal of Information Technology Education Innovations in Practice · doi:10.28945/3861 · Cited by 46
Interrupted Time Experience in Autism Spectrum Disorder: Empirical Evidence from Content Analysis
Vogel, D. H. V., Falter‐Wagner, C. M., Schoofs, T., Krämer, K., Kupke, C., & Vogeley, K. T. (2018) · Journal of Autism and Developmental Disorders · doi:10.1007/s10803-018-3771-y · Cited by 47
Maerz, S. F., & Schneider, C. Q. (2019) · Quality & Quantity · doi:10.1007/s11135-019-00885-7 · Cited by 44
“Ideally, all manual coders classify all speech independent from each other and their scores are then tested for intercoder reliability, e.g. with Krippendorff’s alpha (Hayes and Krippendorff 2007).”
MethodAssessing the essential pre-conditions of an authentic sustainability curriculum
Lengyel, A., Szőke, S., Kovács, S., Dávid, L. D., Bába, É. B., & Müller, A. (2019) · International Journal of Sustainability in Higher Education · doi:10.1108/ijshe-09-2018-0150 · Cited by 51
Hintz, E. A., & Brown, C. L. (2019) · Communication Monographs · doi:10.1080/03637751.2019.1697891 · Cited by 44
van Eldik, A. K., Kneer, J., Lutkenhaus, R., & Jansz, J. (2019) · Frontiers in Psychology · doi:10.3389/fpsyg.2019.02876 · Cited by 44
“The intercoder agreement based on a subsample of the videos had a Krippendorff ’s α = 0.81 (Hayes and Krippendorff, 2007).”
MethodInfluentialCerebral Microbleeds and Treatment Effect of Intravenous Thrombolysis in Acute Stroke
Schlemm, L., Braemswig, T. B., Boutitie, F., Vynckier, J., Jensen, M., Galinović, I., Simonsen, C. Z., Cheng, B., Cho, T., Fiehler, J., et al. (2021) · Neurology · doi:10.1212/wnl.0000000000013055 · Cited by 43
“Large vessel occlusion, n (%) 85 (19) 67 (19) 18 (19) 10 (33) 6 (17) 2 (13) >0.”
InfluentialFactors affecting intercoder reliability: a Monte Carlo experiment
Feng, G. C. (2012) · Quality & Quantity · doi:10.1007/s11135-012-9745-9 · Cited by 47
Systematic and Random Disagreement and the Reliability of Nominal Data
Krippendorff, K. (2008) · Communication Methods and Measures · doi:10.1080/19312450802467134 · Cited by 44
“Software described by Hayes and Krippendorff (2007) also produces confidence limits for α. Confidence limits are important as statistical tests of the null hypotheses, demanded by Berry and Mielke (1988) among others, are quite meaningless as they fail to inform situations in which reliability matters.”
BackgroundInfluentialSelf-citationReconciling Passive and Motivated Learning
Elenbaas, M., de Vreese, C. H., Schuck, A. R. T., & Boomgaarden, H. G. (2012) · Communication Research · doi:10.1177/0093650212467032 · Cited by 44
Pauses in spontaneous written communication: A keystroke logging study
Chukharev‐Hudilainen, E. (2014) · Journal of Writing Research · doi:10.17239/jowr-2014.06.01.3 · Cited by 43
“Krippendorff’s α (Hayes & Krippendorff, 2007; Krippendorff, 2007) was used as a measure of inter-coder reliability.”
MethodFojas, C. L., Kim, J., Minsky-Rowland, J. D., & Algee-Hewitt, B. F. B. (2018) · American Journal of Physical Anthropology · doi:10.1002/ajpa.23342 · Cited by 45
“To evaluate interrater reliability and agreement of the raw scores, Krippendorff’s alpha values were calculated (Hayes & Krippendorff, 2007; Krippendorff, 2011).”
MethodHow to enhance consumer experience over mobile wallet: a data-driven approach
Shankar, A., & Behl, A. (2021) · Journal of Strategic Marketing · doi:10.1080/0965254x.2021.1999306 · Cited by 43
“Kripendoff’s alpha was found to be 83.67, which is above the threshold value of 80% (Hayes & Krippendorff, 2007), confirming the inter-coder reliability.”
BackgroundInfluentialU.S. cities’ integration and evaluation of equity considerations into climate action plans
Caggiano, H., Kocakuşak, D., Kumar, P., & Tier, M. (2023) · npj Urban Sustainability · doi:10.1038/s42949-023-00129-6 · Cited by 43
Nordmann, E., Cleland, A. A., & Bull, R. (2014) · Acta Psychologica · doi:10.1016/j.actpsy.2014.03.009 · Cited by 43
“Hayes and Krippendorff (2007) suggest that the statistic of choice for inter-rater reliability should be Krippendorff's alpha.”
BackgroundStinson, P. M., Liederbach, J., Brewer, S. L., & Mathna, B. E. (2014) · Criminal Justice Policy Review · doi:10.1177/0887403414526231 · Cited by 43
“Police routinely operate alone and largely free from any direct supervision, either from administrators or fellow officers. Police commonly encounter citizens who are vulnerable, usually because they are victims, criminal suspects, or perceived as “suspicious” and subject to the power and coercive authority granted to…”
BackgroundInfluentialMass media coverage of HPV vaccination in Romania: a content analysis
a, M. A. P., & ban, A. B. (2014) · Health Education Research · doi:10.1093/her/cyu027 · Cited by 43
“Inter-coder reliability as measured by Krippendorff’s alpha [66] was 0.”
MethodVergeer, M., & Franses, P. H. (2015) · Information Communication & Society · doi:10.1080/1369118x.2015.1093526 · Cited by 43
“Coding reliability of political issues is sufficient: Krippendorf’s alpha = 0.769 (Hayes & Krippendorff, 2007).”
BackgroundInfluentialWang, L., Pedersen, P. C., Strong, D. M., Tulu, B., Agu, E., Ignotz, R., & He, Q. (2015) · Journal of Diabetes Science and Technology · doi:10.1177/1932296815599004 · Cited by 43
Sorin, G., Pasquier, G. J. M., Drumez, É., Arnould, A., Migаud, H., & Putman, S. (2016) · Orthopaedics & Traumatology Surgery & Research · doi:10.1016/j.otsr.2016.02.009 · Cited by 43
User Interactions and Permission Use on Android
Micinski, K., Votipka, D., Stevens, R., Kofinas, N., Mazurek, M. L., & Foster, J. S. (2017) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3025453.3025706 · Cited by 45
Coverage of Abortion in Select U.S. Newspapers
Woodruff, K. (2018) · Women s Health Issues · doi:10.1016/j.whi.2018.08.008 · Cited by 43
Monitoring stance towards vaccination in twitter messages
Kunneman, F., Lambooij, M., Wong, A., van den Bosch, A. P. J., & Mollema, L. (2020) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-020-1046-y · Cited by 2
“everal days, while participants recruited through the Radboud Research Participation System could only join for the duration of an hour. We calculated inter-annotator agreement by Krippendor’s Alpha [24], which accounts for dierent annotator pairs and empty values. To also zoom in on the particular agreement by catego…”
MethodPiñeiro-Naval, V. (2020) · Communication & Society · doi:10.15581/003.33.3.1-15 · Cited by 51
Detecting suicide risk using knowledge-aware natural language processing and counseling service data
Xu, Z., Xu, Y., Cheung, F., Cheng, M. N. S., Lung, D. W. M., Law, Y. W., Chiang, B., Zhang, Q., & Yip, P. S. F. (2021) · Social Science & Medicine · doi:10.1016/j.socscimed.2021.114176 · Cited by 44
Linder, N., Rosenthal, S., Sörqvist, P., & Barthel, S. (2021) · Frontiers in Psychology · doi:10.3389/fpsyg.2021.699410 · Cited by 56
“We assessed interrater reliability using Krippendorff ’s alpha (Hayes and Krippendorff, 2007) and found good reliability (α = 0.92, [95% CI: 0.88, 0.96]).”
MethodInfluentialBehavioral and cognitive differences in early childhood related to prenatal marijuana exposure
Murnan, A., Keim, S. A., Yeates, K. O., Boone, K. M., Sheppard, K. W., & Klebanoff, M. A. (2021) · Journal of Applied Developmental Psychology · doi:10.1016/j.appdev.2021.101348 · Cited by 43
Retirement intentions of general practitioners aged 45–65 years
Brett, T. D., Arnold-Reed, D. E., Hince, D. A., Wood, I. K., & Moorhead, R. G. (2009) · The Medical Journal of Australia · doi:10.5694/j.1326-5377.2009.tb02804.x · Cited by 43
Kaendler, C., Wiedmann, M., Leuders, T., Rummel, N., & Spada, H. (2016) · Psychology Learning & Teaching · doi:10.1177/1475725716638010 · Cited by 43
“As was expected due to the complexity of the subject, interrater reliability of these ratings was unsatisfactory, Krippendorff’s a¼ .34 (Freelon, 2010; Hayes & Krippendorff, 2007).”
BackgroundPress Effects, Public Opinion, and Gender: Coverage of Sarah Palin’s Vice-Presidential Campaign
Miller, M. K., & Peake, J. S. (2013) · The International Journal of Press/Politics · doi:10.1177/1940161213495456 · Cited by 42
Explainable Claim Verification via Knowledge-Grounded Reasoning with Large Language Models
Wang, H., & Shu, K. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.416 · Cited by 83
A systemic review of AI for interdisciplinary learning: Application contexts, roles, and influences
Cai, C., Zhu, G., & Ma, M. (2024) · Education and Information Technologies · doi:10.1007/s10639-024-13193-x · Cited by 42
“We used Krippendorff’s alpha, which is commonly used in content analysis (Hayes & Krippendorff, 2007), to measure the agreement level between the two researchers, and found that the Krippendorff’s alpha reliability was 0.83, indicating good inter-coder agreement (Krippendorff, 2018).”
InfluentialWorkers' Moral Mum Effect: On Facework and Unethical Behavior in the Workplace
Bisel, R. S., Kelley, K. M., Ploeger, N. A., & Messersmith, J. G. (2011) · Communication Studies · doi:10.1080/10510974.2010.551314 · Cited by 47
“Measures of initial coding reliability and coder drift were computed as Krippendorff's α on a randomly selected sample of 29 responses (15%; Hayes & Krippendorff, 2007 ; Krippendorff, 2004 ).”
MethodEnvironmental scanning the future of event design
Adema, K. L., & Roehl, W. S. (2009) · International Journal of Hospitality Management · doi:10.1016/j.ijhm.2009.10.017 · Cited by 44
Wilson, S. R., Wilkum, K., Chernichky, S. M., Wadsworth, S. M., & Broniarczyk, K. M. (2011) · Journal of Applied Communication Research · doi:10.1080/00909882.2011.585399 · Cited by 42
“Hayes and Krippendorff (2007) argue that S will be inflated when coders must place each unit into one of a large number of categories, some of which are rarely used.”
BackgroundInvestigating design: A comparison of manifest and latent approaches
Cash, P. J., & Snider, C. M. (2014) · Design Studies · doi:10.1016/j.destud.2014.02.005 · Cited by 42
“To test inter-‐coder reliability a number of different measures can be used, such as, Cohen’s kappa (Cohen, 1960), and Krippendorff’s alpha (Hayes and Krippendorff, 2007), although percentage agreement should not be used (Hayes and …”
The Rural Context of Illicit Substance Offers
Pettigrew, J., Miller-Day, M. A., Krieger, J. L. R., & Hecht, M. L. (2012) · Journal of Adolescent Research · doi:10.1177/0743558411432639 · Cited by 42
“Next, an additional 10% of total units were analyzed using the agreed on coding categories and employing Krippendorff’s alpha (alpha = .92; Hayes & Krippendorff, 2007) as the index of reliability between two independent observers.”
MethodVandevelde, S., Keer, H. V., & Merchie, E. (2016) · The Journal of Educational Research · doi:10.1080/00220671.2014.999363 · Cited by 50
“Two trained coders independently double-coded 38% of the protocols, resulting in high interrater reliability for the main categories (Krippendorff’s a D .97) and subcategories (Krippendorff’s a D .96) of the coding scheme (Hayes & Krippendorff, 2007).”
MethodOral Motor Abilities Are Task Dependent: A Factor Analytic Approach to Performance Rate
Staiger, A., Schölderle, T., Brendel, B., Bötzel, K., & Ziegler, W. (2016) · Journal of Motor Behavior · doi:10.1080/00222895.2016.1241747 · Cited by 42
“Computations of Krippendorff’s alpha using SPSS 23 (Hayes & Krippendorff, 2007) with 3,000 bootstrapping cycles revealed reliability values of .98 (95% CI: [.96, .99]) for articulation rates, .96 (95% CI: [.87, .99]) for buh repetition rates, .94 (95% CI: [.86, .99]) for duh repetition rates, .99…”
BackgroundMethodTarragó, R., Iglesias, X., Ajamil, D. L., Anguera, M. T., Sanchís, L. R., & Arana, J. (2017) · Anales de Psicología · doi:10.6018/analesps.33.3.271041 · Cited by 42
“The agreement coefficient for canonical data (agreement among panel of experts) was calculated using an external command (Hayes & Krippendorff, 2007) linked to PASW Statistics for Windows (v.18, SPSS Inc., Chicago, IL).”
Methodde Korte, E., Wiezer, N. M., Roozeboom, M. B., Vink, P., & Kraaij, W. (2018) · JMIR mhealth and uhealth · doi:10.2196/mhealth.6363 · Cited by 45
Context-Aware Intent Identification in Email Conversations
Wang, W., Hosseini, S., Awadallah, A. H., Bennett, P. N., & Quirk, C. (2019) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3331184.3331260 · Cited by 42
“To make sure that the comparison between these two annotations is fair and not biased toward the annotation behavior, we calculated the Krippendorff’s α agreement score [20] for inter-annotator agreement among multiple judges.”
MethodEvaluating an inclusive blended learning environment in EFL: a flipped approach
Andújar, A., & Nadif, F. Z. (2020) · Computer Assisted Language Learning · doi:10.1080/09588221.2020.1774613 · Cited by 51
RACE: Retrieval-augmented Commit Message Generation
Shi, E., Wang, Y., Tao, W., Du, L., Zhang, H., Han, S., Zhang, D., & Sun, H. (2022) · Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2022.emnlp-main.372 · Cited by 42
Validity and reliability of grade scoring in the diagnosis of exercise-induced laryngeal obstruction
Walsted, E. S., Hull, J. H., Hvedstrup, J., Maat, R. C., & Backer, V. (2017) · ERJ Open Research · doi:10.1183/23120541.00070-2017 · Cited by 42
Lanosga, G., & Martin, J. A. (2016) · Journalism · doi:10.1177/1464884916683555 · Cited by 41
“Reliability testing based on conservative estimates using Krippendorff’s alpha yielded coefficients ranging from .74 to .95 for the 10 main variables analyzed (Hayes and Krippendorff, 2007).”
BackgroundMethodInfluentialRobnett, R. D., Wertheimer, M., & Tenenbaum, H. R. (2017) · Sex Roles · doi:10.1007/s11199-017-0856-6 · Cited by 41
Reviewing and analyzing peer review Inter-Rater Reliability in a MOOC platform
García-Loro, F., Martín, S., Ruipérez‐Valiente, J. A., Sancristobal, E., & Castro, M. A. (2020) · Computers & Education · doi:10.1016/j.compedu.2020.103894 · Cited by 47
Ardiana, P. A. (2021) · Meditari Accountancy Research · doi:10.1108/medar-12-2019-0666 · Cited by 41
Nguyen, D., & Hekman, E. (2022) · Global Media and China · doi:10.1177/20594364221078626 · Cited by 44
“The intercoder reliability scores reached satisfactory levels for all variables (KALPHA = 0.75–0.85, Hayes & Krippendorff 2007).”
BackgroundInfluentialSafety intelligence: An exploration of senior managers' characteristics
Fruhen, L. S., Mearns, K. J., Flin, R. H., & Kirwan, B. (2013) · Applied Ergonomics · doi:10.1016/j.apergo.2013.11.012 · Cited by 41
“Interrater reliability of the coding was tested with Krippendorff's alpha (Hayes & Krippendorff, 2007) and achieved an agreement of α = .76 (95% CILL .73 to CIUL.80).”
MethodComparison of Visual and DNA Breed Identification of Dogs and Inter-Observer Reliability
Voith, V. L., Trevejo, R. T., Dowling‐Guyer, S., Chadik, C., Marder, A. R., Johnson, V., & Irizarry, K. J. L. (2013) · Journal of Sociological Research · Cited by 41
The Development of a Checklist to Enhance Methodological Quality in Intervention Programs
Moscoso, S. C., Sanduvete‐Chaves, S., & Sánchez-Martín, M. (2016) · Frontiers in Psychology · doi:10.3389/fpsyg.2016.01811 · Cited by 41
Mason, S. M., Kuczawski, M., Teare, M. D., Stevenson, M., Goodacre, S. W., Ramlakhan, S. L., Morris, F. A., & Rothwell, J. C. (2017) · BMJ Open · doi:10.1136/bmjopen-2016-014324 · Cited by 41
“862) suggesting good degree of reliability.(30)”
Inter-observer reliability of 4 fetal heart rate classifications
Garabédian, C., Butruille, L., Drumez, É., Servan-Schreiber, E., Bartolo, S., Bleu, G., Mesdag, V., Deruelle, P., Jonckheere, J. D., & Houfflin‐Debarge, V. (2017) · Journal of Gynecology Obstetrics and Human Reproduction · doi:10.1016/j.jogoh.2016.11.002 · Cited by 41
Arthur, C. L., Wu, J., Yago, M., & Zhang, J. (2017) · Corporate Governance · doi:10.1108/cg-05-2016-0124 · Cited by 41
Pontillo, G., Penna, S., Cocozza, S., Quarantelli, M., Gravina, M., Lanzillo, R., Marrone, S., Costabile, T., Inglese, M., Morra, V. B., et al. (2022) · European Radiology · doi:10.1007/s00330-022-08610-z · Cited by 44
“The stability of the SuStaIn subtypes over time was expressed with Krippendorf ’ s α [18].”
MethodKeinz, P., & Prügl, R. (2010) · Creativity and Innovation Management · doi:10.1111/j.1467-8691.2010.00570.x · Cited by 46
“Variables (1) to (5) were coded independently by two geographically dislocated coders based on a three-step process (Krippendorff 2004, Hayes and Krippendorff 2007): (1) Training, (2) Individual Coding, (3) Discussion and optional modification.”
Survivors of institutional abuse in long-term child care in Scotland
Carr, A., Nearchou, F., Duff, H., Mhaoileoin, D. N., Cullen, K., O’Dowd, A., & Battigelli, L. (2019) · Child Abuse & Neglect · doi:10.1016/j.chiabu.2019.04.018 · Cited by 40
Alert Now or Never: Understanding and Predicting Notification Preferences of Smartphone Users
Li, T., Haines, J. K., Eguino, M. F. R. D., Hong, J., & Nichols, J. W. (2022) · ACM Transactions on Computer-Human Interaction · doi:10.1145/3478868 · Cited by 40
Analyzing the public sentiment on COVID-19 vaccination in social media: Bangladesh context
Zulfiker, M. S., Kabir, N., Biswas, A. A., Zulfiker, S., & Uddin, M. S. (2022) · Array · doi:10.1016/j.array.2022.100204 · Cited by 40
“For validating the reliability of the polarity labelling of the volunteers, inter-rater reliability tests like Fleiss’ Kappa and Krippendorff’s Alpha reliability tests were performed.”
InfluentialMaxwell, B. G., Posner, K. L., Wong, J., Oakes, D. A., Kelly, N., Domino, K. B., & Ramamoorthy, C. (2014) · Congenital Heart Disease · doi:10.1111/chd.12188 · Cited by 40
Communication of climate projections in US media amid politicization of model science
Akerlof, K., Rowan, K. E., Fitzgerald, D., & Cedeno, A. Y. (2012) · Nature Climate Change · doi:10.1038/nclimate1542 · Cited by 42
Causal discourse analyzer: improving automated feedback on academic ESL writing
Chukharev‐Hudilainen, E., & Sarıcaoğlu, A. (2014) · Computer Assisted Language Learning · doi:10.1080/09588221.2014.991795 · Cited by 40
“%) was annotated by at least two people, and Krippendorff’s a (Hayes & Krippendorff, 2007; Krippendorff, 2011) was calculated to assess inter-coder reliability.”
MethodJain, P., & Slater, M. D. (2013) · Journal of Health Communication · doi:10.1080/10810730.2012.757388 · Cited by 40
Generative sensing in design evaluation
Dong, A., Garbuio, M., & Lovallo, D. P. (2016) · Design Studies · doi:10.1016/j.destud.2016.01.003 · Cited by 40
Inter‐Rater Reliability of Total Body Score—A Scale for Quantification of Corpse Decomposition
Nawrocka, M., Frątczak‐Łagiewska, K., & Matuszewski, S. (2016) · Journal of Forensic Sciences · doi:10.1111/1556-4029.13105 · Cited by 41
“which is used to estimate the degree of agreement between scores that different people assign to the same objects (17).”
BackgroundEN-ES-CS: An English-Spanish Code-Switching Twitter Corpus for Multilingual Sentiment Analysis
Vilares, D., Alonso, M. Á., & Gómez‐Rodríguez, C. (2016) · Language Resources and Evaluation Conference (LREC) · doi:10.63317/2cm8di77f76c · Cited by 40
Exploring high impact scholarship in research on student's evaluation of teaching (SET)
Spooren, P., Vandermoere, F., Vanderstraeten, R., & Pepermans, K. (2017) · Educational Research Review · doi:10.1016/j.edurev.2017.09.001 · Cited by 45
New labour rights indicators: Method and trends for 2000–15
Kučera, D., & Sari, D. (2018) · International Labour Review · doi:10.1111/ilr.12084 · Cited by 40
“This is one of the issues we intend to address in the future through more formal statistical tests of inter-coder reliability (e.g. Hayes and Krippendorff, 2007).”
BackgroundDisagreement Among Gastroenterologists Using the Mayo and Rutgeerts Endoscopic Scores
Fernandes, S. R., Pinto, J. S. L. D., da Costa, P. M., Correia, L., & GEDII (2018) · Inflammatory Bowel Diseases · doi:10.1093/ibd/izx066 · Cited by 40
Carr, A., Duff, H., & Craddock, F. (2018) · Trauma Violence & Abuse · doi:10.1177/1524838018777788 · Cited by 44
“Percentage agreement and Krippendorff’s a (Hayes & Krippendorff, 2007) were used to determine interrater agreement and reliability.”
MethodLandberg, M., Lee, B., & Noack, P. (2019) · Emerging Adulthood · doi:10.1177/2167696819831793 · Cited by 40
Study of the academic production on communication in Spain and Latin America
Piñeiro-Naval, V., & Morais, R. J. P. (2019) · Comunicar · doi:10.3916/c61-2019-10 · Cited by 40
“The statistical parameter used for the reliability calculationwas the Krippendorff alpha (Krippendorff, 2011; 2017), found by using the “Kalpha macro” (Hayes & Krippendorff, 2007) for SPSS (version 24).”
MethodThe Role of Social Media during Social Movements – Observations from the #metoo Debate on Twitter
Brünker, F., Wischnewski, M., Mirbabaie, M., & Meinert, J. (2020) · Hawaii International Conference on System Sciences (HICSS) · doi:10.24251/hicss.2020.288 · Cited by 40
“898, signaling inter-coder reliability [19].”
MethodBozarth, L., Saraf, A., & Budak, C. (2018) · Proceedings of the International AAAI Conference on Web and Social Media · doi:10.1609/icwsm.v14i1.7278 · Cited by 40
Theruvath, A. J., Siedek, F., Muehe, A. M., Garcia-Diaz, J., Kirchner, J., Martin, O., Link, M. P., Spunt, S. L., Pribnow, A. K., Rosenberg, J., et al. (2020) · Radiology · doi:10.1148/radiol.2020192508 · Cited by 40
“Agreements between whole-body DW MRI– and FDG PET/ MRI–based therapy response assessments after chemotherapy induction (complete response, partial response, stable disease, and progressive disease) were assessed using Krippendorff a (27).”
MethodRetirement intentions of general practitioners aged 45–65 years
Brett, T. D., Arnold-Reed, D. E., Hince, D., Wood, I. K., & Moorhead, R. G. (2009) · The Medical Journal of Australia · doi:10.5694/j.1326-5377.2009.tb02696.x · Cited by 40
R&D and innovation efforts during the COVID-19 pandemic: The role of universities
Bachmann, P., & Frutos-Bencze, D. (2022) · Journal of Innovation & Knowledge · doi:10.1016/j.jik.2022.100238 · Cited by 36
The Verbal and Nonverbal Correlates of the Five Flirting Styles
Hall, J. A., & Xing, C. (2014) · Journal of Nonverbal Behavior · doi:10.1007/s10919-014-0199-8 · Cited by 40
Semi-formal Evaluation of Conversational Characters
Artstein, R., Gandhe, S., Gerten, J., Leuski, A., & Traum, D. R. (2009) · Lecture notes in computer science · doi:10.1007/978-3-642-01748-3_2 · Cited by 40
“We also calculated confidence intervals for α following a bootstrapping method similar to that of [14]; however, we found that variation in reliability among different subsets of raters was typically larger than the 95% confidence interval for a specific set of raters.”
MethodEffect of alteplase on the CT hyperdense artery sign and outcome after ischemic stroke
Mair, G., von Kummer, R., Morris, Z., von Heijne, A., Bradey, N., Cala, L., Peeters, A. P., Farrall, A., Adami, A., Potter, G. M., et al. (2015) · Neurology · doi:10.1212/wnl.0000000000002236 · Cited by 40
“0 implies perfect disagreement.(15) We used multivariate ordinal regression to calculate common odds ratios (ORs) with 6-month OHS and change in HAS segment number as dependent variables.”
MethodExploring OHS trainers’ role in the transfer of training
Freitas, A. C., & da Silva, S. A. (2016) · Safety Science · doi:10.1016/j.ssci.2016.08.007 · Cited by 40
Muis, J. (2015) · Mobilization An International Quarterly · doi:10.17813/maiq.20.1.f211773qm103w030 · Cited by 39
Conceptual layers and strategies in tour planning
Tenbrink, T., & Seifert, I. (2010) · Cognitive Processing · doi:10.1007/s10339-010-0373-9 · Cited by 39
“For a representative subset of the data coded independently, the standard reliability measure Krippendorff’s alpha (computed using the SPSS macro reported in Hayes and Krippendorff 2007 ) was 0.777 for strategy indicators.”
MethodInfluentialStudents’ uncertainty management in the college classroom
Sollitto, M., Brott, J. B., Cole, C., Gil, E., & Selim, H. (2017) · Communication Education · doi:10.1080/03634523.2017.1372586 · Cited by 41
“Alpha reliability figures above .80 are considered appropriate (Hayes & Krippendorff, 2007; Krippendorff, 2004); we achieved a satisfactory reliability of .87.”
BackgroundDíaz, J., Pérez, J., Gallardo, C., & González-Prieto, Á. (2022) · Journal of Systems and Software · doi:10.1016/j.jss.2022.111520 · Cited by 47
Towards a framework for identifying business process redesign patterns
Zellner, G. (2013) · Business Process Management Journal · doi:10.1108/bpmj-mar-2012-0020 · Cited by 39
“To accomplish reliability the observers need to agree on the data they generate (Hayes and Krippendorff, 2007, p. 78).”
BackgroundMatthes, J., & Rauchfleisch, A. (2013) · Communication Quarterly · doi:10.1080/01463373.2013.822405 · Cited by 39
“Krippendorff’s alpha was calculated with a bootstrapping sample of 10,000 (Hayes & Krippendorff, 2007; a¼ .84).”
MethodTurner, J. S. (2013) · Communication & Sport · doi:10.1177/2167479513496222 · Cited by 39
“The .70 cut point used in this study is based on suggestions made in two in-depth examinations of intercoder reliability standards used in content analyses (Hayes & Krippendorff, 2007; Lombard et al., 2002).”
MethodKnobloch, L. K., Basinger, E. D., Wehrman, E. C., Ebata, A. T., & McGlaughlin, P. C. (2016) · Journal of Family Communication · doi:10.1080/15267431.2016.1146723 · Cited by 39
“Advantages of Krippendorff’s α include its computable sampling distribution, its flexibility for use with any number of judges, and its independence from the frequency distribution of thematic units across categories (Hayes & Krippendorff, 2007).”
BackgroundIn the line of fire: managing expatriates in hostile environments
Gannon, J. M., & Paraskevas, A. (2017) · The International Journal of Human Resource Management · doi:10.1080/09585192.2017.1322122 · Cited by 39
“Inter-coder reliability was calculated to 83.5% which is higher than generally accepted as the norm for a good reliability test 80% (Hayes & Krippendorff, 2007).”
BackgroundInfluentialFletcher‐Brown, J., Pereira, V. E., & Nyadzayo, M. W. (2017) · Journal of Business Research · doi:10.1016/j.jbusres.2017.05.031 · Cited by 39
Barber, A. T., Buehl, M. M., Beck, J. S., Ramirez, E. M., Gallagher, M. A., Nuland, L. R., & Archer, C. J. (2017) · Reading & Writing Quarterly · doi:10.1080/10573569.2017.1344942 · Cited by 45
“Krippendorff’s alpha can be used with multiple coders, any level of measurement, and various sample sizes to account for reliability beyond chance agreements (Hayes & Krippendorff, 2007).”
MethodRice, R. E., Gustafson, A., & Hoffman, Z. (2018) · Environmental Communication · doi:10.1080/17524032.2018.1430046 · Cited by 39
Blasbalg, U., Hershkowitz, I., Lamb, M. E., Karni‐Visel, Y., & Ahern, E. C. (2018) · Law and Human Behavior · doi:10.1037/lhb0000303 · Cited by 47
Metadata
- Authors
- Hayes, A. F., & Krippendorff, K.
- Affiliations
- The Ohio State University (US)
- University of Pennsylvania (US)
- Venue
- Communication Methods and Measures
- Publisher
- Informa UK Limited
- Published
- 2007-04
- Volume
- 1
- Issue
- 1
- Pages
- 77-89
- ISSN
- 1931-2458 (ISSN-L)
- 1931-2466
- OpenAlex
- W2061504941
- Cited by
- 4,225
- References
- 30
- Citations by year
- 2026: 284 · 2025: 278 · 2024: 280 · 2023: 289 · 2022: 331 · 2021: 379 · 2020: 369 · 2019: 296 · 2018: 260 · 2017: 300 · 2016: 248 · 2015: 205 · 2014: 206 · 2013: 150 · 2012: 107
- FWCI
- 50.07
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Communication Methods and Measures (publishedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Reliability and Agreement in Measurement
- Social Sciences › General Social Sciences › Computational and Text Analysis Methods
- Social Sciences › Decision Sciences › Management Science and Operations Research › Psychometric Methodologies and Testing
- Keywords
- Krippendorff's alpha, inter-rater reliability, content analysis, coding reliability, reliability coefficients, observer agreement, SAS macro, missing data
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
