The DOI matches its Crossref record; 5 of the 5 references it cites were found in a registry or an index.
Reliability in content analysis: some common misconceptions and recommendations.
Open accessAbstract
In a recent article published in this journal, Lombard, Snyder-Duch, and Bracken (2002) surveyed 200 content analyses for their reporting of reliability tests; compared the virtues and drawbacks of five popular reliability measures; and proposed guidelines and standards for their use. Their discussion revealed that numerous misconceptions circulate in the content analysis literature regarding how these measures behave and can aid or deceive content analysts in their effort to ensure the reliability of their data. This paper proposes three conditions for statistical measures to serve as indices of the reliability of data and examines the mathematical structure and the behavior of the five coefficients discussed by the authors, plus two others. It compares common beliefs about these coefficients with what they actually do and concludes with alternative recommendations for testing reliability in content analysis and similar data-making efforts.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“For the 6 moderator variables, Krippendorff’s alpha ranged from 0.81 to 0.99, and therefore above the acceptable level (Krippendorff, 2004).”
From The Power of Feedback Revisited: A Meta-Analysis of Educational Feedback Research (2020) · cites this paper as Krippendorff, 2004
The cited source says what the claim attributes to it.
In the citing paper
“Krippendorff's (2004) criterion of significance is therefore satisfied here too.”
From Weighted Krippendorff's alpha is a more reliable metrics for multi-coders ordinal annotations: experimental studies on emotion, opinion and coreference annotation (2014) · cites this paper as Krippendorff's (2004)
The cited source says what the claim attributes to it.
In the citing paper
“Krippendorff's α α α α (Krippendorff, 2004) considers chance independently from coders like Scott's π, but data reliability is estimated depending on disagreement instead of agreement:”
From Weighted Krippendorff's alpha is a more reliable metrics for multi-coders ordinal annotations: experimental studies on emotion, opinion and coreference annotation (2014) · cites this paper as Krippendorff, 2004
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“The criterion of data significance -at least five chance agreements per category -proposed by (Krippendorff, 2004) is greatly satisfied for the valence annotation (3 categories).”
From Weighted Krippendorff's alpha is a more reliable metrics for multi-coders ordinal annotations: experimental studies on emotion, opinion and coreference annotation (2014) · cites this paper as Krippendorff, 2004
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The publisher hasn’t deposited this paper’s reference list.
Research works printed but not linked: 13
- Bennett, E. M., Alpert, R., & Goldstein, A. C. (1954). Communications through limited response questioning. Public Opinion Quarterly, 18, 303-308.
- Craig, R. T. (1981). Generalization of Scott"s index of intercoder agreement. Public Opinion Quarterly, 45, 260-264.
- Goodman, L. A, & Kruskal, W. H. (1954). Measures of association for cross classifications. Journal of the American Statistical Association, 49, 732-764.
- Holsti, O. R. (1969). Content analysis for the social sciences and humanities. Reading, MA: Addison- Wesley.
- Hughes, M. A., & Garrett, D. E. (1990). Intercoder reliability estimation -Approaches in marketing: A generalizability theory framework for quantitative data. Journal of Marketing Research, 27, 185- 195.
- Krippendorff, K. (1970). Bivariate agreement coefficients for reliability data. In E. R. Borgatta & G. W. Bohrnstedt (Eds.), Sociological methodology 1970 (pp. 139-150). San Francisco, CA: Jossey Bass.
- Krippendorff, K. (2004). Content analysis: An introduction to its methodology. Second Edition. Thousand Oaks, CA: Sage.
- Maxwell, A. E. (1970). Comparing the classification of subjects by two independent judges. British Journal of Psychiatry, 116, 651-655.
- Osgood, C. E. (1959). The representational model and relevant research. In I. de Sola Pool (Ed.), Trends in content analysis (pp. 33-88). Urbana: University of Illinois Press.
- Potter, W. J., & Levine-Donnerstein, D. (1999). Rethinking validity and reliability in content analysis. Journal of Applied Communication Research, 27, 258-284.
- Riffe, D., & Freitag, A. (1996, August). Twenty-five years of content analyses in Journalism & Mass Communication Quarterly. Paper presented to the annual convention of the Association for Education in Journalism and Mass Communication, Anaheim, CA, cited in D. Riffe, S. Lacy, & F. G. Fico (1998). Analyzing media messages. Mahwah, NJ: Erlbaum.
- Scott, W. A. (1955). Reliability of content analysis: The case of nominal scale coding. Public Opinion Quarterly, 19, 321-325.
- Zwick, R. (1988). Another look at interrater agreement. Psychological Bulletin, 103, 347-387.
Cited by
Sorted by citations
Answering the Call for a Standard Reliability Measure for Coding Data
Hayes, A. F., & Krippendorff, K. (2007) · Communication Methods and Measures · doi:10.1080/19312450709336664 · Cited by 4,225
Self-citationRadianti, J., Majchrzak, T. A., Fromm, J., & Wohlgenannt, I. (2020) · Computers & Education · doi:10.1016/j.compedu.2019.103778 · Cited by 3,585
Qualitative content analysis: theoretical foundation, basic procedures and software solution
Mayring, P. (2014) · Social Science Open Access Repository (GESIS – Leibniz Institute for the Social Sciences) · Cited by 2,173
Inter-Coder Agreement for Computational Linguistics
Artstein, R., & Poesio, M. (2008) · Computational Linguistics · doi:10.1162/coli.07-034-r2 · Cited by 1,665
DeCuir-Gunby, J. T., Marshall, P. L., & McCulloch, A. W. (2011) · Field Methods · doi:10.1177/1525822x10388468 · Cited by 1,456
CSR reporting practices and the quality of disclosure: An empirical analysis
Michelon, G., Pilonato, S., & Ricceri, F. (2014) · Critical Perspectives on Accounting · doi:10.1016/j.cpa.2014.10.003 · Cited by 1,506
The Power of Feedback Revisited: A Meta-Analysis of Educational Feedback Research
Wisniewski, B., Zierer, K., & Hattie, J. (2020) · Frontiers in Psychology · doi:10.3389/fpsyg.2019.03087 · Cited by 1,199
“For the 6 moderator variables, Krippendorff’s alpha ranged from 0.81 to 0.99, and therefore above the acceptable level (Krippendorff, 2004).”
InfluentialDeveloping a unified definition of digital transformation
Gong, C., & Ribière, V. (2020) · Technovation · doi:10.1016/j.technovation.2020.102217 · Cited by 1,117
Social Media Use for Health Purposes: Systematic Review
Chen, J., & Wang, Y. (2021) · Journal of Medical Internet Research · doi:10.2196/17917 · Cited by 1,073
Kleinheksel, A. J., Rockich-Winston, N., Tawfik, H., & Wyatt, T. R. (2020) · American Journal of Pharmaceutical Education · doi:10.5688/ajpe7113 · Cited by 1,054
CREMA-D: Crowd-Sourced Emotional Multimodal Actors Dataset
Cao, H., Cooper, D. G., Keutmann, M. K., Gur, R. C., Nenkova, A., & Verma, R. (2014) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2014.2336244 · Cited by 954
Conducting systematic literature review in operations management
Thomé, A. M. T., Scavarda, L. F., & Scavarda, A. (2016) · Production Planning & Control · doi:10.1080/09537287.2015.1129464 · Cited by 904
Computing Krippendorff's Alpha-Reliability
Krippendorff, K. (2011) · ScholarlyCommons (University of Pennsylvania) · Cited by 809
Self-citationThe “I Designed It Myself” Effect in Mass Customization
Franke, N., Schreier, M., & Kaiser, U. (2009) · Management Science · doi:10.1287/mnsc.1090.1077 · Cited by 788
Big data analytics in E-commerce: a systematic review and agenda for future research
Akter, S., & Wamba, S. F. (2016) · Electronic Markets · doi:10.1007/s12525-016-0219-0 · Cited by 684
Ex Machina: Personal Attacks Seen at Scale
Wulczyn, E., Thain, N., & Dixon, L. (2017) · Proceedings of the 26th International Conference on World Wide Web · doi:10.1145/3038912.3052591 · Cited by 636
A literature review and perspectives in reverse logistics
Agrawal, S., Singh, R. K., & Murtaza, Q. (2015) · Resources Conservation and Recycling · doi:10.1016/j.resconrec.2015.02.009 · Cited by 624
A Systematic Review of Propensity Score Methods in the Social Sciences
Thoemmes, F. J., & Kim, E. S. (2011) · Multivariate Behavioral Research · doi:10.1080/00273171.2011.540475 · Cited by 577
Jacobson, N. C., & Newman, M. G. (2017) · Psychological Bulletin · doi:10.1037/bul0000111 · Cited by 576
Amendola, L. M., Jarvik, G. P., Leo, M. C., McLaughlin, H. M., Akkari, Y. M. N., Amaral, M. D., Berg, J. S., Biswas, S., Bowling, K. M., Conlin, L. K., et al. (2016) · The American Journal of Human Genetics · doi:10.1016/j.ajhg.2016.03.024 · Cited by 572
What is frugal innovation? Three defining criteria
Weyrauch, T., & Herstatt, C. (2016) · Journal of Frugal Innovation · doi:10.1186/s40669-016-0005-y · Cited by 526
Nguyen, T. H. H., Elmagrhi, M., Ntim, C. G., & Wu, Y. (2021) · Business Strategy and the Environment · doi:10.1002/bse.2748 · Cited by 516
Reliability from α to ω: A tutorial.
Revelle, W., & Condon, D. M. (2019) · Psychological Assessment · doi:10.1037/pas0000754 · Cited by 569
The application of AI technologies in STEM education: a systematic review from 2011 to 2021
Xu, W., & Ouyang, F. (2022) · International Journal of STEM Education · doi:10.1186/s40594-022-00377-5 · Cited by 495
A team-based approach to open coding: considerations for creating intercoder consensus
Cascio, M. A., Lee, E., Vaudrin, N., & Freedman, D. A. (2019) · Field Methods · doi:10.1177/1525822x19838237 · Cited by 492
Corporate Social Responsibility in China: A Corporate Governance Approach
Lau, C., Lü, Y., & Liang, Q. (2014) · Journal of Business Ethics · doi:10.1007/s10551-014-2513-0 · Cited by 486
Issues and Best Practices in Content Analysis
Lacy, S. R., Watson, B. R., Riffe, D., & Lovejoy, J. (2015) · Journalism & Mass Communication Quarterly · doi:10.1177/1077699015607338 · Cited by 530
Sutherland, L., Howard, S., & Markauskaite, L. (2010) · Teaching and Teacher Education · doi:10.1016/j.tate.2009.06.006 · Cited by 443
Web Versus Other Survey Modes: An Updated and Extended Meta-Analysis Comparing Response Rates
Daikeler, J., Bošnjak, M., & Manfreda, K. L. (2019) · Journal of Survey Statistics and Methodology · doi:10.1093/jssam/smz008 · Cited by 422
Berke, P. R., & Godschalk, D. R. (2008) · Journal of Planning Literature · doi:10.1177/0885412208327014 · Cited by 407
Translating the ICAP Theory of Cognitive Engagement Into Practice
Michelene, T. H., Adams, J., Bogusch, E. B., Bruchok, C., Kang, S., Lancaster, M. E., Levy, R., Li, N., McEldoon, K. L., Stump, G., et al. (2018) · Cognitive Science · doi:10.1111/cogs.12626 · Cited by 433
I'm Going to Instagram It! An Analysis of Athlete Self-Presentation on Instagram
Smith, L. R., & Sanderson, J. (2015) · Journal of Broadcasting & Electronic Media · doi:10.1080/08838151.2015.1029125 · Cited by 433
Measuring the Reliability of Hate Speech Annotations: The Case of the European Refugee CrisisPreprint
Roß, B., Rist, M., Carbonell, G., Cabrera, B., Kurowsky, N., & Wojatzki, M. (2016) · arXiv (Cornell University) · doi:10.17185/duepublico/42132 · Cited by 359
A Systematic Literature Review on Recent Lean Research: State‐of‐the‐art and Future Directions
Danese, P., Manfè, V., & Romano, P. (2017) · International Journal of Management Reviews · doi:10.1111/ijmr.12156 · Cited by 356
Rosé, C., Wang, Y.-C., Cui, Y., Arguello, J., Stegmann, K., Weinberger, A., & Fischer, F. (2008) · International Journal of Computer-Supported Collaborative Learning · doi:10.1007/s11412-007-9034-0 · Cited by 351
Snapping, pinning, liking or texting: Investigating social media in higher education beyond Facebook
Manca, S. (2020) · The Internet and Higher Education · doi:10.1016/j.iheduc.2019.100707 · Cited by 347
Board diversity and CSR reporting: an Australian study
Rao, K. K., & Tilt, C. (2016) · Meditari Accountancy Research · doi:10.1108/medar-08-2015-0052 · Cited by 346
Hayter, C. S. (2016) · Small Business Economics · doi:10.1007/s11187-016-9756-3 · Cited by 328
Why Cohen’s Kappa should be avoided as performance measure in classification
Delgado, R., & Tibau, X.-A. (2019) · PLOS ONE · doi:10.1371/journal.pone.0222916 · Cited by 324
Martinez, D. L., & Brusoni, S. (2018) · Strategic Management Journal · doi:10.1002/smj.2774 · Cited by 333
Kim, N. J., & Kim, M. K. (2022) · Frontiers in Education · doi:10.3389/feduc.2022.755914 · Cited by 298
Naeem, M., Ozuem, W., Howell, K., & Ranfagni, S. (2024) · International Journal of Qualitative Methods · doi:10.1177/16094069241229777 · Cited by 294
Inter-Rater Reliability Methods in Qualitative Case Study Research
Cole, R. (2023) · Sociological Methods & Research · doi:10.1177/00491241231156971 · Cited by 294
Lumineau, F., & Henderson, J. E. (2012) · Journal of Operations Management · doi:10.1016/j.jom.2012.03.005 · Cited by 293
Understanding the Third-Person Perception: Evidence From a Meta-Analysis
Sun, Y., Pan, Z., & Shen, L. (2008) · Journal of Communication · doi:10.1111/j.1460-2466.2008.00385.x · Cited by 284
Why do servitized firms fail? A risk-based explanation
Benedettini, O., Neely, A. D., & Swink, M. L. (2015) · International Journal of Operations & Production Management · doi:10.1108/ijopm-02-2014-0052 · Cited by 293
Motivations of manufacturing reshoring: an interpretative framework
Fratocchi, L., Ancarani, A., Barbieri, P., Mauro, C. D., Nassimbeni, G., Sartor, M., Vignoli, M., & Zanoni, A. B. (2016) · International Journal of Physical Distribution & Logistics Management · doi:10.1108/ijpdlm-06-2014-0131 · Cited by 266
The effects of educational robotics in STEM education: a multilevel meta-analysis
Ouyang, F., & Xu, W. (2024) · International Journal of STEM Education · doi:10.1186/s40594-024-00469-4 · Cited by 300
Ex Machina: Personal Attacks Seen at ScalePreprint
Wulczyn, E., Thain, N., & Dixon, L. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1610.08914 · Cited by 260
Izah, S. C., Sylva, L., & Hait, M. (2023) · ES Energy & Environments · doi:10.30919/esee1057 · Cited by 310
Effects of Sesame Street: A meta-analysis of children’s learning in 15 countries
Mares, M.-L., & Pan, Z. (2013) · Journal of Applied Developmental Psychology · doi:10.1016/j.appdev.2013.01.001 · Cited by 245
Mining Social Media Data for Understanding Students’ Learning Experiences
Chen, X., Vorvoreanu, M., & Madhavan, K. P. C. (2014) · IEEE Transactions on Learning Technologies · doi:10.1109/tlt.2013.2296520 · Cited by 242
Adnan, N., Nordin, S. M., bin Bahruddin, M. A., & Tareq, A. H. (2019) · Trends in Food Science & Technology · doi:10.1016/j.tifs.2019.02.040 · Cited by 238
Embodiment in Socially Interactive Robots
Deng, E., Mutlu, B., & Matarić, M. J. (2019) · Foundations and Trends in Robotics · doi:10.1561/2300000056 · Cited by 261
Content analysis of e-petitions with topic modeling: How to train and evaluate LDA models?
Hagen, L. (2018) · Information Processing & Management · doi:10.1016/j.ipm.2018.05.006 · Cited by 233
Vaio, A. D., Hassan, R., & Alavoine, C. (2021) · Technological Forecasting and Social Change · doi:10.1016/j.techfore.2021.121201 · Cited by 255
Big Social Data Analytics in Journalism and Mass Communication
Гуо, Л., Vargo, C. J., Pan, Z., Ding, W., & Ishwar, P. (2016) · Journalism & Mass Communication Quarterly · doi:10.1177/1077699016639231 · Cited by 250
The Social News Gap: How News Reading and News Sharing Diverge
Bright, J. (2016) · Journal of Communication · doi:10.1111/jcom.12232 · Cited by 228
Sciarelli, M., Cosimato, S., Landi, G. C., & Iandolo, F. (2021) · The TQM Journal · doi:10.1108/tqm-08-2020-0180 · Cited by 227
What to Do (and Not to Do) with the Comparative Manifestos Project Data
Gemenis, K. (2013) · Political Studies · doi:10.1111/1467-9248.12015 · Cited by 226
Cultural Brokerage and Creative Performance in Multicultural Teams
Jang, S. (2017) · Organization Science · doi:10.1287/orsc.2017.1162 · Cited by 231
Persuasion for Good: Towards a Personalized Persuasive Dialogue System for Social Good
Wang, X., Shi, W., Kim, R., Oh, Y., Yang, S., Zhang, J., & Yu, Z. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1566 · Cited by 222
Barbour, J. B., Rintamaki, L. S., Ramsey, J. A., & Brashers, D. E. (2011) · Journal of Health Communication · doi:10.1080/10810730.2011.585691 · Cited by 222
Performance Pressure as a Double-edged Sword
Gardner, H. K. (2012) · Administrative Science Quarterly · doi:10.1177/0001839212446454 · Cited by 230
It’s not all about autism: The emerging landscape of anti-vaccination sentiment on Facebook
Hoffman, B. L., Felter, E. M., Chu, K., Shensa, A. S., Hermann, C., Wolynn, T. H., Williams, D., & Primack, B. A. (2019) · Vaccine · doi:10.1016/j.vaccine.2019.03.003 · Cited by 222
Lazar, J. K., Feng, J. H., & Hochheiser, H. (2017) · Elsevier eBooks · doi:10.1016/b978-0-12-805390-4.00011-x · Cited by 221
van de Pol, J., Volman, M., Oort, F., & Beishuizen, J. (2015) · Instructional Science · doi:10.1007/s11251-015-9351-z · Cited by 220
Giudice, M. D., Vaio, A. D., Hassan, R., & Palladino, R. (2021) · Maritime Policy & Management · doi:10.1080/03088839.2021.1903600 · Cited by 215
Quantitative indicators for social sustainability assessment of supply chains
Popović, T., Barbosa‐Póvoa, A. P., Krasławski, A., & Carvalho, A. I. C. D. S. G. (2018) · Journal of Cleaner Production · doi:10.1016/j.jclepro.2018.01.142 · Cited by 214
Expert, Crowdsourced, and Machine Assessment of Suicide Risk via Online Postings
Shing, H.-C., Nair, S., Zirikly, A., Friedenberg, M., Daumé, H., & Resnik, P. (2018) · Workshop on Computational Linguistics and Clinical Psychology (CLPsych) · doi:10.18653/v1/w18-0603 · Cited by 233
Goal-oriented requirements engineering: an extended systematic mapping study
Horkoff, J., Aydemir, F. B., Cardoso, E. C. S., Li, T., Maté, A., Paja, E., Salnitri, M., Piras, L., Mylopoulos, J., & Giorgini, P. (2017) · Requirements Engineering · doi:10.1007/s00766-017-0280-z · Cited by 212
Maintaining the Double Standard: Portrayals of Age and Gender in Popular Films
Lauzen, M. M., & Dozier, D. M. (2005) · Sex Roles · doi:10.1007/s11199-005-3710-1 · Cited by 220
Artstein, R. (2017) · Springer eBooks · doi:10.1007/978-94-024-0881-2_11 · Cited by 229
The Changing Misrepresentation of Race and Crime on Network and Cable News
Dixon, T. L., & Williams, C. L. (2014) · Journal of Communication · doi:10.1111/jcom.12133 · Cited by 202
Vaio, A. D., Hasan, S., Palladino, R., & Hassan, R. (2022) · Environment Development and Sustainability · doi:10.1007/s10668-021-02078-5 · Cited by 200
NoRBERT: Transfer Learning for Requirements Classification
Hey, T., Keim, J., Koziolek, A., & Tichy, W. F. (2020) · IEEE International Requirements Engineering Conference (RE) · doi:10.1109/re48521.2020.00028 · Cited by 216
Hayter, C. S. (2015) · Research Policy · doi:10.1016/j.respol.2015.11.003 · Cited by 198
Measuring and Reporting Intercoder Reliability in Plan Quality Evaluation Research
Stevens, M. R., Lyles, W., & Berke, P. R. (2014) · Journal of Planning Education and Research · doi:10.1177/0739456x13513614 · Cited by 197
Ziegele, M., Breiner, T., & Quiring, O. (2014) · Journal of Communication · doi:10.1111/jcom.12123 · Cited by 224
Akter, S., Bandara, R., Hani, U., Fosso Wamba, S., Foropon, C., & Papadopoulos, T. (2019) · International Journal of Information Management · doi:10.1016/j.ijinfomgt.2019.01.020 · Cited by 193
‘Let’s Get Them Involved’ . . . to Some Extent: Analyzing Online News Participation
Almgren, S., & Olsson, T. (2015) · Social Media + Society · doi:10.1177/2056305115621934 · Cited by 189
Antecedents to Psychological Reactance: The Impact of Threat, Message Frame, and Choice
Shen, L. (2014) · Health Communication · doi:10.1080/10410236.2014.910882 · Cited by 187
Constructing Gender Stereotypes Through Social Roles in Prime-Time Television
Lauzen, M. M., Dozier, D. M., & Horan, N. M. (2008) · Journal of Broadcasting & Electronic Media · doi:10.1080/08838150801991971 · Cited by 186
Assessing the Reliability and Validity of Expert Interviews
Dorussen, H., Lenz, H., & Blavoukos, S. (2005) · European Union Politics · doi:10.1177/1465116505054835 · Cited by 215
Scott, S. R., Rivera, K. M., Rushing, E., Manczak, E. M., Rozek, C. S., & Doom, J. R. (2021) · Journal of Adolescent Health · doi:10.1016/j.jadohealth.2020.11.010 · Cited by 185
Schiavone, F., Mancini, D., Leone, D., & Lavorato, D. (2021) · Technological Forecasting and Social Change · doi:10.1016/j.techfore.2021.120647 · Cited by 182
Mion, G., & Loza Adaui, C. R. (2019) · Sustainability · doi:10.3390/su11174612 · Cited by 180
Svenningsson, J., Höst, G., Hultén, M., & Hallström, J. (2022) · International Journal of Technology and Design Education · doi:10.1007/s10798-021-09657-7 · Cited by 200
Fankhauser, C., & Christie, J. M. (2015) · Current Biology · doi:10.1016/j.cub.2015.03.020 · Cited by 167
Implementing Data-Driven Smart City Applications for Future Cities
Kaluarachchi, Y. (2022) · Smart Cities · doi:10.3390/smartcities5020025 · Cited by 169
An empirical examination of online restaurant reviews on Yelp.com
Yang, S., Hlee, S., Lee, J., & Koo, C. (2017) · International Journal of Contemporary Hospitality Management · doi:10.1108/ijchm-11-2015-0643 · Cited by 167
Challenge‐based learning implementation in engineering education: A systematic literature review
Doulougeri, K., Vermunt, J. D. H. M., Bombaerts, G. J. T., & Bots, M. (2024) · Journal of Engineering Education · doi:10.1002/jee.20588 · Cited by 167
Making Sense of Intractable Multiparty Conflict: A Study of Framing in Four Environmental Disputes
Brummans, B. H. J. M., Putnam, L. L., Gray, B., Hanke, R., Lewicki, R. J., & Wiethoff, C. (2008) · Communication Monographs · doi:10.1080/03637750801952735 · Cited by 177
Mistry, B., de Ramirez, S. S., Kelen, G. D., Schmitz, P. S. K., Balhara, K. S., Levin, S., Martínez, D. A., Psoter, K. J., Anton, X. P., & Hinson, J. S. (2017) · Annals of Emergency Medicine · doi:10.1016/j.annemergmed.2017.09.036 · Cited by 160
Kafyulilo, A., Fisser, P., Pieters, J. M., & Voogt, J. M. (2015) · Australasian Journal of Educational Technology · doi:10.14742/ajet.1240 · Cited by 158
Bertuol‐Garcia, D., Morsello, C., El-Hani, C. N., & Pardini, R. (2017) · Biological reviews/Biological reviews of the Cambridge Philosophical Society · doi:10.1111/brv.12385 · Cited by 169
Predicting Opinion Leaders in Twitter Activism Networks
Xu, W. W., Sang, Y., Blasiola, S., & Park, H. W. (2014) · American Behavioral Scientist · doi:10.1177/0002764214527091 · Cited by 169
Herding and Social Media Word-of-Mouth: Evidence from Groupon1
Li, X., & Wu, L. (2013) · MIS Quarterly · doi:10.25300/misq/2018/14108 · Cited by 155
Hybrid Activism: Social Movement Mobilization in a Multimovement Environment
Heaney, M. T., & Rojas, F. G. (2014) · American Journal of Sociology · doi:10.1086/674897 · Cited by 154
EpiTools, A software suite for presurgical brain mapping in epilepsy: Intracerebral EEG
Villalon, S. M., Paz, R. M., Roehri, N., Lagarde, S., Pizzo, F., Colombet, B., Bartoloméi, F., Carron, R., & Bénar, C. (2018) · Journal of Neuroscience Methods · doi:10.1016/j.jneumeth.2018.03.018 · Cited by 154
Metadata
- Authors
- Krippendorff, K.
- Corresponding author
- Krippendorff, K.
- ORCID
- Krippendorff, K.
- Venue
- Human Communication Research
- Publisher
- Oxford University Press (OUP)
- Published
- 2004-07-01
- Volume
- 30
- Issue
- 3
- Pages
- 411-433
- ISSN
- 0360-3989 (ISSN-L)
- 1468-2958
- OpenAlex
- W4243989635
- Cited by
- 1,027
- References
- 0
- Citations by year
- 2026: 42 · 2025: 100 · 2024: 108 · 2023: 108 · 2022: 124 · 2021: 135 · 2020: 105 · 2019: 57 · 2018: 44 · 2017: 29 · 2016: 41 · 2015: 25 · 2014: 26 · 2013: 21 · 2012: 15
- FWCI
- 5.93
- Type
- journal article
- Language
- en
- Open access
- green
- repository.upenn.edu
- Locations
- Human Communication Research (publishedVersion)
- ScholarlyCommons (University of Pennsylvania) (submittedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Reliability and Agreement in Measurement
- Social Sciences › General Social Sciences › Computational and Text Analysis Methods
- Social Sciences › Decision Sciences › Management Science and Operations Research › Psychometric Methodologies and Testing
- Keywords
- inter-rater reliability, reliability coefficients, interobserver agreement, reporting reliability
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
