Reliability in content analysis: some common misconceptions and recommendations.
Open accessAbstract
In a recent article published in this journal, Lombard, Snyder-Duch, and Bracken (2002) surveyed 200 content analyses for their reporting of reliability tests; compared the virtues and drawbacks of five popular reliability measures; and proposed guidelines and standards for their use. Their discussion revealed that numerous misconceptions circulate in the content analysis literature regarding how these measures behave and can aid or deceive content analysts in their effort to ensure the reliability of their data. This paper proposes three conditions for statistical measures to serve as indices of the reliability of data and examines the mathematical structure and the behavior of the five coefficients discussed by the authors, plus two others. It compares common beliefs about these coefficients with what they actually do and concludes with alternative recommendations for testing reliability in content analysis and similar data-making efforts.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Krippendorff's α considers chance independently from coders like Scott's π, but data reliability is estimated depending on disagreement instead of agreement.
“Krippendorff's α α α α (Krippendorff, 2004) considers chance independently from coders like Scott's π, but data reliability is estimated depending on disagreement instead of agreement:”
From Weighted Krippendorff's alpha is a more reliable metrics for multi-coders ordinal annotations: experimental studies on emotion, opinion and coreference annotation (2014) · cites this paper as Krippendorff, 2004
Intercoder reliability for the 6 moderator variables ranged from 0.81 to 0.99, above the acceptable level.
“For the 6 moderator variables, Krippendorff’s alpha ranged from 0.81 to 0.99, and therefore above the acceptable level (Krippendorff, 2004).”
From The Power of Feedback Revisited: A Meta-Analysis of Educational Feedback Research (2020) · cites this paper as Krippendorff, 2004
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
The publisher hasn’t deposited this paper’s reference list.
Research works printed but not linked: 13
- Bennett, E. M., Alpert, R., & Goldstein, A. C. (1954). Communications through limited response questioning. Public Opinion Quarterly, 18, 303-308.
- Craig, R. T. (1981). Generalization of Scott"s index of intercoder agreement. Public Opinion Quarterly, 45, 260-264.
- Goodman, L. A, & Kruskal, W. H. (1954). Measures of association for cross classifications. Journal of the American Statistical Association, 49, 732-764.
- Holsti, O. R. (1969). Content analysis for the social sciences and humanities. Reading, MA: Addison- Wesley.
- Hughes, M. A., & Garrett, D. E. (1990). Intercoder reliability estimation -Approaches in marketing: A generalizability theory framework for quantitative data. Journal of Marketing Research, 27, 185- 195.
- Krippendorff, K. (1970). Bivariate agreement coefficients for reliability data. In E. R. Borgatta & G. W. Bohrnstedt (Eds.), Sociological methodology 1970 (pp. 139-150). San Francisco, CA: Jossey Bass.
- Krippendorff, K. (2004). Content analysis: An introduction to its methodology. Second Edition. Thousand Oaks, CA: Sage.
- Maxwell, A. E. (1970). Comparing the classification of subjects by two independent judges. British Journal of Psychiatry, 116, 651-655.
- Osgood, C. E. (1959). The representational model and relevant research. In I. de Sola Pool (Ed.), Trends in content analysis (pp. 33-88). Urbana: University of Illinois Press.
- Potter, W. J., & Levine-Donnerstein, D. (1999). Rethinking validity and reliability in content analysis. Journal of Applied Communication Research, 27, 258-284.
- Riffe, D., & Freitag, A. (1996, August). Twenty-five years of content analyses in Journalism & Mass Communication Quarterly. Paper presented to the annual convention of the Association for Education in Journalism and Mass Communication, Anaheim, CA, cited in D. Riffe, S. Lacy, & F. G. Fico (1998). Analyzing media messages. Mahwah, NJ: Erlbaum.
- Scott, W. A. (1955). Reliability of content analysis: The case of nominal scale coding. Public Opinion Quarterly, 19, 321-325.
- Zwick, R. (1988). Another look at interrater agreement. Psychological Bulletin, 103, 347-387.
Cited by
Sorted by citations
Answering the Call for a Standard Reliability Measure for Coding Data
Hayes, A. F., & Krippendorff, K. (2007) · Communication Methods and Measures · doi:10.1080/19312450709336664 · Cited by 4,225 · see citing works
Self-citationRadianti, J., Majchrzak, T. A., Fromm, J., & Wohlgenannt, I. (2020) · Computers & Education · doi:10.1016/j.compedu.2019.103778 · Cited by 3,585 · see citing works
Qualitative content analysis: theoretical foundation, basic procedures and software solution
Mayring, P. (2014) · Social Science Open Access Repository (GESIS – Leibniz Institute for the Social Sciences) · Cited by 2,173 · see citing works
Inter-Coder Agreement for Computational Linguistics
Artstein, R., & Poesio, M. (2008) · Computational Linguistics · doi:10.1162/coli.07-034-r2 · Cited by 1,665 · see citing works
DeCuir-Gunby, J. T., Marshall, P. L., & McCulloch, A. W. (2011) · Field Methods · doi:10.1177/1525822x10388468 · Cited by 1,456 · see citing works
CSR reporting practices and the quality of disclosure: An empirical analysis
Michelon, G., Pilonato, S., & Ricceri, F. (2014) · Critical Perspectives on Accounting · doi:10.1016/j.cpa.2014.10.003 · Cited by 1,346 · see citing works
The Power of Feedback Revisited: A Meta-Analysis of Educational Feedback Research
Wisniewski, B., Zierer, K., & Hattie, J. (2020) · Frontiers in Psychology · doi:10.3389/fpsyg.2019.03087 · Cited by 1,199 · see citing works
“For the 6 moderator variables, Krippendorff’s alpha ranged from 0.81 to 0.99, and therefore above the acceptable level (Krippendorff, 2004).”
InfluentialDeveloping a unified definition of digital transformation
Gong, C., & Ribière, V. (2020) · Technovation · doi:10.1016/j.technovation.2020.102217 · Cited by 1,117 · see citing works
Social Media Use for Health Purposes: Systematic Review
Chen, J., & Wang, Y. (2021) · Journal of Medical Internet Research · doi:10.2196/17917 · Cited by 1,073 · see citing works
Kleinheksel, A. J., Rockich-Winston, N., Tawfik, H., & Wyatt, T. R. (2020) · American Journal of Pharmaceutical Education · doi:10.5688/ajpe7113 · Cited by 1,054 · see citing works
CREMA-D: Crowd-Sourced Emotional Multimodal Actors Dataset
Cao, H., Cooper, D. G., Keutmann, M. K., Gur, R. C., Nenkova, A., & Verma, R. (2014) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2014.2336244 · Cited by 949 · see citing works
Conducting systematic literature review in operations management
Thomé, A. M. T., Scavarda, L. F., & Scavarda, A. (2016) · Production Planning & Control · doi:10.1080/09537287.2015.1129464 · Cited by 858 · see citing works
Computing Krippendorff's Alpha-Reliability
Krippendorff, K. (2011) · ScholarlyCommons (University of Pennsylvania) · Cited by 809 · see citing works
Self-citationThe “I Designed It Myself” Effect in Mass Customization
Franke, N., Schreier, M., & Kaiser, U. (2009) · Management Science · doi:10.1287/mnsc.1090.1077 · Cited by 761 · see citing works
Big data analytics in E-commerce: a systematic review and agenda for future research
Akter, S., & Wamba, S. F. (2016) · Electronic Markets · doi:10.1007/s12525-016-0219-0 · Cited by 684 · see citing works
Ex Machina: Personal Attacks Seen at Scale
Wulczyn, E., Thain, N., & Dixon, L. (2017) · Proceedings of the 26th International Conference on World Wide Web · doi:10.1145/3038912.3052591 · Cited by 636 · see citing works
A literature review and perspectives in reverse logistics
Agrawal, S., Singh, R. K., & Murtaza, Q. (2015) · Resources Conservation and Recycling · doi:10.1016/j.resconrec.2015.02.009 · Cited by 624 · see citing works
A Systematic Review of Propensity Score Methods in the Social Sciences
Thoemmes, F. J., & Kim, E. S. (2011) · Multivariate Behavioral Research · doi:10.1080/00273171.2011.540475 · Cited by 577 · see citing works
Jacobson, N. C., & Newman, M. G. (2017) · Psychological Bulletin · doi:10.1037/bul0000111 · Cited by 576 · see citing works
Amendola, L. M., Jarvik, G. P., Leo, M. C., McLaughlin, H. M., Akkari, Y. M. N., Amaral, M. D., Berg, J. S., Biswas, S., Bowling, K. M., Conlin, L. K., et al. (2016) · The American Journal of Human Genetics · doi:10.1016/j.ajhg.2016.03.024 · Cited by 556 · see citing works
What is frugal innovation? Three defining criteria
Weyrauch, T., & Herstatt, C. (2016) · Journal of Frugal Innovation · doi:10.1186/s40669-016-0005-y · Cited by 526 · see citing works
Nguyen, T. H. H., Elmagrhi, M., Ntim, C. G., & Wu, Y. (2021) · Business Strategy and the Environment · doi:10.1002/bse.2748 · Cited by 516 · see citing works
Reliability from α to ω: A tutorial.
Revelle, W., & Condon, D. M. (2019) · Psychological Assessment · doi:10.1037/pas0000754 · Cited by 514 · see citing works
The application of AI technologies in STEM education: a systematic review from 2011 to 2021
Xu, W., & Ouyang, F. (2022) · International Journal of STEM Education · doi:10.1186/s40594-022-00377-5 · Cited by 495 · see citing works
A team-based approach to open coding: considerations for creating intercoder consensus
Cascio, M. A., Lee, E., Vaudrin, N., & Freedman, D. A. (2019) · Field Methods · doi:10.1177/1525822x19838237 · Cited by 492 · see citing works
Corporate Social Responsibility in China: A Corporate Governance Approach
Lau, C., Lü, Y., & Liang, Q. (2014) · Journal of Business Ethics · doi:10.1007/s10551-014-2513-0 · Cited by 486 · see citing works
Issues and Best Practices in Content Analysis
Lacy, S. R., Watson, B. R., Riffe, D., & Lovejoy, J. (2015) · Journalism & Mass Communication Quarterly · doi:10.1177/1077699015607338 · Cited by 478 · see citing works
Sutherland, L., Howard, S., & Markauskaite, L. (2010) · Teaching and Teacher Education · doi:10.1016/j.tate.2009.06.006 · Cited by 443 · see citing works
Web Versus Other Survey Modes: An Updated and Extended Meta-Analysis Comparing Response Rates
Daikeler, J., Bošnjak, M., & Manfreda, K. L. (2019) · Journal of Survey Statistics and Methodology · doi:10.1093/jssam/smz008 · Cited by 422 · see citing works
Berke, P. R., & Godschalk, D. R. (2008) · Journal of Planning Literature · doi:10.1177/0885412208327014 · Cited by 407 · see citing works
Translating the ICAP Theory of Cognitive Engagement Into Practice
Michelene, T. H., Adams, J., Bogusch, E. B., Bruchok, C., Kang, S., Lancaster, M. E., Levy, R., Li, N., McEldoon, K. L., Stump, G., et al. (2018) · Cognitive Science · doi:10.1111/cogs.12626 · Cited by 397 · see citing works
I'm Going to Instagram It! An Analysis of Athlete Self-Presentation on Instagram
Smith, L. R., & Sanderson, J. (2015) · Journal of Broadcasting & Electronic Media · doi:10.1080/08838151.2015.1029125 · Cited by 377 · see citing works
Measuring the Reliability of Hate Speech Annotations: The Case of the European Refugee CrisisPreprint
Roß, B., Rist, M., Carbonell, G., Cabrera, B., Kurowsky, N., & Wojatzki, M. (2016) · arXiv (Cornell University) · doi:10.17185/duepublico/42132 · Cited by 359 · see citing works
A Systematic Literature Review on Recent Lean Research: State‐of‐the‐art and Future Directions
Danese, P., Manfè, V., & Romano, P. (2017) · International Journal of Management Reviews · doi:10.1111/ijmr.12156 · Cited by 356 · see citing works
Rosé, C., Wang, Y.-C., Cui, Y., Arguello, J., Stegmann, K., Weinberger, A., & Fischer, F. (2008) · International Journal of Computer-Supported Collaborative Learning · doi:10.1007/s11412-007-9034-0 · Cited by 351 · see citing works
Snapping, pinning, liking or texting: Investigating social media in higher education beyond Facebook
Manca, S. (2020) · The Internet and Higher Education · doi:10.1016/j.iheduc.2019.100707 · Cited by 347 · see citing works
Board diversity and CSR reporting: an Australian study
Rao, K. K., & Tilt, C. (2016) · Meditari Accountancy Research · doi:10.1108/medar-08-2015-0052 · Cited by 346 · see citing works
Hayter, C. S. (2016) · Small Business Economics · doi:10.1007/s11187-016-9756-3 · Cited by 328 · see citing works
Why Cohen’s Kappa should be avoided as performance measure in classification
Delgado, R., & Tibau, X.-A. (2019) · PLOS ONE · doi:10.1371/journal.pone.0222916 · Cited by 324 · see citing works
Martinez, D. L., & Brusoni, S. (2018) · Strategic Management Journal · doi:10.1002/smj.2774 · Cited by 309 · see citing works
Kim, N. J., & Kim, M. K. (2022) · Frontiers in Education · doi:10.3389/feduc.2022.755914 · Cited by 298 · see citing works
Naeem, M., Ozuem, W., Howell, K., & Ranfagni, S. (2024) · International Journal of Qualitative Methods · doi:10.1177/16094069241229777 · Cited by 294 · see citing works
Inter-Rater Reliability Methods in Qualitative Case Study Research
Cole, R. (2023) · Sociological Methods & Research · doi:10.1177/00491241231156971 · Cited by 294 · see citing works
Lumineau, F., & Henderson, J. E. (2012) · Journal of Operations Management · doi:10.1016/j.jom.2012.03.005 · Cited by 285 · see citing works
Understanding the Third-Person Perception: Evidence From a Meta-Analysis
Sun, Y., Pan, Z., & Shen, L. (2008) · Journal of Communication · doi:10.1111/j.1460-2466.2008.00385.x · Cited by 277 · see citing works
Why do servitized firms fail? A risk-based explanation
Benedettini, O., Neely, A. D., & Swink, M. L. (2015) · International Journal of Operations & Production Management · doi:10.1108/ijopm-02-2014-0052 · Cited by 274 · see citing works
Motivations of manufacturing reshoring: an interpretative framework
Fratocchi, L., Ancarani, A., Barbieri, P., Mauro, C. D., Nassimbeni, G., Sartor, M., Vignoli, M., & Zanoni, A. B. (2016) · International Journal of Physical Distribution & Logistics Management · doi:10.1108/ijpdlm-06-2014-0131 · Cited by 266 · see citing works
The effects of educational robotics in STEM education: a multilevel meta-analysis
Ouyang, F., & Xu, W. (2024) · International Journal of STEM Education · doi:10.1186/s40594-024-00469-4 · Cited by 263 · see citing works
Ex Machina: Personal Attacks Seen at ScalePreprint
Wulczyn, E., Thain, N., & Dixon, L. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1610.08914 · Cited by 260 · see citing works
Izah, S. C., Sylva, L., & Hait, M. (2023) · ES Energy & Environments · doi:10.30919/esee1057 · Cited by 255 · see citing works
Effects of Sesame Street: A meta-analysis of children’s learning in 15 countries
Mares, M.-L., & Pan, Z. (2013) · Journal of Applied Developmental Psychology · doi:10.1016/j.appdev.2013.01.001 · Cited by 245 · see citing works
Mining Social Media Data for Understanding Students’ Learning Experiences
Chen, X., Vorvoreanu, M., & Madhavan, K. P. C. (2014) · IEEE Transactions on Learning Technologies · doi:10.1109/tlt.2013.2296520 · Cited by 242 · see citing works
Adnan, N., Nordin, S. M., bin Bahruddin, M. A., & Tareq, A. H. (2019) · Trends in Food Science & Technology · doi:10.1016/j.tifs.2019.02.040 · Cited by 238 · see citing works
Embodiment in Socially Interactive Robots
Deng, E., Mutlu, B., & Matarić, M. J. (2019) · Foundations and Trends in Robotics · doi:10.1561/2300000056 · Cited by 238 · see citing works
Content analysis of e-petitions with topic modeling: How to train and evaluate LDA models?
Hagen, L. (2018) · Information Processing & Management · doi:10.1016/j.ipm.2018.05.006 · Cited by 233 · see citing works
Vaio, A. D., Hassan, R., & Alavoine, C. (2021) · Technological Forecasting and Social Change · doi:10.1016/j.techfore.2021.121201 · Cited by 233 · see citing works
Big Social Data Analytics in Journalism and Mass Communication
Гуо, Л., Vargo, C. J., Pan, Z., Ding, W., & Ishwar, P. (2016) · Journalism & Mass Communication Quarterly · doi:10.1177/1077699016639231 · Cited by 231 · see citing works
The Social News Gap: How News Reading and News Sharing Diverge
Bright, J. (2016) · Journal of Communication · doi:10.1111/jcom.12232 · Cited by 228 · see citing works
Sciarelli, M., Cosimato, S., Landi, G. C., & Iandolo, F. (2021) · The TQM Journal · doi:10.1108/tqm-08-2020-0180 · Cited by 227 · see citing works
What to Do (and Not to Do) with the Comparative Manifestos Project Data
Gemenis, K. (2013) · Political Studies · doi:10.1111/1467-9248.12015 · Cited by 226 · see citing works
Cultural Brokerage and Creative Performance in Multicultural Teams
Jang, S. (2017) · Organization Science · doi:10.1287/orsc.2017.1162 · Cited by 223 · see citing works
Persuasion for Good: Towards a Personalized Persuasive Dialogue System for Social Good
Wang, X., Shi, W., Kim, R., Oh, Y., Yang, S., Zhang, J., & Yu, Z. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1566 · Cited by 222 · see citing works
Barbour, J. B., Rintamaki, L. S., Ramsey, J. A., & Brashers, D. E. (2011) · Journal of Health Communication · doi:10.1080/10810730.2011.585691 · Cited by 222 · see citing works
Performance Pressure as a Double-edged Sword
Gardner, H. K. (2012) · Administrative Science Quarterly · doi:10.1177/0001839212446454 · Cited by 222 · see citing works
It’s not all about autism: The emerging landscape of anti-vaccination sentiment on Facebook
Hoffman, B. L., Felter, E. M., Chu, K., Shensa, A. S., Hermann, C., Wolynn, T. H., Williams, D., & Primack, B. A. (2019) · Vaccine · doi:10.1016/j.vaccine.2019.03.003 · Cited by 222 · see citing works
Lazar, J. K., Feng, J. H., & Hochheiser, H. (2017) · Elsevier eBooks · doi:10.1016/b978-0-12-805390-4.00011-x · Cited by 221 · see citing works
van de Pol, J., Volman, M., Oort, F., & Beishuizen, J. (2015) · Instructional Science · doi:10.1007/s11251-015-9351-z · Cited by 220 · see citing works
Giudice, M. D., Vaio, A. D., Hassan, R., & Palladino, R. (2021) · Maritime Policy & Management · doi:10.1080/03088839.2021.1903600 · Cited by 215 · see citing works
Quantitative indicators for social sustainability assessment of supply chains
Popović, T., Barbosa‐Póvoa, A. P., Krasławski, A., & Carvalho, A. I. C. D. S. G. (2018) · Journal of Cleaner Production · doi:10.1016/j.jclepro.2018.01.142 · Cited by 214 · see citing works
Expert, Crowdsourced, and Machine Assessment of Suicide Risk via Online Postings
Shing, H.-C., Nair, S., Zirikly, A., Friedenberg, M., Daumé, H., & Resnik, P. (2018) · Workshop on Computational Linguistics and Clinical Psychology (CLPsych) · doi:10.18653/v1/w18-0603 · Cited by 214 · see citing works
Goal-oriented requirements engineering: an extended systematic mapping study
Horkoff, J., Aydemir, F. B., Cardoso, E. C. S., Li, T., Maté, A., Paja, E., Salnitri, M., Piras, L., Mylopoulos, J., & Giorgini, P. (2017) · Requirements Engineering · doi:10.1007/s00766-017-0280-z · Cited by 212 · see citing works
Maintaining the Double Standard: Portrayals of Age and Gender in Popular Films
Lauzen, M. M., & Dozier, D. M. (2005) · Sex Roles · doi:10.1007/s11199-005-3710-1 · Cited by 209 · see citing works
Artstein, R. (2017) · Springer eBooks · doi:10.1007/978-94-024-0881-2_11 · Cited by 206 · see citing works
The Changing Misrepresentation of Race and Crime on Network and Cable News
Dixon, T. L., & Williams, C. L. (2014) · Journal of Communication · doi:10.1111/jcom.12133 · Cited by 202 · see citing works
Vaio, A. D., Hasan, S., Palladino, R., & Hassan, R. (2022) · Environment Development and Sustainability · doi:10.1007/s10668-021-02078-5 · Cited by 200 · see citing works
NoRBERT: Transfer Learning for Requirements Classification
Hey, T., Keim, J., Koziolek, A., & Tichy, W. F. (2020) · IEEE International Requirements Engineering Conference (RE) · doi:10.1109/re48521.2020.00028 · Cited by 199 · see citing works
Hayter, C. S. (2015) · Research Policy · doi:10.1016/j.respol.2015.11.003 · Cited by 198 · see citing works
Measuring and Reporting Intercoder Reliability in Plan Quality Evaluation Research
Stevens, M. R., Lyles, W., & Berke, P. R. (2014) · Journal of Planning Education and Research · doi:10.1177/0739456x13513614 · Cited by 197 · see citing works
Ziegele, M., Breiner, T., & Quiring, O. (2014) · Journal of Communication · doi:10.1111/jcom.12123 · Cited by 194 · see citing works
Akter, S., Bandara, R., Hani, U., Fosso Wamba, S., Foropon, C., & Papadopoulos, T. (2019) · International Journal of Information Management · doi:10.1016/j.ijinfomgt.2019.01.020 · Cited by 193 · see citing works
‘Let’s Get Them Involved’ . . . to Some Extent: Analyzing Online News Participation
Almgren, S., & Olsson, T. (2015) · Social Media + Society · doi:10.1177/2056305115621934 · Cited by 189 · see citing works
Antecedents to Psychological Reactance: The Impact of Threat, Message Frame, and Choice
Shen, L. (2014) · Health Communication · doi:10.1080/10410236.2014.910882 · Cited by 187 · see citing works
Constructing Gender Stereotypes Through Social Roles in Prime-Time Television
Lauzen, M. M., Dozier, D. M., & Horan, N. M. (2008) · Journal of Broadcasting & Electronic Media · doi:10.1080/08838150801991971 · Cited by 186 · see citing works
Assessing the Reliability and Validity of Expert Interviews
Dorussen, H., Lenz, H., & Blavoukos, S. (2005) · European Union Politics · doi:10.1177/1465116505054835 · Cited by 186 · see citing works
Scott, S. R., Rivera, K. M., Rushing, E., Manczak, E. M., Rozek, C. S., & Doom, J. R. (2021) · Journal of Adolescent Health · doi:10.1016/j.jadohealth.2020.11.010 · Cited by 185 · see citing works
Schiavone, F., Mancini, D., Leone, D., & Lavorato, D. (2021) · Technological Forecasting and Social Change · doi:10.1016/j.techfore.2021.120647 · Cited by 182 · see citing works
Mion, G., & Loza Adaui, C. R. (2019) · Sustainability · doi:10.3390/su11174612 · Cited by 180 · see citing works
Svenningsson, J., Höst, G., Hultén, M., & Hallström, J. (2022) · International Journal of Technology and Design Education · doi:10.1007/s10798-021-09657-7 · Cited by 125 · see citing works
Fankhauser, C., & Christie, J. M. (2015) · Current Biology · doi:10.1016/j.cub.2015.03.020 · Cited by 167 · see citing works
Implementing Data-Driven Smart City Applications for Future Cities
Kaluarachchi, Y. (2022) · Smart Cities · doi:10.3390/smartcities5020025 · Cited by 169 · see citing works
An empirical examination of online restaurant reviews on Yelp.com
Yang, S., Hlee, S., Lee, J., & Koo, C. (2017) · International Journal of Contemporary Hospitality Management · doi:10.1108/ijchm-11-2015-0643 · Cited by 167 · see citing works
Challenge‐based learning implementation in engineering education: A systematic literature review
Doulougeri, K., Vermunt, J. D. H. M., Bombaerts, G. J. T., & Bots, M. (2024) · Journal of Engineering Education · doi:10.1002/jee.20588 · Cited by 167 · see citing works
Making Sense of Intractable Multiparty Conflict: A Study of Framing in Four Environmental Disputes
Brummans, B. H. J. M., Putnam, L. L., Gray, B., Hanke, R., Lewicki, R. J., & Wiethoff, C. (2008) · Communication Monographs · doi:10.1080/03637750801952735 · Cited by 164 · see citing works
Mistry, B., de Ramirez, S. S., Kelen, G. D., Schmitz, P. S. K., Balhara, K. S., Levin, S., Martínez, D. A., Psoter, K. J., Anton, X. P., & Hinson, J. S. (2017) · Annals of Emergency Medicine · doi:10.1016/j.annemergmed.2017.09.036 · Cited by 160 · see citing works
Kafyulilo, A., Fisser, P., Pieters, J. M., & Voogt, J. M. (2015) · Australasian Journal of Educational Technology · doi:10.14742/ajet.1240 · Cited by 158 · see citing works
Bertuol‐Garcia, D., Morsello, C., El-Hani, C. N., & Pardini, R. (2017) · Biological reviews/Biological reviews of the Cambridge Philosophical Society · doi:10.1111/brv.12385 · Cited by 158 · see citing works
Predicting Opinion Leaders in Twitter Activism Networks
Xu, W. W., Sang, Y., Blasiola, S., & Park, H. W. (2014) · American Behavioral Scientist · doi:10.1177/0002764214527091 · Cited by 156 · see citing works
Herding and Social Media Word-of-Mouth: Evidence from Groupon1
Li, X., & Wu, L. (2013) · MIS Quarterly · doi:10.25300/misq/2018/14108 · Cited by 155 · see citing works
Hybrid Activism: Social Movement Mobilization in a Multimovement Environment
Heaney, M. T., & Rojas, F. G. (2014) · American Journal of Sociology · doi:10.1086/674897 · Cited by 154 · see citing works
EpiTools, A software suite for presurgical brain mapping in epilepsy: Intracerebral EEG
Villalon, S. M., Paz, R. M., Roehri, N., Lagarde, S., Pizzo, F., Colombet, B., Bartoloméi, F., Carron, R., & Bénar, C. (2018) · Journal of Neuroscience Methods · doi:10.1016/j.jneumeth.2018.03.018 · Cited by 154 · see citing works
- Authors
- Krippendorff, K.
- Corresponding author
- Krippendorff, K.
- ORCID
- Krippendorff, K.
- Venue
- Human Communication Research
- Publisher
- Oxford University Press (OUP)
- Published
- 2004-07-01
- Volume
- 30
- Issue
- 3
- Pages
- 411-433
- ISSN
- 0360-3989 (ISSN-L)
- 1468-2958
- OpenAlex
- W4243989635
- Cited by
- 1,027
- References
- 0
- Citations by year
- 2026: 42 · 2025: 100 · 2024: 108 · 2023: 108 · 2022: 124 · 2021: 135 · 2020: 105 · 2019: 57 · 2018: 44 · 2017: 29 · 2016: 41 · 2015: 25 · 2014: 26 · 2013: 21 · 2012: 15
- FWCI
- 5.93
- Type
- journal article
- Language
- en
- Open access
- green
- repository.upenn.edu
- Locations
- Human Communication Research (publishedVersion)
- ScholarlyCommons (University of Pennsylvania) (submittedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Reliability and Agreement in Measurement
- Social Sciences › General Social Sciences › Computational and Text Analysis Methods
- Social Sciences › Decision Sciences › Management Science and Operations Research › Psychometric Methodologies and Testing
- Keywords
- inter-rater reliability, reliability coefficients, interobserver agreement, reporting reliability
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
