Cited by 1,027Reliability and Agreement in MeasurementComputational and Text Analysis MethodsPsychometric Methodologies and Testing

The DOI matches its Crossref record; 5 of the 5 references it cites were found in a registry or an index.

Reliability in content analysis: some common misconceptions and recommendations.

Open access

Krippendorff, K.·2004·Human Communication Research

DOI 10.1093/hcr/30.3.411

Open this paper in CoChat — free account Open PDF

Abstract

In a recent article published in this journal, Lombard, Snyder-Duch, and Bracken (2002) surveyed 200 content analyses for their reporting of reliability tests; compared the virtues and drawbacks of five popular reliability measures; and proposed guidelines and standards for their use. Their discussion revealed that numerous misconceptions circulate in the content analysis literature regarding how these measures behave and can aid or deceive content analysts in their effort to ensure the reliability of their data. This paper proposes three conditions for statistical measures to serve as indices of the reliability of data and examines the mathematical structure and the behavior of the five coefficients discussed by the authors, plus two others. It compares common beliefs about these coefficients with what they actually do and concludes with alternative recommendations for testing reliability in content analysis and similar data-making efforts.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

Integrity signals

Registry

Crossref · 10.1093/hcr/30.3.411

Cites

0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

The publisher hasn’t deposited this paper’s reference list.

Research works printed but not linked: 13

Bennett, E. M., Alpert, R., & Goldstein, A. C. (1954). Communications through limited response questioning. Public Opinion Quarterly, 18, 303-308.
Craig, R. T. (1981). Generalization of Scott"s index of intercoder agreement. Public Opinion Quarterly, 45, 260-264.
Goodman, L. A, & Kruskal, W. H. (1954). Measures of association for cross classifications. Journal of the American Statistical Association, 49, 732-764.
Holsti, O. R. (1969). Content analysis for the social sciences and humanities. Reading, MA: Addison- Wesley.
Hughes, M. A., & Garrett, D. E. (1990). Intercoder reliability estimation -Approaches in marketing: A generalizability theory framework for quantitative data. Journal of Marketing Research, 27, 185- 195.
Krippendorff, K. (1970). Bivariate agreement coefficients for reliability data. In E. R. Borgatta & G. W. Bohrnstedt (Eds.), Sociological methodology 1970 (pp. 139-150). San Francisco, CA: Jossey Bass.
Krippendorff, K. (2004). Content analysis: An introduction to its methodology. Second Edition. Thousand Oaks, CA: Sage.
Maxwell, A. E. (1970). Comparing the classification of subjects by two independent judges. British Journal of Psychiatry, 116, 651-655.
Osgood, C. E. (1959). The representational model and relevant research. In I. de Sola Pool (Ed.), Trends in content analysis (pp. 33-88). Urbana: University of Illinois Press.
Potter, W. J., & Levine-Donnerstein, D. (1999). Rethinking validity and reliability in content analysis. Journal of Applied Communication Research, 27, 258-284.
Riffe, D., & Freitag, A. (1996, August). Twenty-five years of content analyses in Journalism & Mass Communication Quarterly. Paper presented to the annual convention of the Association for Education in Journalism and Mass Communication, Anaheim, CA, cited in D. Riffe, S. Lacy, & F. G. Fico (1998). Analyzing media messages. Mahwah, NJ: Erlbaum.
Scott, W. A. (1955). Reliability of content analysis: The case of nominal scale coding. Public Opinion Quarterly, 19, 321-325.
Zwick, R. (1988). Another look at interrater agreement. Psychological Bulletin, 103, 347-387.

Sorted by citations

Items 101–200 of 1,010

Multi-level governance for sustainable innovation in smart communities: an ecosystems approach
Ciasullo, M. V., Troisi, O., Grimaldi, M., & Leone, D. (2020) · International Entrepreneurship and Management Journal · doi:10.1007/s11365-020-00641-6 · Cited by 170
Blockchain technology and gender equality: A systematic literature review
Vaio, A. D., Hassan, R., & Palladino, R. (2022) · International Journal of Information Management · doi:10.1016/j.ijinfomgt.2022.102517 · Cited by 159
A 22 year review of strategic alliance research in the leading management journals
Gomes, E., Barnes, B. R., & Mahmood, T. (2014) · International Business Review · doi:10.1016/j.ibusrev.2014.03.005 · Cited by 152
Depictions of Gender on Primetime Television: A Quantitative Content Analysis
Sink, A., & Mastro, D. E. (2016) · Mass Communication & Society · doi:10.1080/15205436.2016.1212243 · Cited by 152
User engagement for mobile payment service providers – introducing the social media engagement model
Grover, P., & Kar, A. K. (2018) · Journal of Retailing and Consumer Services · doi:10.1016/j.jretconser.2018.12.002 · Cited by 150
Inter-Coder Agreement in Qualitative Coding: Considerations for its Use
Halpin, S. N. (2024) · American Journal of Qualitative Research · doi:10.29333/ajqr/14887 · Cited by 149
Online versus in-person interviews with adolescents: An exploration of data equivalence
Shapka, J. D., Domene, J. F., Khan, S., & Yang, L. (2016) · Computers in Human Behavior · doi:10.1016/j.chb.2016.01.016 · Cited by 147
Social Media Messages in an Emerging Health Crisis: Tweeting Bird Flu
Vos, S. C., & Buckner, M. M. (2015) · Journal of Health Communication · doi:10.1080/10810730.2015.1064495 · Cited by 155
An Emerging ‘European’ News Portrayal of Immigration?
Caviedes, A. (2015) · Journal of Ethnic and Migration Studies · doi:10.1080/1369183x.2014.1002199 · Cited by 146
Software Startups - A Research Agenda
Unterkalmsteiner, M., Abrahamsson, P., Wang, X., Nguyen‐Duc, A., Shah, S. M. A., Bajwa, S. S., Baltes, G. H., Conboy, K., Cullina, E., Dennehy, D., et al. (2016) · Maynooth University ePrints and eTheses Archive (Maynooth University) · doi:10.5277/e-inf160105 · Cited by 146
Hackers vs. Testers: A Comparison of Software Vulnerability Discovery Processes
Votipka, D., Stevens, R., Redmiles, E. M., Hu, J., & Mazurek, M. L. (2018) · IEEE Symposium on Security and Privacy (S&P) · doi:10.1109/sp.2018.00003 · Cited by 146
Controversial Rap Themes, Gender Portrayals and Skin Tone Distortion: A Content Analysis of Rap Music Videos
Conrad, K. M., Dixon, T. L., & Zhang, Y. (2009) · Journal of Broadcasting & Electronic Media · doi:10.1080/08838150802643795 · Cited by 145
Dimensions of lodging guest satisfaction among guests with mobility challenges: A mixed-method analysis of web-based texts
Zhang, Y., & Cole, S. T. (2015) · Tourism Management · doi:10.1016/j.tourman.2015.09.001 · Cited by 142
See Paris and… found a business? The impact of cross-cultural experience on opportunity recognition capabilities
Vandor, P., & Franke, N. (2016) · Journal of Business Venturing · doi:10.1016/j.jbusvent.2016.03.003 · Cited by 152
“Think globally, act locally”: A glocal approach to the development of social media literacy
Manca, S., Bocconi, S., & Gleason, B. (2020) · Computers & Education · doi:10.1016/j.compedu.2020.104025 · Cited by 142
A temporal analysis of how entrepreneurial goal intentions, positive fantasies, and action planning affect starting a new venture and when the effects wear off
Gielnik, M. M., Barabas, S., Fresé, M., Namatovu-Dawa, R., Scholz, F. A., Metzger, J. R., & Walter, T. J. (2013) · Journal of Business Venturing · doi:10.1016/j.jbusvent.2013.09.002 · Cited by 150
Twitter, Incivility and “Everyday” Gendered Othering: An Analysis of Tweets Sent to UK Members of Parliament
Southern, R., & Harmer, E. (2019) · Social Science Computer Review · doi:10.1177/0894439319865519 · Cited by 140
“Integrated thinking and reporting” towards sustainable business models: a concise bibliometric analysis
Vaio, A. D., Syriopoulos, T. C., Alvino, F., & Palladino, R. (2020) · Meditari Accountancy Research · doi:10.1108/medar-12-2019-0641 · Cited by 140
Multichannel integration quality: A systematic review and agenda for future research
Hossain, T. M. T., Akter, S., Kattiyapornpong, U., & Dwivedi, Y. K. (2019) · Journal of Retailing and Consumer Services · doi:10.1016/j.jretconser.2019.03.019 · Cited by 154
Navigating the validity tradeoffs of entrepreneurship research experiments: A systematic review and best-practice suggestions
Grégoire, D. A., Binder, J. K., & Rauch, A. (2018) · Journal of Business Venturing · doi:10.1016/j.jbusvent.2018.10.002 · Cited by 149
How Dominant Narratives Rise and Fall: Military Conflict, Politics, and the Cold War Consensus
Krebs, R. R. (2015) · International Organization · doi:10.1017/s0020818315000181 · Cited by 137
INSPIRED: Toward Sociable Recommendation Dialog Systems
Hayati, S. A., Kang, D., Zhu, Q., Shi, W., & Yu, Z. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.654 · Cited by 159
Why Do Incumbents Respond Heterogeneously to Disruptive Innovations? The Interplay of Domain Identity and Role Identity
Kammerlander, N. H., Konig, A. S., & Richards, M. (2018) · Journal of Management Studies · doi:10.1111/joms.12345 · Cited by 136
Exploring big data-driven innovation in the manufacturing sector: evidence from UK firms
Babu, M. M., Rahman, M., Alam, A., & Dey, B. L. (2021) · Annals of Operations Research · doi:10.1007/s10479-021-04077-1 · Cited by 150
The State of Digital Marketing in Academia: An Examination of Marketing Curriculum’s Response to Digital Disruption
Langan, R. J., Cowley, S., & Nguyen, C. A. (2019) · Journal of Marketing Education · doi:10.1177/0273475318823849 · Cited by 135
Wind(ow) of Change: A Systematic Review of Policy Entrepreneurship Characteristics and Strategies
Aviram, N. F., Cohen, N., & Beeri, I. (2019) · Policy Studies Journal · doi:10.1111/psj.12339 · Cited by 135
Building dynamic service analytics capabilities for the digital marketplace
Akter, S., Motamarri, S., Hani, U., Shams, S. M. R., Fernando, M., Babu, M. M., & Shen, K. N. (2020) · Journal of Business Research · doi:10.1016/j.jbusres.2020.06.016 · Cited by 136
Mandatory vs voluntary exercise on non-financial reporting: does a normative/coercive isomorphism facilitate an increase in quality?
Carungu, J., Pietra, R. D., & Molinari, M. (2020) · Meditari Accountancy Research · doi:10.1108/medar-08-2019-0540 · Cited by 132
The individual side of ambidexterity: Do individuals’ perceptions match actual behaviors in reconciling the exploration and exploitation trade-off?
Bonesso, S., Gerli, F., & Scapolan, A. (2013) · European Management Journal · doi:10.1016/j.emj.2013.07.003 · Cited by 141
Improving general chemistry performance through a growth mindset intervention: selective effects on underrepresented minorities
Fink, A., Cahill, M. J., McDaniel, M. A., Hoffman, A., & Frey, R. F. (2018) · Chemistry Education Research and Practice · doi:10.1039/c7rp00244k · Cited by 138
Recipes for Safety in Open-domain ChatbotsPreprint
Xu, J., Ju, D., Li, M., Boureau, Y.-L., Weston, J., & Dinan, E. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.07079 · Cited by 236
Understanding how and why audits work in improving the quality of hospital care: A systematic realist review
Hut-Mossel, L., Ahaus, K., Welker, G. A., & Gans, R. O. B. (2021) · PLoS ONE · doi:10.1371/journal.pone.0248677 · Cited by 142
The Blooming Anatomy Tool (BAT): A discipline‐specific rubric for utilizing Bloom's taxonomy in the design and evaluation of assessments in the anatomical sciences
Thompson, A. R., & O’Loughlin, V. D. (2014) · Anatomical Sciences Education · doi:10.1002/ase.1507 · Cited by 135
The antecedents of boredom in L2 classroom learning
Nakamura, S., Darasawang, P., & Reinders, H. (2021) · System · doi:10.1016/j.system.2021.102469 · Cited by 135
Resources and governance in “base of the pyramid”-partnerships: Assessing collaborations between businesses and non-business actors
Hahn, R., & Gold, S. (2013) · Journal of Business Research · doi:10.1016/j.jbusres.2013.09.002 · Cited by 126
Reliability of response region: A novel mechanism in visual tracking by edge computing for IIoT environments
Liu, S., Guo, C., Al-Turjman, F. M., Muhammad, K., & de Albuquerque, V. H. C. (2020) · Mechanical Systems and Signal Processing · doi:10.1016/j.ymssp.2019.106537 · Cited by 125
An artificial intelligence-driven learning analytics method to examine the collaborative problem-solving process from the complex adaptive systems perspective
Ouyang, F., Xu, W., & Cukurova, M. (2023) · International Journal of Computer-Supported Collaborative Learning · doi:10.1007/s11412-023-09387-z · Cited by 124
Dynamic Capabilities and MNE Global Strategy: A Systematic Literature Review‐Based Novel Conceptual Framework
Pitelis, C. N., Teece, D. J., & Yang, H. (2023) · Journal of Management Studies · doi:10.1111/joms.13021 · Cited by 124
What do people like to “share” about obesity? A content analysis of frequent retweets about obesity on Twitter
So, J., Prestin, A., Lee, L., Wang, Y., Yen, J., & Chou, W.-Y. S. (2016) · Health Communication · doi:10.1080/10410236.2014.940675 · Cited by 123
In Validations We Trust? The Impact of Imperfect Human Annotations as a Gold Standard on the Quality of Validation of Automated Content Analysis
Song, H., Tolochko, P., Eberl, J.-M., Eisele, O., Greussing, E., Heidenreich, T., Lind, F., Galyga, S., & Boomgaarden, H. G. (2020) · Political Communication · doi:10.1080/10584609.2020.1723752 · Cited by 123
Policy implementation lessons from six road pricing cases
Noordegraaf, D. V., Annema, J. A., & van Wee, B. (2013) · Transportation Research Part A Policy and Practice · doi:10.1016/j.tra.2013.11.003 · Cited by 123
What's congress doing on twitter?
Hemphill, L., Otterbacher, J., & Shapiro, M. A. (2013) · ACM Conference on Computer Supported Cooperative Work (CSCW) · doi:10.1145/2441776.2441876 · Cited by 123
Radiation risk perception: a discrepancy between the experts and the general population
Perko, T. (2013) · Journal of Environmental Radioactivity · doi:10.1016/j.jenvrad.2013.04.005 · Cited by 123
Would I be Helped? Cross-National CCTV Footage Shows That Intervention Is the Norm in Public Conflicts
Philpot, R., Liebst, L. S., Levine, M., Bernasco, W., & Lindegaard, M. R. (2019) · American Psychologist · doi:10.1037/amp0000469 · Cited by 137
Mistakes and How to Avoid Mistakes in Using Intercoder Reliability Indices
Feng, G. C. (2014) · Methodology · doi:10.1027/1614-2241/a000086 · Cited by 132
Meme Templates as Expressive Repertoires in a Globalizing World: A Cross-Linguistic Study
Nissenbaum, A., & Shifman, L. (2018) · Journal of Computer-Mediated Communication · doi:10.1093/jcmc/zmy016 · Cited by 120
Examining the extent of and determinants for sustainability assurance quality: The role of audit committees
Zaman, R., Farooq, M. B., Khalid, F., & Mahmood, Z. (2021) · Business Strategy and the Environment · doi:10.1002/bse.2777 · Cited by 120
How Much Is Enough? New Recommendations for Using Constructed Week Sampling in Newspaper Content Analysis of Health Stories
Luke, D. A., Caburnay, C. A., & Cohen, E. L. (2011) · Communication Methods and Measures · doi:10.1080/19312458.2010.547823 · Cited by 119
Portrayals of Male and Female Scientists in Television Programs Popular Among Middle School-Age Children
Long, M., Steinke, J., Applegate, B. E., Lapinski, M. K., Johnson, M. J., & Ghosh, S. (2010) · Science Communication · doi:10.1177/1075547009357779 · Cited by 119
Boosting the Potency of Resistance: Combining the Motivational Forces of Inoculation and Psychological Reactance
Miller, C. H., Ivanov, B., Sims, J. D., Compton, J., Harrison, K. J., Parker, K. A., Parker, J. L., & Averbeck, J. M. (2012) · Human Communication Research · doi:10.1111/j.1468-2958.2012.01438.x · Cited by 132
TikTok and the new language of political communication
Cervi, L., Tejedor, S., & Marín, C. (2021) · Cultura Lenguaje y Representación · doi:10.6035/clr.5817 · Cited by 118
Intercoder reliability indices: disuse, misuse, and abuse
Feng, G. C. (2013) · Quality & Quantity · doi:10.1007/s11135-013-9956-8 · Cited by 126
The App Behavior Change Scale: Creation of a Scale to Assess the Potential of Apps to Promote Behavior Change
McKay, F. H., Slykerman, S., & Dunn, M. (2018) · JMIR mhealth and uhealth · doi:10.2196/11130 · Cited by 117
Families and Networks of Internet Memes: The Relationship Between Cohesiveness, Uniqueness, and Quiddity Concreteness
Segev, E., Nissenbaum, A., Stolero, N., & Shifman, L. (2015) · Journal of Computer-Mediated Communication · doi:10.1111/jcc4.12120 · Cited by 127
Semantic similarity metrics for evaluating source code summarization
Haque, S., Eberhart, Z., Bansal, A., & McMillan, C. (2022) · Proceedings of the 30th IEEE/ACM International Conference on Program Comprehension · doi:10.1145/3524610.3527909 · Cited by 116
Good Guys Are Still Always in White? Positive Change and Continued Misrepresentation of Race and Crime on Local Television News
Dixon, T. L. (2015) · Communication Research · doi:10.1177/0093650215579223 · Cited by 116
Developing peer mentors’ collaborative and metacognitive skills with a technology-enhanced peer learning program
Carvalho, A. R., & Santos, C. (2021) · Computers and Education Open · doi:10.1016/j.caeo.2021.100070 · Cited by 116
Cross-Cultural Differences in Sexual Advertising Content in a Transnational Women's Magazine
Nelson, M. R., & Paek, H. (2005) · Sex Roles · doi:10.1007/s11199-005-6760-5 · Cited by 116
The Hispanic Stress Inventory Version 2: Improving the assessment of acculturation stress.
Cervantes, R. C., Fisher, D. G., Padilla, A. M., & Napper, L. E. (2015) · Psychological Assessment · doi:10.1037/pas0000200 · Cited by 115
How Pro-social Framing Affects the Success of Crowdfunding Projects: The Role of Emphasis and Information Crowdedness
Defazio, D., Franzoni, C., & Rossi‐Lamastra, C. (2020) · Journal of Business Ethics · doi:10.1007/s10551-020-04428-1 · Cited by 118
THE EFFECT OF LEARNER CHOICE ON L2 TASK ENGAGEMENT
Nakamura, S., Phung, L., & Reinders, H. (2020) · Studies in Second Language Acquisition · doi:10.1017/s027226312000042x · Cited by 122
Rethinking innovation through industry and society 5.0 paradigms: a multileveled approach for management and policy-making
Troisi, O., Visvizi, A., & Grimaldi, M. (2023) · European Journal of Innovation Management · doi:10.1108/ejim-08-2023-0659 · Cited by 114
Assumptions behind Intercoder Reliability Indices
Zhao, X., Liu, J. S., & Deng, K. (2013) · Annals of the International Communication Association · doi:10.1080/23808985.2013.11679142 · Cited by 113
The sociology of scientific validity: How professional networks shape judgement in peer review
Teplitskiy, M., Acuna, D., Elamrani-Raoult, A., Körding, K., & Evans, J. (2018) · Research Policy · doi:10.1016/j.respol.2018.06.014 · Cited by 113
Knowledge, Skills, and Abilities for Entry‐Level Business Analytics Positions: A Multi‐Method Study
Cegielski, C. G., & Jones‐Farmer, L. A. (2016) · Decision Sciences Journal of Innovative Education · doi:10.1111/dsji.12086 · Cited by 112
Analyzing public sentiments online: combining human- and computer-based content analysis
Su, L. Y.-F., Cacciatore, M. A., Liang, X., Brossard, D., Scheufele, D. A., & Xenos, M. A. (2016) · Information Communication & Society · doi:10.1080/1369118x.2016.1182197 · Cited by 112
Normative Change and Culture of Hate: An Experiment in Online Environments
Álvarez-Benjumea, A., & Winter, F. (2018) · European Sociological Review · doi:10.1093/esr/jcy005 · Cited by 117
Three Pathways to Case Selection in International Business: A Twenty–Year Review, Analysis and Synthesis
Fletcher, M. A., Zhao, Y., Plakoyiannaki, E., & Buck, T. (2018) · International Business Review · doi:10.1016/j.ibusrev.2017.12.004 · Cited by 125
A Study of Misinformation in WhatsApp groups with a focus on the Brazilian Presidential Elections.
Machado, C. C. V., Kira, B., Narayanan, V., Kollányi, B., & Howard, P. N. (2019) · International World Wide Web Conference (WWW) · doi:10.1145/3308560.3316738 · Cited by 110
Destination image through social media analytics and survey method
Lin, M. S., Liang, Y., Xue, X., Pan, B., & Schroeder, A. (2021) · International Journal of Contemporary Hospitality Management · doi:10.1108/ijchm-08-2020-0861 · Cited by 118
Conceptualizing “Dark Platforms”. Covid-19-Related Conspiracy Theories on 8kun and Gab
Zeng, J., & Schäfer, M. S. (2021) · Digital Journalism · doi:10.1080/21670811.2021.1938165 · Cited by 125
“THE RUSSIANS ARE HACKING MY BRAIN!” investigating Russia's internet research agency twitter tactics during the 2016 United States presidential campaign
Linvill, D. L., Boatwright, B., Grant, W. J., & Warren, P. L. (2019) · Computers in Human Behavior · doi:10.1016/j.chb.2019.05.027 · Cited by 109
Thematic content analysis using supervised machine learning: An empirical evaluation using German online news
Scharkow, M. (2011) · Quality & Quantity · doi:10.1007/s11135-011-9545-7 · Cited by 109
Reliability Measurement without Limits
Reidsma, D., & Carletta, J. (2008) · Computational Linguistics · doi:10.1162/coli.2008.34.3.319 · Cited by 109
Maritime safety and the ISM code: a study of investigated casualties and incidents
Batalden, B.-M., & Sydnes, A. K. (2013) · WMU Journal of Maritime Affairs · doi:10.1007/s13437-013-0051-8 · Cited by 108
Free Association in Psychology and the Grid Elaboration Method
Joffé, H., & Elsey, J. W. B. (2014) · Review of General Psychology · doi:10.1037/gpr0000014 · Cited by 126
“Know What to Do If You Encounter a Flash Flood”: Mental Models Analysis for Improving Flash Flood Risk Communication and Public Decision Making
Lazrus, H., Morss, R. E., Demuth, J. L., Lazo, J. K., & Bostrom, A. (2015) · Risk Analysis · doi:10.1111/risa.12480 · Cited by 107
COVID-19, a tale of two pandemics: novel coronavirus and fake news messaging
Atehortua, N., & Patino, S. (2020) · Health Promotion International · doi:10.1093/heapro/daaa140 · Cited by 107
Exploring car manufacturers’ responses to technology-forcing regulation: The case of California's ZEV mandate
Wesseling, J. H., Farla, J. C. M., & Hekkert, M. P. (2015) · Environmental Innovation and Societal Transitions · doi:10.1016/j.eist.2015.03.001 · Cited by 119
Crowdsourced validation of a machine-learning classification system for autism and ADHD
Duda, M., Haber, N., Daniels, J., & Wall, D. P. (2017) · Translational Psychiatry · doi:10.1038/tp.2017.86 · Cited by 105
What your Facebook Profile Picture Reveals about your Personality
Segalin, C., Celli, F., Polonio, L., Kosiński, M., Stillwell, D. J., Sebe, N., Cristani, M., & Lepri, B. (2017) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3123266.3123331 · Cited by 106
Liking, sharing, commenting and reacting on Facebook: User behaviors’ impact on sentiment intensity
Kaur, W., Balakrishnan, V., RANA, O. F., & Sinniah, A. (2018) · Telematics and Informatics · doi:10.1016/j.tele.2018.12.005 · Cited by 105
Content analytic approach to measuring constructs in operations and supply chain management
Tangpong, C. (2011) · Journal of Operations Management · doi:10.1016/j.jom.2010.08.001 · Cited by 104
Designs for learning about climate change as a complex system
Jacobson, M. J., Markauskaitė, L., Portolese, A., Kapur, M., Lai, P. K., & Roberts, G. (2017) · Learning and Instruction · doi:10.1016/j.learninstruc.2017.03.007 · Cited by 109
If and how do 360° videos fit into education settings? Results from a scoping review of empirical research
Ranieri, M., Luzzi, D., Cuomo, S., & Bruni, I. (2022) · Journal of Computer Assisted Learning · doi:10.1111/jcal.12683 · Cited by 118
Interrater Reliability of mHealth App Rating Measures: Analysis of Top Depression and Smoking Cessation Apps
Powell, A. C., Torous, J. B., Chan, S. R., Raynor, G., Shwarts, E., Shanahan, M. E., & Landman, A. (2016) · JMIR mhealth and uhealth · doi:10.2196/mhealth.5176 · Cited by 103
The current state of methods for establishing reliability in qualitative chemistry education research articles
Watts, F. M., & Finkenstaedt-Quinn, S. A. (2021) · Chemistry Education Research and Practice · doi:10.1039/d1rp00007a · Cited by 115
Complementarity of circular economy practices: an empirical analysis of Chinese manufacturers
Yang, Y., Chen, L., Jia, F., & Xu, Z. (2019) · International Journal of Production Research · doi:10.1080/00207543.2019.1566664 · Cited by 106
Corporations and sustainable development goals communication on social media: Corporate social responsibility or just another buzzword?
ElAlfy, A., Darwish, K. M., & Weber, O. (2020) · Sustainable Development · doi:10.1002/sd.2095 · Cited by 102
Sustainability reporting after the Costa Concordia disaster: a multi-theory study on legitimacy, impression management and image restoration
Corazza, L., Truant, E., Scagnelli, S. D., & Mio, C. (2020) · Accounting Auditing & Accountability Journal · doi:10.1108/aaaj-05-2018-3488 · Cited by 102
Different patterns in the evolution of digital and non-digital ventures' business models
König, M., Ungerer, C., Baltes, G. H., & Terzidis, O. (2018) · Technological Forecasting and Social Change · doi:10.1016/j.techfore.2018.05.006 · Cited by 101
A content analysis of cross-cultural motivational studies in tourism relating to nationalities
Soldatenko, D., & Backer, E. R. (2019) · Journal of Hospitality and Tourism Management · doi:10.1016/j.jhtm.2018.12.004 · Cited by 101
A Content Analysis of Smoking Fetish Videos on YouTube: Regulatory Implications for Tobacco Control
Kim, K., Paek, H., & Lynn, J. (2010) · Health Communication · doi:10.1080/10410230903544415 · Cited by 100
Theory of Mind in Social Robots: Replication of Five Established Human Tests
Banks, J. (2019) · International Journal of Social Robotics · doi:10.1007/s12369-019-00588-x · Cited by 100
Unpacking Epistemic Insights of Artificial Intelligence (AI) in Science Education: A Systematic Review
Cheung, K. K. C., Yun, L., Liu, Q., & Chan, H. (2024) · Science & Education · doi:10.1007/s11191-024-00511-5 · Cited by 109
Teacher Scaffolding in Small-Group Work: An Intervention Study
van de Pol, J., Volman, M. L. L., Oort, F. J., & Beishuizen, J. J. (2013) · Journal of the Learning Sciences · doi:10.1080/10508406.2013.805300 · Cited by 118
Effectiveness of corporate social media activities in increasing relational outcomes
Risius, M., & Beck, R. (2015) · Information & Management · doi:10.1016/j.im.2015.06.004 · Cited by 100
Changes in Media Portrayal of Human-wildlife Conflict During Successive Fatal Shark Bites
Sabatier, E., & Huveneers, C. P. M. (2018) · Conservation and Society · doi:10.4103/cs.cs_18_5 · Cited by 99
Express yourself: Facial expression of happiness, anger, fear, and sadness in funding pitches
Warnick, B. J., Davis, B. C., Allison, T. H., & Anglin, A. H. (2021) · Journal of Business Venturing · doi:10.1016/j.jbusvent.2021.106109 · Cited by 100
Does intellectual capital disclosure affect the cost of equity capital? An empirical analysis in the integrated reporting context
Salvi, A., Vitolla, F., Raimo, N., Rubino, M., & Petruzzella, F. (2020) · Journal of Intellectual Capital · doi:10.1108/jic-12-2019-0283 · Cited by 98

Authors: Krippendorff, K.
Corresponding author: Krippendorff, K.
ORCID: Krippendorff, K.
Venue: Human Communication Research
Publisher: Oxford University Press (OUP)
Published: 2004-07-01
Volume: 30
Issue: 3
Pages: 411-433
ISSN: 0360-3989 (ISSN-L)
1468-2958
DOI: 10.1093/hcr/30.3.411
OpenAlex: W4243989635
Cited by: 1,027
References: 0
Citations by year: 2026: 42 · 2025: 100 · 2024: 108 · 2023: 108 · 2022: 124 · 2021: 135 · 2020: 105 · 2019: 57 · 2018: 44 · 2017: 29 · 2016: 41 · 2015: 25 · 2014: 26 · 2013: 21 · 2012: 15
FWCI: 5.93
Type: journal article
Language: en
Open access: green
repository.upenn.edu
Licence: doi.wiley.com/10.1002/tdm_license_1 (tdm, 4079 d)
Locations: Human Communication Research (publishedVersion)
ScholarlyCommons (University of Pennsylvania) (submittedVersion)
Topics: Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Reliability and Agreement in Measurement
Social Sciences › General Social Sciences › Computational and Text Analysis Methods
Social Sciences › Decision Sciences › Management Science and Operations Research › Psychometric Methodologies and Testing
Keywords: inter-rater reliability, reliability coefficients, interobserver agreement, reporting reliability

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Reliability in content analysis: some common misconceptions and recommendations.

Abstract

Overview

Integrity signals

Verification

Claims the paper makes

Claims other papers make about this one

Self-check

Citations

Cites

Cited by

Related

Metadata

Record

About these checks