Cited by 1,027Reliability and Agreement in MeasurementComputational and Text Analysis MethodsPsychometric Methodologies and Testing

The DOI matches its Crossref record; 5 of the 5 references it cites were found in a registry or an index.

Reliability in content analysis: some common misconceptions and recommendations.

Open access

Krippendorff, K.·2004·Human Communication Research

DOI 10.1093/hcr/30.3.411

Open this paper in CoChat — free account Open PDF

Abstract

In a recent article published in this journal, Lombard, Snyder-Duch, and Bracken (2002) surveyed 200 content analyses for their reporting of reliability tests; compared the virtues and drawbacks of five popular reliability measures; and proposed guidelines and standards for their use. Their discussion revealed that numerous misconceptions circulate in the content analysis literature regarding how these measures behave and can aid or deceive content analysts in their effort to ensure the reliability of their data. This paper proposes three conditions for statistical measures to serve as indices of the reliability of data and examines the mathematical structure and the behavior of the five coefficients discussed by the authors, plus two others. It compares common beliefs about these coefficients with what they actually do and concludes with alternative recommendations for testing reliability in content analysis and similar data-making efforts.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

Integrity signals

Registry

Crossref · 10.1093/hcr/30.3.411

Cites

0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

The publisher hasn’t deposited this paper’s reference list.

Research works printed but not linked: 13

Bennett, E. M., Alpert, R., & Goldstein, A. C. (1954). Communications through limited response questioning. Public Opinion Quarterly, 18, 303-308.
Craig, R. T. (1981). Generalization of Scott"s index of intercoder agreement. Public Opinion Quarterly, 45, 260-264.
Goodman, L. A, & Kruskal, W. H. (1954). Measures of association for cross classifications. Journal of the American Statistical Association, 49, 732-764.
Holsti, O. R. (1969). Content analysis for the social sciences and humanities. Reading, MA: Addison- Wesley.
Hughes, M. A., & Garrett, D. E. (1990). Intercoder reliability estimation -Approaches in marketing: A generalizability theory framework for quantitative data. Journal of Marketing Research, 27, 185- 195.
Krippendorff, K. (1970). Bivariate agreement coefficients for reliability data. In E. R. Borgatta & G. W. Bohrnstedt (Eds.), Sociological methodology 1970 (pp. 139-150). San Francisco, CA: Jossey Bass.
Krippendorff, K. (2004). Content analysis: An introduction to its methodology. Second Edition. Thousand Oaks, CA: Sage.
Maxwell, A. E. (1970). Comparing the classification of subjects by two independent judges. British Journal of Psychiatry, 116, 651-655.
Osgood, C. E. (1959). The representational model and relevant research. In I. de Sola Pool (Ed.), Trends in content analysis (pp. 33-88). Urbana: University of Illinois Press.
Potter, W. J., & Levine-Donnerstein, D. (1999). Rethinking validity and reliability in content analysis. Journal of Applied Communication Research, 27, 258-284.
Riffe, D., & Freitag, A. (1996, August). Twenty-five years of content analyses in Journalism & Mass Communication Quarterly. Paper presented to the annual convention of the Association for Education in Journalism and Mass Communication, Anaheim, CA, cited in D. Riffe, S. Lacy, & F. G. Fico (1998). Analyzing media messages. Mahwah, NJ: Erlbaum.
Scott, W. A. (1955). Reliability of content analysis: The case of nominal scale coding. Public Opinion Quarterly, 19, 321-325.
Zwick, R. (1988). Another look at interrater agreement. Psychological Bulletin, 103, 347-387.

Sorted by citations

Items 201–300 of 1,010

Multimodal learning analytics of collaborative patterns during pair programming in higher education
Xu, W., Wu, Y., & Ouyang, F. (2023) · International Journal of Educational Technology in Higher Education · doi:10.1186/s41239-022-00377-z · Cited by 98
Corporate social responsibility and circular economy integration framework within sustainable supply chain management: Building blocks for industry 5.0
SANTIAGO, B. D. S., Scavarda, L. F., Caiado, R. G. G., Santos, R. S., & Nascimento, D. L. D. M. (2024) · Corporate Social Responsibility and Environmental Management · doi:10.1002/csr.2949 · Cited by 99
Animal Welfare Concerns and Values of Stakeholders Within the Dairy Industry
Ventura, B. A., von Keyserlingk, M. A. G., & Weary, D. M. (2014) · Journal of Agricultural and Environmental Ethics · doi:10.1007/s10806-014-9523-x · Cited by 97
Understanding knowledge hiding in business organizations: A bibliometric analysis of research trends, 1988–2020
Vaio, A. D., Hasan, S., Palladino, R., Profita, F., & Mejri, I. (2021) · Journal of Business Research · doi:10.1016/j.jbusres.2021.05.040 · Cited by 96
Plan integration for resilience scorecard: evaluating networks of plans in six US coastal cities
Berke, P. R., Malecha, M. L., Yu, S., Lee, J., & Masterson, J. H. (2018) · Journal of Environmental Planning and Management · doi:10.1080/09640568.2018.1453354 · Cited by 95
Are We Planning for Equity?
Loh, C. G., & Kim, R. (2020) · Journal of the American Planning Association · doi:10.1080/01944363.2020.1829498 · Cited by 95
Can you hear me now? Engendering passion and preparedness perceptions with vocal expressions in crowdfunding pitches
Allison, T. H., Warnick, B. J., Davis, B. C., & Cardon, M. S. (2022) · Journal of Business Venturing · doi:10.1016/j.jbusvent.2022.106193 · Cited by 96
Unveiling the multimedia unconscious
Cristani, M., Vinciarelli, A., Segalin, C., & Perina, A. (2013) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/2502081.2502280 · Cited by 94
A performance measurement framework for socially sustainable and resilient supply chains using environmental goods valuation methods
Hervani, A. A., Nandi, S., Helms, M. M., & Sarkis, J. (2021) · Sustainable Production and Consumption · doi:10.1016/j.spc.2021.11.026 · Cited by 94
Detecting misogyny in Spanish tweets. An approach based on linguistics features and word embeddings
García-Díaz, J. A., Cánovas-García, M., Colomo‐Palacios, R., & Valencia‐García, R. (2020) · Future Generation Computer Systems · doi:10.1016/j.future.2020.08.032 · Cited by 96
Low Stakes Opportunism
Jap, S. D., Robertson, D. C., Rindfleisch, A., & Hamilton, R. (2012) · Journal of Marketing Research · doi:10.1509/jmr.10.0121 · Cited by 95
Influence of Scaffolding on Information Literacy and Argumentation Skills in Virtual Field Trips and Problem-Based Learning for Scientific Problem Solving
Kim, N. J., Vicentini, C. R., & Belland, B. R. (2021) · International Journal of Science and Mathematics Education · doi:10.1007/s10763-020-10145-y · Cited by 92
Social Identity Threat Motivates Science-Discrediting Online Comments
Nauroth, P., Gollwitzer, M., Bender, J., & Rothmund, T. (2015) · PLoS ONE · doi:10.1371/journal.pone.0117476 · Cited by 91
Robotics at workplace: An integrated Twitter analytics – SEM based approach for behavioral intention to accept
Sinha, N., Singh, P., Gupta, M., & Singh, P. (2020) · International Journal of Information Management · doi:10.1016/j.ijinfomgt.2020.102210 · Cited by 91
Snakes and Ladders: Unpacking the Personalisation-Privacy Paradox in the Context of AI-Enabled Personalisation in the Physical Retail Environment
Canhoto, A. I., Keegan, B. J., & Ryzhikh, M. (2023) · Information Systems Frontiers · doi:10.1007/s10796-023-10369-7 · Cited by 104
Nurses’ accuracy and self-perceived ability using the Emergency Severity Index triage tool: a cross-sectional study in four Swiss hospitals
Jordi, K., Grossmann, F. F., Gaddis, G., Cignacco, E., Denhaerynck, K., Schwendimann, R., & Nickel, C. H. (2015) · Scandinavian Journal of Trauma Resuscitation and Emergency Medicine · doi:10.1186/s13049-015-0142-y · Cited by 89
Risk Allocation in U.S. Public-Private Partnership Highway Project Contracts
Nguyen, A., Garvin, M. J., & González, E. E. (2018) · Journal of Construction Engineering and Management · doi:10.1061/(asce)co.1943-7862.0001465 · Cited by 89
Regulated Dependence: Platform Workers’ Responses to New Forms of Organizing
Karanović, J., Berends, H., & Engel, Y. (2020) · Journal of Management Studies · doi:10.1111/joms.12577 · Cited by 93
Exploring the effect of three scaffoldings on the collaborative problem-solving processes in China’s higher education
Ouyang, F., Chen, Z., Cheng, M., Tang, Z., & Su, C. (2021) · International Journal of Educational Technology in Higher Education · doi:10.1186/s41239-021-00273-y · Cited by 90
Towards a framework for developing semantic relatedness reference standards
Pakhomov, S., Ted Pedersen, A., McInnes, B. T., Melton, G. B., Ruggieri, A. P., & Chute, C. G. (2010) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2010.10.004 · Cited by 89
Managing for climate change on federal lands of the western United States: perceived usefulness of climate science, effectiveness of adaptation strategies, and barriers to implementation
Kemp, K. B., Blades, J. J., Klos, P. Z., Hall, T. E., Force, J. E., Morgan, P., & Tinkham, W. T. (2015) · Ecology and Society · doi:10.5751/es-07522-200217 · Cited by 89
Silencing nonstandard speakers: A content analysis of accent portrayals on American primetime television
Dragojević, M., Mastro, D. E., Giles, H., & Sink, A. (2016) · Language in Society · doi:10.1017/s0047404515000743 · Cited by 88
Three Decades of Reliability in Communication Content Analyses
Lovejoy, J., Watson, B. R., Lacy, S. R., & Riffe, D. (2016) · Journalism & Mass Communication Quarterly · doi:10.1177/1077699016644558 · Cited by 97
A content analysis of JUUL discussions on social media: Using Reddit to understand patterns and perceptions of JUUL use
Brett, E. I., Stevens, E. M., Wagener, T. L., Leavens, E. L. S., Morgan, T. L., Cotton, W. D., & Hébert, E. T. (2018) · Drug and Alcohol Dependence · doi:10.1016/j.drugalcdep.2018.10.014 · Cited by 88
New Trends in the Global Digital Transformation Process of the Agri-Food Sector: An Exploratory Study Based on Twitter
Ancín, M., Pindado, E., & Sánchez, M. (2022) · Agricultural Systems · doi:10.1016/j.agsy.2022.103520 · Cited by 88
Reframing data ethics in research methods education: a pathway to critical data literacy
Atenas, J., Havemann, L., & Timmermann, C. (2023) · International Journal of Educational Technology in Higher Education · doi:10.1186/s41239-023-00380-y · Cited by 88
Characterizing Instructional Practices in the Laboratory: The Laboratory Observation Protocol for Undergraduate STEM
Velasco, J. B., Knedeisen, A., Xue, D., Vickrey, T. L., Abebe, M. D., & Stains, M. (2016) · Journal of Chemical Education · doi:10.1021/acs.jchemed.6b00062 · Cited by 88
The Experience of Untapped Potential: Towards a Subjective Temporal Understanding of Work Meaningfulness
Boeck, G. D., Dries, N., & Tierens, H. (2018) · Journal of Management Studies · doi:10.1111/joms.12417 · Cited by 95
(Re)Claiming Our Expertise: Parsing Large Text Corpora With Manually Validated and Organic Dictionaries
Muddiman, A., McGregor, S. C., & Stroud, N. (2018) · Political Communication · doi:10.1080/10584609.2018.1517843 · Cited by 71
Selecting suppliers for socially sustainable supply chain management:post-exchange supplier development activities as pre-selection requirements
Cole, R., & Aitken, J. M. (2019) · Production Planning & Control · doi:10.1080/09537287.2019.1595208 · Cited by 87
Using data mining techniques to explore security issues in smart living environments in Twitter
Saura, J. R., Palacios‐Marqués, D., & Soriano, D. R. (2021) · Computer Communications · doi:10.1016/j.comcom.2021.08.021 · Cited by 88
Evaluating Discourse and Dialogue Coding Schemes
Craggs, R., & Wood, M. M. (2005) · Computational Linguistics · doi:10.1162/089120105774321109 · Cited by 85
Examining the extent of and drivers for materiality assessment disclosures in sustainability reports
Farooq, M. B., Zaman, R., Sarraj, D., & Khalid, F. (2021) · Sustainability Accounting Management and Policy Journal · doi:10.1108/sampj-04-2020-0113 · Cited by 84
The Effects of Message Framing and Visual Image on Persuasion
Seo, K., Dillard, J. P., & Shen, F. (2013) · Communication Quarterly · doi:10.1080/01463373.2013.822403 · Cited by 90
The Integration of Lean Construction and Sustainable Construction: A Stakeholder Perspective in Analyzing Sustainable Lean Construction Strategies in Malaysia
Jamil, A. H. A., & Fathi, M. S. (2016) · Procedia Computer Science · doi:10.1016/j.procs.2016.09.205 · Cited by 92
Operationalizing resilience: A content analysis of flood disaster planning in two coastal cities in Central Java, Indonesia
Handayani, W., Fisher, M. R., Rudiarto, I., Setyono, J. S., & Foley, D. (2019) · International Journal of Disaster Risk Reduction · doi:10.1016/j.ijdrr.2019.101073 · Cited by 84
Gun Stories: How Evidence Shapes Firearm Policy in the United States
Smith‐Walter, A., Peterson, H. L., Jones, M. D., & Marshall, A. N. R. (2016) · Politics & Policy · doi:10.1111/polp.12187 · Cited by 87
Acil Uzaktan Eğitim Sürecinin Analizi: Öne Çıkan Kavramlar, Sorunlar ve Çıkarılan Dersler
Sezgin, S. (2021) · Anadolu Üniversitesi Sosyal Bilimler Dergisi · doi:10.18037/ausbd.902616 · Cited by 92
An analysis of teachers questioning strategies
Döş, B., Erdal, B., Ceyda, A., Betul, T., Nurgul, C., & DUMAN, C. (2016) · Educational Research and Reviews · doi:10.5897/err2016.3014 · Cited by 82
Retweeting Risk Communication: The Role of Threat and Efficacy
Vos, S. C., Sutton, J. N., Yu, Y., Renshaw, S. L., Olson, M. K., Gibson, C. B., & Butts, C. T. (2018) · Risk Analysis · doi:10.1111/risa.13140 · Cited by 82
Commentary: A Dissenting View on So-Called Paradoxes of Reliability Coefficients
Krippendorff, K. (2013) · Annals of the International Communication Association · doi:10.1080/23808985.2013.11679143 · Cited by 82
Self-citation
A Bayesian Network Meta-Analysis to Synthesize the Influence of Contexts of Scaffolding Use on Cognitive Outcomes in STEM Education
Belland, B. R., Walker, A. E., & Kim, N. J. (2017) · Review of Educational Research · doi:10.3102/0034654317723009 · Cited by 90
Analyzing the startup ecosystem of India: a Twitter analytics perspective
Singh, S., Chauhan, A., & Dhir, S. (2019) · Journal of Advances in Management Research · doi:10.1108/jamr-08-2019-0164 · Cited by 81
A Study of Quantitative Content Analysis of Health Messages in U.S. Media From 1985 to 2005
Manganello, J. A., & Blake, N. (2010) · Health Communication · doi:10.1080/10410236.2010.483333 · Cited by 81
Stability of market segmentation with cluster analysis – A methodological approach
Müller, H., & Hamm, U. (2014) · Food Quality and Preference · doi:10.1016/j.foodqual.2013.12.004 · Cited by 81
The Impact of Gender Expectations on Meanings of Sex and Sexuality: Results from a Cognitive Interview Study
McCabe, J. M., Tanner, A. E., & Heiman, J. R. (2009) · Sex Roles · doi:10.1007/s11199-009-9723-4 · Cited by 81
The role of idioms in sentiment analysis
Williams, L., Bannister, C. A., Arribas‐Ayllon, M., Preece, A. D., & Spasić, ‪. (2015) · Expert Systems with Applications · doi:10.1016/j.eswa.2015.05.039 · Cited by 81
The ups and downs of being lesbian, gay, and bisexual: A daily experience perspective on minority stress and support processes.
Mohr, J. J., & Sarno, E. L. (2015) · Journal of Counseling Psychology · doi:10.1037/cou0000125 · Cited by 90
Fostering collective and individual learning through knowledge building
Zhao, K., & Chan, C. K. K. (2014) · International Journal of Computer-Supported Collaborative Learning · doi:10.1007/s11412-013-9188-x · Cited by 80
PERSONAL CONSTRUCT PSYCHOLOGY AND CONTENT ANALYSIS
Green, B. (2004) · Cited by 80
Quantifying the feminine self(ie): Gender display and social media feedback in young women’s Instagram selfies
Butkowski, C. P., Dixon, T. L., Weeks, K. R., & Smith, M. A. L. (2019) · New Media & Society · doi:10.1177/1461444819871669 · Cited by 80
Human capital disclosure practices of top Bangladeshi companies
Khan, H. Z., & Khan, R. (2010) · Journal of Human Resource Costing & Accounting · doi:10.1108/14013381011105984 · Cited by 80
Beyond Idea Generation: The Power of Groups in Developing Ideas
McMahon, K., Ruggeri, A., Kämmer, J. E., & Katsikopoulos, K. V. (2016) · Creativity Research Journal · doi:10.1080/10400419.2016.1195637 · Cited by 82
Crossing Cultural Boundaries Using the Internet: Toward Building a Model of Swift Trust Formation in Global Virtual Teams
Zakaria, N., & Yusof, S. A. M. (2018) · Journal of International Management · doi:10.1016/j.intman.2018.10.004 · Cited by 91
A systematic review of personalized learning in higher education: learning content structure, learning materials sequence, and learning readiness support
Zhong, L. (2022) · Interactive Learning Environments · doi:10.1080/10494820.2022.2061006 · Cited by 79
Flash Flood Risks and Warning Decisions: A Mental Models Study of Forecasters, Public Officials, and Media Broadcasters in Boulder, Colorado
Morss, R. E., Demuth, J. L., Bostrom, A., Lazo, J. K., & Lazrus, H. (2015) · Risk Analysis · doi:10.1111/risa.12403 · Cited by 79
Methodological issues in developing a multi-dimensional coding procedure for small-group chat communication
Strijbos, J.-W., & Stahl, G. (2007) · Learning and Instruction · doi:10.1016/j.learninstruc.2007.03.005 · Cited by 79
Car manufacturers’ changing political strategies on the ZEV mandate
Wesseling, J. H., Farla, J. C. M., Sperling, D., & Hekkert, M. P. (2014) · Transportation Research Part D Transport and Environment · doi:10.1016/j.trd.2014.06.006 · Cited by 79
Multiple goals and time constraints: perceived impact on physicians' performance of evidence-based behaviours
Presseau, J., Sniehotta, F. F., Francis, J., & Campbell, N. C. (2009) · Implementation Science · doi:10.1186/1748-5908-4-77 · Cited by 85
Review of Discussions on Internet of Things (IoT)
Joseph, N., Kar, A. K., Ilavarasan, P. V., & Ganesh, S. (2017) · Journal of Global Information Management · doi:10.4018/jgim.2017040103 · Cited by 78
The immediate effect of discrimination on mental health: A meta-analytic review of the causal evidence.
Emmer, C., Dorn, J., & Mata, J. (2024) · Psychological Bulletin · doi:10.1037/bul0000419 · Cited by 93
Learning in the wild: coding for learning and practice on Reddit
Haythornthwaite, C., Kumar, P., Gruzd, A. A., Gilbert, S., Valle, M. E. D., & Paulin, D. (2018) · Learning Media and Technology · doi:10.1080/17439884.2018.1498356 · Cited by 85
Board Gender Diversity and Investment Efficiency: Global Evidence from 83 Country-Level Interventions
Baik, D. (. I., Chen, C. X., & Godsell, D. (2023) · The Accounting Review · doi:10.2308/tar-2022-0251 · Cited by 77
Collective Civic Moderation for Deliberation? Exploring the Links between Citizens’ Organized Engagement in Comment Sections and the Deliberative Quality of Online Discussions
Frieß, D., Ziegele, M., & Heinbach, D. (2020) · Political Communication · doi:10.1080/10584609.2020.1830322 · Cited by 76
Business Lobbying in the European Union
Coen, D., Katsaitis, A., & Vannoni, M. (2021) · doi:10.1093/oso/9780199589753.001.0001 · Cited by 76
A Meta-Analysis of How Country-Level Factors Affect Web Survey Response Rates
Daikeler, J., Silber, H., & Bošnjak, M. (2021) · International Journal of Market Research · doi:10.1177/14707853211050916 · Cited by 75
Tracking the Werther Effect on social media: Emotional responses to prominent suicide deaths on twitter and subsequent increases in suicide
Fahey, R. A., Matsubayashi, T., & Ueda, M. (2018) · Social Science & Medicine · doi:10.1016/j.socscimed.2018.10.004 · Cited by 75
As Goes the Statue, So Goes the War: The Emergence of the Victory Frame in Television Coverage of the Iraq War
Aday, S., Cluverius, J., & Livingston, S. G. (2005) · Journal of Broadcasting & Electronic Media · doi:10.1207/s15506878jobem4903_4 · Cited by 74
Macrostructural Narrative Language of Adolescents and Young Adults With Down Syndrome or Fragile X Syndrome
Finestack, L. H., Palmer, M., & Abbeduto, L. (2012) · American Journal of Speech-Language Pathology · doi:10.1044/1058-0360(2011/10-0095 · Cited by 74
A Friendships and Dating Program for Adults with Intellectual and Developmental Disabilities: A Formative Evaluation
Ward, K. M., Atkinson, J. P., Smith, C. A., & Windsor, R. A. (2013) · Intellectual and developmental disabilities · doi:10.1352/1934-9556-51.01.022 · Cited by 74
What Does the Media Mean by ‘Sustainability’ or ‘Sustainable Development’? an Empirical Analysis of Sustainability Terminology in German Newspapers Over Two Decades
Fischer, D., Haucke, F. V., & Sundermann, A. (2017) · Sustainable Development · doi:10.1002/sd.1681 · Cited by 73
Future design of accessibility in games: A design vocabulary
Cairns, P. A., Power, C., Barlet, M., & Haynes, G. (2019) · International Journal of Human-Computer Studies · doi:10.1016/j.ijhcs.2019.06.010 · Cited by 89
Conducting qualitative content analysis in urban planning research and urban studies
Sheydayi, A., & Dadashpoor, H. (2023) · Habitat International · doi:10.1016/j.habitatint.2023.102878 · Cited by 73
“How We Know What We Know”: A Systematic Comparison of Research Methods Employed in Higher Education Journals, 1996—2000 v. 2006—2010
Wells, R. S., Kolek, E. A., Williams, E. A., & Saunders, D. B. (2015) · The Journal of Higher Education · doi:10.1080/00221546.2015.11777361 · Cited by 94
Addressee Identification In Face-to-Face Meetings
Jovanović, N., den Akker, R. O., & Nijholt, A. (2006) · University of Twente Research Information · Cited by 72
Reaching the Global Community During Disasters: Findings From a Content Analysis of the Organizational Use of Twitter After the 2010 Haiti Earthquake
Gurman, T. A., & Ellenberger, N. (2015) · Journal of Health Communication · doi:10.1080/10810730.2015.1018566 · Cited by 72
The expression of social presence through the use of figurative language in a web-based learning environment
Delfino, M. M., & Manca, S. (2006) · Computers in Human Behavior · doi:10.1016/j.chb.2006.03.001 · Cited by 72
iElectrodes: A Comprehensive Open-Source Toolbox for Depth and Subdural Grid Electrode Localization
Blenkmann, A. O., Phillips, H. N., Princich, J. P., Rowe, J. B., Bekinschtein, T. A., Muravchik, C. H., & Kochen, S. S. (2017) · Frontiers in Neuroinformatics · doi:10.3389/fninf.2017.00014 · Cited by 77
The game frame and political efficacy: Beyond the spiral of cynicism
Pedersen, R. T. (2012) · European Journal of Communication · doi:10.1177/0267323112454089 · Cited by 71
Annotator subjectivity in harmony annotations of popular music
Koops, H. V., Haas, W., Burgoyne, J., Bransen, J., Kent-Muller, A., & Volk, A. (2019) · Journal of New Music Research · doi:10.1080/09298215.2019.1613436 · Cited by 71
Risks and Rewards of Crowdsourcing Marketplaces
Chandler, J., Paolacci, G., & Mueller, P. A. (2013) · Springer eBooks · doi:10.1007/978-1-4614-8806-4_30 · Cited by 73
“I just want to be skinny.”: A content analysis of tweets expressing eating disorder symptoms
Cavazos‐Rehg, P. A., Krauss, M. J., Costello, S. J., Kaiser, N., Cahn, E. S., Fitzsimmons‐Craft, E. E., & Wilfley, D. E. (2019) · PLoS ONE · doi:10.1371/journal.pone.0207506 · Cited by 70
Ontology-driven aspect-based sentiment analysis classification: An infodemiological case study regarding infectious diseases in Latin America
García-Díaz, J. A., Cánovas-García, M., & Valencia‐García, R. (2020) · Future Generation Computer Systems · doi:10.1016/j.future.2020.06.019 · Cited by 69
The Experience of Ethnic and Racial Group Membership Among Immigrant-Origin Adolescents
Tummala–Narra, P., & Sathasivam‐Rueckert, N. (2015) · Journal of Adolescent Research · doi:10.1177/0743558415592178 · Cited by 74
Thematic trends in event management research
Park, S. B., & Park, K. (2017) · International Journal of Contemporary Hospitality Management · doi:10.1108/ijchm-09-2015-0521 · Cited by 68
Social relations and presence of others predict bystander intervention: Evidence from violent incidents captured on CCTV
Liebst, L. S., Philpot, R., Bernasco, W., Dausel, K. L., Ejbye‐Ernst, P., Nicolaisen, M. H., & Lindegaard, M. R. (2019) · Aggressive Behavior · doi:10.1002/ab.21853 · Cited by 84
QADI as a New Method and Alternative to Kappa for Accuracy Assessment of Remote Sensing-Based Image Classification
Feizizadeh, B., Darabi, S., Blaschke, T., & Lakes, T. (2022) · Sensors · doi:10.3390/s22124506 · Cited by 73
Message framing as institutional maintenance: The National Collegiate Athletic Association’s institutional work of addressing legitimate threats
Nite, C. (2016) · Sport Management Review · doi:10.1016/j.smr.2016.10.005 · Cited by 68
Framing by the Flock: Collective Issue Definition and Advocacy Success
Junk, W. M., & Rasmussen, A. (2018) · Comparative Political Studies · doi:10.1177/0010414018784044 · Cited by 76
A Systematic Review of Responsibility Frames and Their Effects in the Health Context
Temmann, L. J., Wiedicke, A., Schaller, S., Scherr, S., & Reifegerste, D. (2021) · Journal of Health Communication · doi:10.1080/10810730.2021.2020381 · Cited by 67
A Call for Standardization in Content Analysis Reliability
Lombard, M. (2004) · Human Communication Research · doi:10.1111/j.1468-2958.2004.tb00739.x · Cited by 67
Nudity of female and male models in primetime TV advertising across seven countries
Nelson, M. R., & Paek, H. (2008) · International Journal of Advertising · doi:10.2501/s0265048708080281 · Cited by 67
Policies in Parallel? A Comparative Study of Journalistic AI Policies in 52 Global News Organisations
Becker, K.-B., Simon, F. M., & Crum, C. (2025) · Digital Journalism · doi:10.1080/21670811.2024.2431519 · Cited by 78
Hot weather, hot topic. Polarization and sceptical framing in the climate debate on Twitter
Moernaut, R., Mast, J., Temmerman, M., & Broersma, M. (2022) · Information, Communication & Society · doi:10.1080/1369118x.2020.1834600 · Cited by 65
From #Ferguson to #Ayotzinapa: Analyzing Differences in Domestic and Foreign Protest News Shared on Social Media
Brown, D. K., Harlow, S., García‐Perdomo, V., & Salaverría, R. (2018) · Mass Communication & Society · doi:10.1080/15205436.2018.1469773 · Cited by 65
When Do Politicians Use Populist Rhetoric? Populism as a Campaign Gamble
Dai, Y., & Kustov, A. (2022) · Political Communication · doi:10.1080/10584609.2022.2025505 · Cited by 65
Cohen’s kappa is a weighted average
Warrens, M. J. (2011) · Statistical Methodology · doi:10.1016/j.stamet.2011.06.002 · Cited by 65
Tourism destination image development: a lesson from Macau
Kong, W. H., du Cros, H., & Ong, C. (2015) · International Journal of Tourism Cities · doi:10.1108/ijtc-08-2014-0016 · Cited by 65
Strategic Planners in More Turbulent Times: The Changing Job Characteristics of Strategy Professionals, 1960–2003
Whittington, R., Yakis‐Douglas, B., Ahn, K., & Cailluet, L. (2016) · Long Range Planning · doi:10.1016/j.lrp.2015.12.021 · Cited by 65
Computerized Analysis of Speech and Language to Identify Psycholinguistic Correlates of Frontotemporal Lobar Degeneration
Pakhomov, S. V. S., Smith, G. E., Chacon, D., Feliciano, Y., Graff-Radford, N., Caselli, R., & Knopman, D. S. (2010) · Cognitive and Behavioral Neurology · doi:10.1097/wnn.0b013e3181c5dde3 · Cited by 64
“…annotation of linguistic features in spontaneous speech samples is a very labor intensive process and is subject to variable agreement among the annotators particularly in content analysis involving multiple semantic categories(Artstein & Poesio, 2008; Krippendorff, 2004; Poesio & Vieira, 1998).”
Background

Authors: Krippendorff, K.
Corresponding author: Krippendorff, K.
ORCID: Krippendorff, K.
Venue: Human Communication Research
Publisher: Oxford University Press (OUP)
Published: 2004-07-01
Volume: 30
Issue: 3
Pages: 411-433
ISSN: 0360-3989 (ISSN-L)
1468-2958
DOI: 10.1093/hcr/30.3.411
OpenAlex: W4243989635
Cited by: 1,027
References: 0
Citations by year: 2026: 42 · 2025: 100 · 2024: 108 · 2023: 108 · 2022: 124 · 2021: 135 · 2020: 105 · 2019: 57 · 2018: 44 · 2017: 29 · 2016: 41 · 2015: 25 · 2014: 26 · 2013: 21 · 2012: 15
FWCI: 5.93
Type: journal article
Language: en
Open access: green
repository.upenn.edu
Licence: doi.wiley.com/10.1002/tdm_license_1 (tdm, 4079 d)
Locations: Human Communication Research (publishedVersion)
ScholarlyCommons (University of Pennsylvania) (submittedVersion)
Topics: Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Reliability and Agreement in Measurement
Social Sciences › General Social Sciences › Computational and Text Analysis Methods
Social Sciences › Decision Sciences › Management Science and Operations Research › Psychometric Methodologies and Testing
Keywords: inter-rater reliability, reliability coefficients, interobserver agreement, reporting reliability

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Reliability in content analysis: some common misconceptions and recommendations.

Abstract

Overview

Integrity signals

Verification

Claims the paper makes

Claims other papers make about this one

Self-check

Citations

Cites

Cited by

Related

Metadata

Record

About these checks