Content Analysis in Mass Communication: Assessment and Reporting of Intercoder Reliability

Lombard, Matthew; Snyder-Duch, Jennifer; Bracken, Cheryl Campanella

doi:10.1111/j.1468-2958.2002.tb00826.x

Cited by 3,068Reliability and Agreement in MeasurementMedia Studies and CommunicationComputational and Text Analysis Methods

Content Analysis in Mass Communication: Assessment and Reporting of Intercoder Reliability

Lombard, Matthew, Snyder-Duch, Jennifer, Bracken, Cheryl Campanella·2002·Human Communication Research

DOI 10.1111/j.1468-2958.2002.tb00826.x

Open this paper in CoChat — free account

Abstract

As a method specifically intended for the study of messages, content analysis is fundamental to mass communication research. Intercoder reliability, more specifically termed intercoder agreement, is a measure of the extent to which independent judges make the same coding decisions in evaluating the characteristics of messages, and is at the heart of this method. Yet there are few standard and accessible guidelines available regarding the appropriate procedures to use to assess and report intercoder reliability, or software tools to calculate it. As a result, it seems likely that there is little consistency in how this critical element of content analysis is assessed and reported in published mass communication studies. Following a review of relevant concepts, indices, and tools, a content analysis of 200 studies utilizing content analysis published in the communication literature between 1994 and 1998 is used to characterize practices in the field. The results demonstrate that mass communication researchers often fail to assess (or at least report) intercoder reliability and often rely on percent agreement, an overly liberal index. Based on the review and these results, concrete guidelines are offered regarding procedures for assessment and reporting of this important aspect of content analysis.

Integrity signals

Registry

Crossref · 10.1111/j.1468-2958.2002.tb00826.x

Cites

42 of 42 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Claims the paper makes

This paper’s full text isn’t openly available, so its claims and self-check weren’t run. The registry and citation checks still apply.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Cohen's kappa was calculated for the level codes and form codes in line with recommended approaches.
“In line with recommended approaches [89], we calculated Cohen's kappa for the level codes and form codes.”
From Virtual Museums as Meaning-Modeling Systems in Digital Heritage (2025) · cites this paper as [89]
After clarifying the criteria, the evaluators similarly classified 90.3% of the studies, a satisfactory percentage.
“After clarifying the criteria, the evaluators similarly classified 90.3% of the studies, a satisfactory percentage (Lombard et al., 2002).”
From Online and Mobile Interventions for Problem Gambling, Alcohol, and Drugs: A Systematic Review (2017) · cites this paper as Lombard et al., 2002

A Coefficient of Agreement for Nominal Scales
Cohen, J. (1960) · Educational and Psychological Measurement · doi:10.1177/001316446002000104 · Cited by 42,398 · see citing works
Content Analysis: An Introduction to its Methodology.
Shelley, M., & Krippendorff, K. (1984) · Journal of the American Statistical Association · doi:10.2307/2288384 · Cited by 24,305 · see citing works
The Content Analysis Guidebook
Neuendorf, K. A. (2017) · doi:10.4135/9781071802878 · Cited by 9,286 · cited 10× · see citing works
“), but this technique has also been questioned (Neuendorf, 2002). With the coding data in hand, the researcher calculates and reports one or more indices of reliability. Popping (1988) identified 39 different “agreement indices” for coding nominal categories, which excludes several techniques for ratio and interval le…”
BackgroundMethodInfluential
A concordance correlation coefficient to evaluate reproducibility.
Lin, L. I.-K. (1989) · Biometrics · doi:10.2307/2532051 · Cited by 8,785 · see citing works
Weighted kappa: nominal scale agreement provision for scaled disagreement or partial credit
Cohen, J. (1968) · Psychological Bulletin · doi:10.1037/h0026256 · Cited by 8,736 · see citing works
Measuring nominal scale agreement among many raters.
Fleiss, J. L. (1971) · Psychological Bulletin · doi:10.1037/h0031619 · Cited by 8,733 · see citing works
Content Analysis for the Social Sciences and Humanities.
Parker, E. B., & Holsti, O. R. (1970) · American Sociological Review · doi:10.2307/2093233 · Cited by 4,582 · see citing works
Content Analysis in Communication Research.
Pool, I. D. S., & Berelson, B. (1952) · American Sociological Review · doi:10.2307/2088029 · Cited by 3,119 · see citing works
Analyzing media messages using quantitative content analysis in research
Riffe, D., Lacy, S. R., Watson, B. R., & Fico, F. G. (2005) · doi:10.4324/9780429464287 · Cited by 2,236 · see citing works
Content Analysis for the Social Sciences and Humanities
Parker, S. (1970) · Sociology · doi:10.1177/003803857000400231 · Cited by 2,187 · see citing works
Handbook of Research Methods in Social and Personality Psychology
West, T., Reis, H. T., Judd, C. M., Banaji, M. R., Crandall, C., Giner-Sorolla, R., Biernat, M., Fiedler, K., Ermark, F., Tepper, S. J., et al. (2024) · Cambridge University Press eBooks · doi:10.1017/9781009170123 · Cited by 2,133 · see citing works
Reliability of Content Analysis: The Case of Nominal Scale Coding
Scott, W. A. (1955) · Public Opinion Quarterly · doi:10.1086/266577 · Cited by 1,853 · see citing works
Rethinking validity and reliability in content analysis
Potter, W. J., & Levine‐Donnerstein, D. (1999) · Journal of Applied Communication Research · doi:10.1080/00909889909365539 · Cited by 1,662 · see citing works
Coefficient Kappa: Some Uses, Misuses, and Alternatives
Brennan, R. L., & Prediger, D. J. (1981) · Educational and Psychological Measurement · doi:10.1177/001316448104100307 · Cited by 1,590 · see citing works
Reliability of Nominal Data Based on Qualitative Judgments
Perreault, W. D., & Leigh, L. E. (1989) · Journal of Marketing Research · doi:10.2307/3172601 · Cited by 1,575 · see citing works
Content-Analysis Research: An Examination of Applications with Directives for Improving Research Reliability and Objectivity
Kolbe, R. H., & Burnett, M. S. (1991) · Journal of Consumer Research · doi:10.1086/209256 · Cited by 1,516 · see citing works
Content Analysis in Communication Research
Burns, T., & Berelson, B. (1953) · British Journal of Sociology · doi:10.2307/587175 · Cited by 1,433 · see citing works
Handbook of Research Methods in Social and Personality Psychology: Author Index
Reis, H. T., & Judd, C. M. (2013) · Cambridge University Press eBooks · doi:10.1017/cbo9780511996481 · Cited by 1,080 · see citing works
Beyond kappa: a review of interrater agreement measures
Banerjee, M., Capozzoli, M., McSweeney, L., & Sinha, D. (1999) · Canadian Journal of Statistics · doi:10.2307/3315487 · Cited by 976 · see citing works
Reliability Measures for Qualitative Data: Theory and Implications
Rust, R. T., & Cooil, B. (1994) · Journal of Marketing Research · doi:10.1177/002224379403100101 · Cited by 955 · see citing works
Investigating Communication: An Introduction to Research Methods
Frey, L. R., Botan, C. H., & Kreps, G. L. (1999) · Cited by 947 · see citing works
Interrater reliability and agreement of subjective judgments.
Tinsley, H. E. A., & Weiss, D. J. (1975) · Journal of Counseling Psychology · doi:10.1037/h0076640 · Cited by 855 · see citing works
Handbook of Applied Multivariate Statistics and Mathematical Modeling
Tinsley, H. E. A., & Brown, S. D. (2000) · Elsevier eBooks · doi:10.1016/b978-0-12-691360-6.x5000-9 · Cited by 842 · see citing works
Interrater Reliability and Agreement
Tinsley, H. E. A., & Weiss, D. J. (2000) · Elsevier eBooks · doi:10.1016/b978-012691360-6/50005-7 · Cited by 401 · see citing works
Communications Through Limited Response Questioning
Bennett, E., Alpert, R., & Goldstein, A. C. (1954) · Public Opinion Quarterly · doi:10.1086/266520 · Cited by 254 · see citing works
Sampling Error and Selecting Intercoder Reliability Samples for Nominal Content Categories
Lacy, S. R., & Riffe, D. (1996) · Journalism & Mass Communication Quarterly · doi:10.1177/107769909607300414 · Cited by 243 · see citing works
A Content Analysis of Content Analyses: Twenty-Five Years of Journalism Quarterly
Riffe, D., & Freitag, A. R. (1997) · Journalism & Mass Communication Quarterly · doi:10.1177/107769909707400306 · Cited by 206 · see citing works
Judgment of counseling process: Reliability, agreement, and error.
Lawlis, G. F., & Lu, E. (1972) · Psychological Bulletin · doi:10.1037/h0032935 · Cited by 174 · see citing works
Intercoder Reliability Estimation Approaches in Marketing: A Generalizability Theory Framework for Quantitative Data
Hughes, M. A., & Garrett, D. E. (1990) · Journal of Marketing Research · doi:10.1177/002224379002700206 · Cited by 172 · see citing works
Behavioral observation and coding.
Bakeman, R. A. (2000) · Cited by 154 · see citing works
Coefficients of Agreement
Dewey, M. E. (1983) · The British Journal of Psychiatry · doi:10.1192/bjp.143.5.487 · Cited by 100 · see citing works
Research Methods in the Social Sciences
Ellis, L. (1994) · Medical Entomology and Zoology · Cited by 93 · see citing works
Mass Communication Research: Contemporary Methods and Applications
Singletary, M. W. (1993) · Medical Entomology and Zoology · Cited by 92 · see citing works
On Agreement Indices for Nominal Data
Popping, R. (1988) · Palgrave Macmillan UK eBooks · doi:10.1007/978-1-349-19051-5_6 · Cited by 89 · see citing works
A Content Analysis of Content Analyses: Twenty-Five Years of Journalism Quarterly
Riffe, D., & Freitag, A. R. (1997) · Journalism & Mass Communication Quarterly · doi:10.1177/107769909707400414 · Cited by 89 · see citing works
Generalization of Scott's Index of Intercoder Agreement
Craig, R. T. (1981) · Public Opinion Quarterly · doi:10.1086/268657 · Cited by 73 · see citing works
A SAS MACRO for Calculating Intercoder Agreement in Content Analysis
Kang, N., Kara, A., Laskey, H. A., & Seaton, F. B. (1993) · Journal of Advertising · doi:10.1080/00913367.1993.10673400 · Cited by 34 · see citing works
Effects of the use of percentage agreement on behavioral observation reliabilities: A reassessment
Suen, H. K., & Lee, P. S. C. (1985) · Journal of Psychopathology and Behavioral Assessment · doi:10.1007/bf00960754 · Cited by 33 · see citing works
Does Hollywood Hate Business or Money?
Lichter, S. R., Lichter, L. S., & Amundson, A. (1997) · Journal of Communication · doi:10.1111/j.1460-2466.1997.tb02693.x · Cited by 18 · see citing works
AGREE, a package for computing nominal scale agreement
Popping, R. (1984) · Computational Statistics & Data Analysis · doi:10.1016/0167-9473(84)90008-2 · Cited by 13 · see citing works
Ethical Advertising Research Standards: Three Case Studies
Martin, C. R. (1994) · Journal of Advertising · doi:10.1080/00913367.1994.10673447 · Cited by 10 · see citing works
Computer Studies Measuring the Joint Agreement between Multiple Raters and a Standard
Berry, K. J., & Mielke, P. W. (1997) · Educational and Psychological Measurement · doi:10.1177/0013164497057003013 · Cited by 6 · see citing works

Flagged first, then by citations

Sorted by citations

Answering the Call for a Standard Reliability Measure for Coding Data
Hayes, A. F., & Krippendorff, K. (2007) · Communication Methods and Measures · doi:10.1080/19312450709336664 · Cited by 4,225 · see citing works
Intercoder Reliability in Qualitative Research: Debates and Practical Guidelines
O’Connor, C., & Joffe, H. (2020) · International Journal of Qualitative Methods · doi:10.1177/1609406919899220 · Cited by 3,871 · see citing works
“Studies that influence important medical, policy, or financial decisions arguably merit a higher ICR threshold than exploratory academic research (Hruschka et al., 2004; Lombard et al., 2002).”
BackgroundMethodInfluential
Coding In-depth Semistructured Interviews: Problems of Unitization and Intercoder Reliability and Agreement
Campbell, J. L., Quincy, C., Osserman, J., & Pedersen, O. K. (2013) · Sociological Methods & Research · doi:10.1177/0049124113500475 · Cited by 2,974 · see citing works
Reliability in content analysis: some common misconceptions and recommendations.
Krippendorff, K. (2004) · Human Communication Research · doi:10.1093/hcr/30.3.411 · Cited by 1,027 · see citing works
Structural Topic Models for Open‐Ended Survey Responses
Roberts, M. E., Stewart, B., Tingley, D. H., Lucas, C., Leder-Luis, J., Gadarian, S. K., Albertson, B. L., & Rand, D. G. (2014) · American Journal of Political Science · doi:10.1111/ajps.12103 · Cited by 1,969 · see citing works
Conducting content‐analysis based literature reviews in supply chain management
Seuring, S. A., & Gold, S. (2012) · Supply Chain Management An International Journal · doi:10.1108/13598541211258609 · Cited by 1,276 · see citing works
Reliability and Inter-rater Reliability in Qualitative Research: Norms and Guidelines for CSCW and HCI Practice
McDonald, N., Schoenebeck, S., & Forte, A. (2019) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3359174 · Cited by 1,206 · see citing works
Content analysis schemes to analyze transcripts of online asynchronous discussion groups: A review
Wever, B. D., Schellens, T., Valcke, M., & Keer, H. V. (2005) · Computers & Education · doi:10.1016/j.compedu.2005.04.005 · Cited by 940 · see citing works
Content Analysis
Drisko, J. W., & Maschi, T. M. (2015) · doi:10.1093/acprof:oso/9780190215491.001.0001 · Cited by 881 · see citing works
A new tool to assess treatment fidelity and evaluation of treatment fidelity across 10 years of health behavior research
Borrelli, B., Sepinwall, D., Ernst, D., Bellg, A. J., Czajkowski, S., Breger, R., DeFrancesco, C., Levesque, C., Sharp, D. L., Ogedegbe, G., et al. (2005) · Journal of Consulting and Clinical Psychology · doi:10.1037/0022-006x.73.5.852 · Cited by 803 · see citing works
Reliability in Coding Open-Ended Data: Lessons Learned from HIV Behavioral Research
Hruschka, D. J., Schwartz, D., St.John, D. C., Picone-Decaro, E., Jenkins, R. A., & Carey, J. W. (2004) · Field Methods · doi:10.1177/1525822x04266540 · Cited by 829 · see citing works
Interrater Reliability in Systematic Review Methodology: Exploring Variation in Coder Decision-Making
Belur, J., Tompson, L., Thornton, A., & Simon, M. (2021) · Sociological Methods & Research · doi:10.1177/0049124118799372 · Cited by 753 · see citing works
A systematic approach to conducting review studies: An assessment of content analysis in 25years of IB research
Gaur, A. S., & Kumar, M. (2017) · Journal of World Business · doi:10.1016/j.jwb.2017.11.003 · Cited by 648 · see citing works
Large language models in education: a focus on the complementary relationship between human teachers and ChatGPT
Jeon, J., & Lee, S. (2023) · Education and Information Technologies · doi:10.1007/s10639-023-11834-1 · Cited by 696 · see citing works
Future work selves: How salient hoped-for identities motivate proactive career behaviors.
Strauss, K., Griffin, M. A., & Parker, S. K. (2011) · Journal of Applied Psychology · doi:10.1037/a0026423 · Cited by 686 · see citing works
Preparing, conducting, and analyzing Delphi surveys: Cross-disciplinary practices, new directions, and advancements
Beiderbeck, D., Frevel, N., von der Gracht, H. A., Schmidt, S. L., & Schweitzer, V. M. (2021) · MethodsX · doi:10.1016/j.mex.2021.101401 · Cited by 642 · see citing works
Sample Selection in Systematic Literature Reviews of Management Research
Hiebl, M. R. W. (2023) · Organizational Research Methods · doi:10.1177/1094428120986851 · Cited by 635 · see citing works
Cognitive Processes of Opportunity Recognition: The Role of Structural Alignment
Grégoire, D. A., Barr, P. S., & Shepherd, D. A. (2010) · Organization Science · doi:10.1287/orsc.1090.0462 · Cited by 582 · see citing works
Like, comment, and share on Facebook: How each behavior differs from the other
Kim, C., & Yang, S.-U. (2017) · Public Relations Review · doi:10.1016/j.pubrev.2017.02.006 · Cited by 550 · see citing works
Process guidelines for establishing Intercoder Reliability in qualitative studies
MacPhail, C., Khoza, N., Abler, L., & Ranganathan, M. (2016) · Qualitative Research · doi:10.1177/1468794115577012 · Cited by 534 · see citing works
Media Content Analysis: Its Uses, Benefits and Best Practice Methodology
Macnamara, J. (2005) · UTS ePRESS (University of Technology Sydney) · Cited by 519 · see citing works
Knowledge-based innovation: emergence and embedding of new practice areas in management consulting firms
Anand, N., Gardner, H. K., & Morris, T. (2007) · Academy of Management Journal · doi:10.5465/amj.2007.24634457 · Cited by 509 · see citing works
A team-based approach to open coding: considerations for creating intercoder consensus
Cascio, M. A., Lee, E., Vaudrin, N., & Freedman, D. A. (2019) · Field Methods · doi:10.1177/1525822x19838237 · Cited by 492 · see citing works
A standardized approach to qualitative content analysis of focus group discussions from different countries
Moretti, F., van Vliet, L., Bensing, J. M., Deledda, G., Mazzi, M. A., Rimondini, M., Zimmermann, C., & Fletcher, I. J. (2011) · Patient Education and Counseling · doi:10.1016/j.pec.2011.01.005 · Cited by 453 · see citing works
Qualitative Content Analysis
Forman, J., & Damschroder, L. (2007) · Advances in Bioethics · doi:10.1016/s1479-3709(07)11003-7 · Cited by 808 · see citing works
A River Runs Through It: A Multiple Streams Meta‐Review
Jones, M. D., Peterson, H. L., Pierce, J. J., Herweg, N., Bernal, A., Raney, H. L., & Zahariadis, N. (2016) · Policy Studies Journal · doi:10.1111/psj.12115 · Cited by 399 · see citing works
Environmental Education Research
(2010) · Environmental Education Research · doi:10.1080/13504622.2010.529990 · Cited by 443 · see citing works
A reconceptualization of fear of failure in entrepreneurship
Cacciotti, G., Hayton, J. C., Mitchell, J., & Giazitzoglu, A. (2016) · Journal of Business Venturing · doi:10.1016/j.jbusvent.2016.02.002 · Cited by 434 · see citing works
Evaluating the enablers and barriers for successful implementation of sustainable business practice in ‘lean’ SMEs
Caldera, S., Desha, C., & Dawes, L. A. (2019) · Journal of Cleaner Production · doi:10.1016/j.jclepro.2019.01.239 · Cited by 432 · see citing works
Entrepreneurship Education Research Revisited: The Case of Higher Education
Béchard, J., & Grégoire, D. A. (2005) · Academy of Management Learning and Education · doi:10.5465/amle.2005.16132536 · Cited by 423 · see citing works
Searching for the Good Plan
Berke, P. R., & Godschalk, D. R. (2008) · Journal of Planning Literature · doi:10.1177/0885412208327014 · Cited by 407 · see citing works
Defining service innovation: A review and synthesis
Witell, L., Snyder, H., Gustafsson, A., Fombelle, P. W., & Kristensson, P. (2016) · Journal of Business Research · doi:10.1016/j.jbusres.2015.12.055 · Cited by 418 · see citing works
Application of digital technologies for sustainable product management in a circular economy: A review
Rusch, M., Schöggl, J., & Baumgartner, R. J. (2022) · Business Strategy and the Environment · doi:10.1002/bse.3099 · Cited by 429 · see citing works
SHIFTing artificial intelligence to be responsible in healthcare: a systematic review
Siala, H., & Wang, Y. (2022) · Social Science & Medicine · doi:10.1016/j.socscimed.2022.114782 · Cited by 358 · see citing works
From Words to Numbers: How to Transform Qualitative Data into Meaningful Quantitative Results
Srnka, K. J., & Koeszegi, S. T. (2007) · Schmalenbach Business Review · doi:10.1007/bf03396741 · Cited by 353 · see citing works
Measuring consumer perception of social media marketing activities in e-commerce industry: Scale development & validation
Yadav, M. K., & Rahman, Z. (2017) · Telematics and Informatics · doi:10.1016/j.tele.2017.06.001 · Cited by 352 · see citing works
Agency in Action: Entrepreneurs' Networking Style and Initiation of Economic Exchange
Vissa, B. (2010) · Organization Science · doi:10.1287/orsc.1100.0567 · Cited by 350 · see citing works
The Accuracy of Citizen Science Data: A Quantitative Review
Aceves‐Bueno, E., Adeleye, A. S., Feraud, M., Huang, Y., Tao, M., Yang, Y., & Anderson, S. E. (2017) · Bulletin of the Ecological Society of America · doi:10.1002/bes2.1336 · Cited by 341 · see citing works
Automated content analysis for construction safety: A natural language processing system to extract precursors and outcomes from unstructured injury reports
Tixier, A. J. ‐., Hallowell, M. R., Rajagopalan, B., & Bowman, D. (2015) · Automation in Construction · doi:10.1016/j.autcon.2015.11.001 · Cited by 338 · see citing works
Developing and Testing a Theoretical Framework for Computer‐Mediated Transparency of Local Governments
Grimmelikhuijsen, S., & Welch, E. W. (2012) · Public Administration Review · doi:10.1111/j.1540-6210.2011.02532.x · Cited by 343 · see citing works
Civility 2.0: a comparative analysis of incivility in online political discussion
Rowe, I. (2014) · Information Communication & Society · doi:10.1080/1369118x.2014.940365 · Cited by 338 · see citing works
Drones for parcel and passenger transportation: A literature review
Kellermann, R., Biehle, T., & Fischer, L. (2020) · Transportation Research Interdisciplinary Perspectives · doi:10.1016/j.trip.2019.100088 · Cited by 324 · see citing works
The impacts of an invasive species citizen science training program on participant attitudes, behavior, and science literacy
Crall, A., Jordan, R., Holfelder, K. A., Newman, G., Graham, J., & Waller, D. M. (2012) · Public Understanding of Science · doi:10.1177/0963662511434894 · Cited by 313 · see citing works
An investigation of learners' collaborative knowledge construction performances and behavior patterns in an augmented reality simulation system
Lin, T., Duh, H. B., Li, N., Wang, H.-Y., & Tsai, C. (2013) · Computers & Education · doi:10.1016/j.compedu.2013.05.011 · Cited by 311 · see citing works
Branding co-creation with members of online brand communities
Hajli, N., Shanmugam, M., Papagiannidis, S., Zahay, D. L., & Richard, M. (2016) · Journal of Business Research · doi:10.1016/j.jbusres.2016.08.026 · Cited by 312 · see citing works
How to conduct a Narrative Policy Framework study
Shanahan, E. A., Jones, M. D., & McBeth, M. K. (2018) · The Social Science Journal · doi:10.1016/j.soscij.2017.12.002 · Cited by 301 · see citing works
Cognitive flexibility and adaptive decision‐making: Evidence from a laboratory study of expert decision makers
Martinez, D. L., & Brusoni, S. (2018) · Strategic Management Journal · doi:10.1002/smj.2774 · Cited by 309 · see citing works
The future and social impact of Big Data Analytics in Supply Chain Management: Results from a Delphi study
Roßmann, B., Canzaniello, A., von der Gracht, H. A., & Hartmann, E. (2017) · Technological Forecasting and Social Change · doi:10.1016/j.techfore.2017.10.005 · Cited by 282 · see citing works
The value of social media for innovation: A capability perspective
Muninger, M.-I., Hammedi, W., & Mahr, D. (2019) · Journal of Business Research · doi:10.1016/j.jbusres.2018.10.012 · Cited by 277 · see citing works
Analysis of Five High School Biology Textbooks Used in the United States for Inclusion of the Nature of Science
Chiappetta, E. L., & Fillman, D. A. (2007) · International Journal of Science Education · doi:10.1080/09500690601159407 · Cited by 272 · see citing works
Content Analysis—A Methodological Primer for Gender Research
Neuendorf, K. A. (2010) · Sex Roles · doi:10.1007/s11199-010-9893-0 · Cited by 271 · see citing works
The Multiple Facets of Influence
Dubois, E., & Gaffney, D. (2014) · American Behavioral Scientist · doi:10.1177/0002764214527088 · Cited by 263 · see citing works
Travel photos : motivations, image dimensions, and affective qualities of places
Pan, S., Lee, J.-S., & Tsai, H. (2014) · Tourism Management · doi:10.1016/j.tourman.2013.05.007 · Cited by 242 · see citing works
How organizations framed the 2009 H1N1 pandemic via social and traditional media: Implications for U.S. health communicators
Liu, B. F., & Kim, S. (2011) · Public Relations Review · doi:10.1016/j.pubrev.2011.03.005 · Cited by 253 · see citing works
Is partially automated driving a bad idea? Observations from an on-road study
Banks, V. A., Eriksson, A., O'Donoghue, J., & Stanton, N. A. (2017) · Applied Ergonomics · doi:10.1016/j.apergo.2017.11.010 · Cited by 244 · see citing works
Tweeting as Health Communication: Health Organizations’ Use of Twitter for Health Promotion and Public Engagement
Park, H., Reber, B. H., & Chon, M. (2015) · Journal of Health Communication · doi:10.1080/10810730.2015.1058435 · Cited by 243 · see citing works
Measures of fidelity of delivery of, and engagement with, complex, face‐to‐face health behaviour change interventions: A systematic review of measure quality
Walton, H., Spector, A. E., Tombor, I., & Michie, S. (2017) · British Journal of Health Psychology · doi:10.1111/bjhp.12260 · Cited by 244 · see citing works
Mining Social Media Data for Understanding Students’ Learning Experiences
Chen, X., Vorvoreanu, M., & Madhavan, K. P. C. (2014) · IEEE Transactions on Learning Technologies · doi:10.1109/tlt.2013.2296520 · Cited by 242 · see citing works
Social Construction and Policy Design: A Review of Past Applications
Pierce, J. J., Siddiki, S., Jones, M. D., Schumacher, K., Pattison, A., & Peterson, H. L. (2014) · Policy Studies Journal · doi:10.1111/psj.12040 · Cited by 238 · see citing works
Beyond a coefficient: an interactive process for achieving inter-rater consistency in qualitative coding
Hemmler, V. L., Kenney, A. W., Langley, S. D., Callahan, C. M., Gubbins, E. J., & Holder, S. (2020) · Qualitative Research · doi:10.1177/1468794120976072 · Cited by 241 · see citing works
Gender differences in recommendation letters for postdoctoral fellowships in geoscience
Dutt, K., Pfaff, D. L., Bernstein, A. F., Dillard, J. S., & Block, C. J. (2016) · Nature Geoscience · doi:10.1038/ngeo2819 · Cited by 232 · see citing works
Big Social Data Analytics in Journalism and Mass Communication
Гуо, Л., Vargo, C. J., Pan, Z., Ding, W., & Ishwar, P. (2016) · Journalism & Mass Communication Quarterly · doi:10.1177/1077699016639231 · Cited by 231 · see citing works
Towards integrating construction risk management and stakeholder management: A systematic literature review and future research agendas
Xia, N., Zou, P. X. W., Griffin, M. A., Wang, X., & Zhong, R. (2018) · International Journal of Project Management · doi:10.1016/j.ijproman.2018.03.006 · Cited by 230 · see citing works
Learning presence: Additional research on a new conceptual element within the Community of Inquiry (CoI) framework
Shea, P., Hayes, S., Smith, S. U., Vickers, J. C., Bidjerano, T., Pickett, A., Gozza‐Cohen, M., Wilde, J., & Jian, S.-B. (2011) · The Internet and Higher Education · doi:10.1016/j.iheduc.2011.08.002 · Cited by 229 · see citing works
How the Media Shape Perceptions of Right-Wing Populist Leaders
Bos, L., van der Brug, W., & de Vreese, C. H. (2011) · Political Communication · doi:10.1080/10584609.2011.564605 · Cited by 230 · see citing works
Expediting the Analysis of Qualitative Data in Evaluation
Neal, J. W., Neal, Z. P., VanDyke, E., & Kornbluh, M. (2014) · American Journal of Evaluation · doi:10.1177/1098214014536601 · Cited by 227 · see citing works
What to Do (and Not to Do) with the Comparative Manifestos Project Data
Gemenis, K. (2013) · Political Studies · doi:10.1111/1467-9248.12015 · Cited by 226 · see citing works
Content analysis of e-petitions with topic modeling: How to train and evaluate LDA models?
Hagen, L. (2018) · Information Processing & Management · doi:10.1016/j.ipm.2018.05.006 · Cited by 233 · see citing works
Avoiding Health Information
Barbour, J. B., Rintamaki, L. S., Ramsey, J. A., & Brashers, D. E. (2011) · Journal of Health Communication · doi:10.1080/10810730.2011.585691 · Cited by 222 · see citing works
Plan Quality Evaluation 1994–2012
Lyles, W., & Stevens, M. R. (2014) · Journal of Planning Education and Research · doi:10.1177/0739456x14549752 · Cited by 219 · see citing works
Corporate social responsibility communication of Chinese and global corporations in China
Tang, L., & Li, H. (2009) · Public Relations Review · doi:10.1016/j.pubrev.2009.05.016 · Cited by 218 · see citing works
Inter-Rater Reliability Methods in Qualitative Case Study Research
Cole, R. (2023) · Sociological Methods & Research · doi:10.1177/00491241231156971 · Cited by 294 · see citing works
What's Being Sold and To What End?: A Content Analysis of College Viewbooks
Hartley, M., & Morphew, C. C. (2008) · The Journal of Higher Education · doi:10.1353/jhe.0.0025 · Cited by 212 · see citing works
Goal-oriented requirements engineering: an extended systematic mapping study
Horkoff, J., Aydemir, F. B., Cardoso, E. C. S., Li, T., Maté, A., Paja, E., Salnitri, M., Piras, L., Mylopoulos, J., & Giorgini, P. (2017) · Requirements Engineering · doi:10.1007/s00766-017-0280-z · Cited by 212 · see citing works
Users as innovators in developing countries: The global sources of innovation and diffusion in mobile banking services
van der Boor, P., Oliveira, P., & Veloso, F. M. (2014) · Research Policy · doi:10.1016/j.respol.2014.05.003 · Cited by 209 · see citing works
A linguistic ontology of space for natural language processing
Bateman, J. A., Hois, J., Ross, R., & Tenbrink, T. (2010) · Artificial Intelligence · doi:10.1016/j.artint.2010.05.008 · Cited by 208 · see citing works
Who are you and what do you value? Investigating the role of personality traits and customer-perceived value in online customer engagement
Marbach, J., Lages, C. R., & Nunan, D. (2016) · Journal of Marketing Management · doi:10.1080/0267257x.2015.1128472 · Cited by 207 · see citing works
The Consumer Affinity Construct: Conceptualization, Qualitative Investigation, and Research Agenda
Oberecker, E. M., Riefler, P., & Diamantopoulos, A. (2008) · Journal of International Marketing · doi:10.1509/jimk.16.3.23 · Cited by 204 · see citing works
On pins and needles: How vaccines are portrayed on Pinterest
Guidry, J. P. D., Carlyle, K. E., Messner, M., & Jin, Y. (2015) · Vaccine · doi:10.1016/j.vaccine.2015.08.064 · Cited by 202 · see citing works
The Struggle Over Borders
de Wilde, P., Junk, W. M., & Palmtag, T. (2019) · Cambridge University Press eBooks · doi:10.1017/9781108652698 · Cited by 199 · see citing works
Measuring and Reporting Intercoder Reliability in Plan Quality Evaluation Research
Stevens, M. R., Lyles, W., & Berke, P. R. (2014) · Journal of Planning Education and Research · doi:10.1177/0739456x13513614 · Cited by 197 · see citing works
There and Back Again: A Tale of the Advocacy Coalition Framework
Pierce, J. J., Peterson, H. L., Jones, M. D., Garrard, S. P., & Vu, T. (2017) · Policy Studies Journal · doi:10.1111/psj.12197 · Cited by 198 · see citing works
Content analysis in tourism research
Camprubí, R., & Coromina, L. (2016) · Tourism Management Perspectives · doi:10.1016/j.tmp.2016.03.002 · Cited by 197 · see citing works
Judicial Independence and Political Uncertainty: How the Risk of Override Affects the Court of Justice of the EU
Larsson, O., & Naurin, D. (2016) · International Organization · doi:10.1017/s0020818316000047 · Cited by 195 · see citing works
Australian and U.S. News Media Portrayal of Sharks and Their Conservation
Muter, B. A., Gore, M. L., Gledhill, K. S., Lamont, C., & Huveneers, C. P. M. (2012) · Conservation Biology · doi:10.1111/j.1523-1739.2012.01952.x · Cited by 194 · see citing works
What Creates Interactivity in Online News Discussions? An Exploratory Analysis of Discussion Factors in User Comments on News Items
Ziegele, M., Breiner, T., & Quiring, O. (2014) · Journal of Communication · doi:10.1111/jcom.12123 · Cited by 194 · see citing works
Corporate Social Performance, Firm Size, and Organizational Visibility: Distinct and Joint Effects on Voluntary Sustainability Reporting
Schreck, P., & Raithel, S. (2015) · Business & Society · doi:10.1177/0007650315613120 · Cited by 193 · see citing works
Revisiting the Epistemology of Fact-Checking
Amazeen, M. A. (2015) · Critical Review · doi:10.1080/08913811.2014.993890 · Cited by 192 · see citing works
Teachers' emotions and emotion regulation strategies: Self- and students' perceptions
Jiang, J., Vauras, M., Volet, S., & Wang, Y. (2015) · Teaching and Teacher Education · doi:10.1016/j.tate.2015.11.008 · Cited by 192 · see citing works
Knowledge Exchange and Symbolic Action in Social Media-Enabled Electronic Networks of Practice: A Multilevel Perspective on Knowledge Seekers and Contributors1
Beck, R., Pahlke, I., & Seebach, C. (2014) · MIS Quarterly · doi:10.25300/misq/2014/38.4.14 · Cited by 185 · see citing works
The role of academic motivation in Computer-Supported Collaborative Learning
Rienties, B., Tempelaar, D. T., den Bossche, P. V., Gijselaers, W., & Segers, M. S. R. (2009) · Computers in Human Behavior · doi:10.1016/j.chb.2009.05.012 · Cited by 186 · see citing works
The relationship between sustainability and customer satisfaction in hospitality: An explorative investigation using eWOM as a data source
Gerdt, S., Wagner, E., & Schewe, G. (2019) · Tourism Management · doi:10.1016/j.tourman.2019.02.010 · Cited by 185 · see citing works
Understanding a Period of Policy Change: The Case of Hydraulic Fracturing Disclosure Policy in C olorado
Heikkila, T., Pierce, J. J., Gallaher, S., Kagan, J. A., Crow, D. A., & Weible, C. M. (2014) · Review of Policy Research · doi:10.1111/ropr.12058 · Cited by 184 · see citing works
Six decades of project management research: Thematic trends and future opportunities
Padalkar, M., & Gopinath, S. (2016) · International Journal of Project Management · doi:10.1016/j.ijproman.2016.06.006 · Cited by 183 · see citing works
Assessing the Reliability and Validity of Expert Interviews
Dorussen, H., Lenz, H., & Blavoukos, S. (2005) · European Union Politics · doi:10.1177/1465116505054835 · Cited by 186 · see citing works
An experiential model of consumer engagement in social media
Tafesse, W. (2016) · Journal of Product & Brand Management · doi:10.1108/jpbm-05-2015-0879 · Cited by 200 · see citing works
Studies of user-generated content: A systematic review
Naab, T. K., & Sehl, A. (2016) · Journalism · doi:10.1177/1464884916673557 · Cited by 180 · see citing works
Issue Importance as a Moderator of Framing Effects
Lecheler, S. K., de Vreese, C. H., & Slothuus, R. (2009) · Communication Research · doi:10.1177/0093650209333028 · Cited by 178 · see citing works
Stigma among patients with lung cancer: a patient-reported measurement model
Hamann, H. A., Ostroff, J. S., Marks, E. G., Gerber, D. E., Schiller, J. H., & Lee, S. J. C. (2013) · Psycho-Oncology · doi:10.1002/pon.3371 · Cited by 177 · see citing works
Why Individuals Participate in Micro-task Crowdsourcing Work Environment: Revealing Crowdworkers’ Perceptions
Deng, X. N., & Joshi, K. D. (2016) · Journal of the Association for Information Systems · doi:10.17705/1jais.00441 · Cited by 175 · see citing works

Authors: Lombard, M., Snyder-Duch, J., & Bracken, C. C.
Affiliations: Temple University (US)
Cleveland State University (US)
Venue: Human Communication Research
Publisher: Oxford University Press (OUP)
Published: 2002-10
Volume: 28
Issue: 4
Pages: 587-604
ISSN: 0360-3989 (ISSN-L)
1468-2958
DOI: 10.1111/j.1468-2958.2002.tb00826.x
OpenAlex: W2160375265
Cited by: 3,068
References: 42
Citations by year: 2026: 234 · 2025: 173 · 2024: 136 · 2023: 143 · 2022: 212 · 2021: 238 · 2020: 206 · 2019: 206 · 2018: 174 · 2017: 168 · 2016: 174 · 2015: 153 · 2014: 168 · 2013: 158 · 2012: 117
FWCI: 19.75
Type: journal article
Language: en
Open access: closed
Licence: doi.wiley.com/10.1002/tdm_license_1.1 (tdm, 4718 d)
Locations: Human Communication Research (publishedVersion)
EngagedScholarship @ Cleveland State University (Cleveland State University) (submittedVersion)
Topics: Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Reliability and Agreement in Measurement
Social Sciences › Communication › Media Studies and Communication
Social Sciences › General Social Sciences › Computational and Text Analysis Methods
Keywords: intercoder reliability, inter-coder agreement, content analysis, mass communication research, reporting practices, percent agreement, reliability index, coding procedures, research guidelines

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Content Analysis in Mass Communication: Assessment and Reporting of Intercoder Reliability

Abstract

Integrity signals

Claims the paper makes

Claims other papers make about this one

Cites

Cited by

Related

Record

About these checks