The DOI matches its Crossref record; 42 of the 42 references it cites were found in a registry or an index.
Content Analysis in Mass Communication: Assessment and Reporting of Intercoder Reliability
Abstract
As a method specifically intended for the study of messages, content analysis is fundamental to mass communication research. Intercoder reliability, more specifically termed intercoder agreement, is a measure of the extent to which independent judges make the same coding decisions in evaluating the characteristics of messages, and is at the heart of this method. Yet there are few standard and accessible guidelines available regarding the appropriate procedures to use to assess and report intercoder reliability, or software tools to calculate it. As a result, it seems likely that there is little consistency in how this critical element of content analysis is assessed and reported in published mass communication studies. Following a review of relevant concepts, indices, and tools, a content analysis of 200 studies utilizing content analysis published in the communication literature between 1994 and 1998 is used to characterize practices in the field. The results demonstrate that mass communication researchers often fail to assess (or at least report) intercoder reliability and often rely on percent agreement, an overly liberal index. Based on the review and these results, concrete guidelines are offered regarding procedures for assessment and reporting of this important aspect of content analysis.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“In line with recommended approaches [89], we calculated Cohen's kappa for the level codes and form codes.”
From Virtual Museums as Meaning-Modeling Systems in Digital Heritage (2025) · cites this paper as [89]
In the citing paper
“After clarifying the criteria, the evaluators similarly classified 90.3% of the studies, a satisfactory percentage (Lombard et al., 2002).”
From Online and Mobile Interventions for Problem Gambling, Alcohol, and Drugs: A Systematic Review (2017) · cites this paper as Lombard et al., 2002
Citations
Cites
A Coefficient of Agreement for Nominal Scales
Cohen, J. (1960) · Educational and Psychological Measurement · doi:10.1177/001316446002000104 · Cited by 42,398
Content Analysis: An Introduction to its Methodology.
Shelley, M., & Krippendorff, K. (1984) · Journal of the American Statistical Association · doi:10.2307/2288384 · Cited by 24,351
The Content Analysis Guidebook
Neuendorf, K. A. (2017) · doi:10.4135/9781071802878 · Cited by 9,286 · cited 10×
“), but this technique has also been questioned (Neuendorf, 2002). With the coding data in hand, the researcher calculates and reports one or more indices of reliability. Popping (1988) identified 39 different “agreement indices” for coding nominal categories, which excludes several techniques for ratio and interval le…”
BackgroundMethodInfluentialA concordance correlation coefficient to evaluate reproducibility.
Lin, L. I.-K. (1989) · Biometrics · doi:10.2307/2532051 · Cited by 8,785
Weighted kappa: nominal scale agreement provision for scaled disagreement or partial credit
Cohen, J. (1968) · Psychological Bulletin · doi:10.1037/h0026256 · Cited by 8,736
Measuring nominal scale agreement among many raters.
Fleiss, J. L. (1971) · Psychological Bulletin · doi:10.1037/h0031619 · Cited by 8,733
Content Analysis for the Social Sciences and Humanities.
Parker, E. B., & Holsti, O. R. (1970) · American Sociological Review · doi:10.2307/2093233 · Cited by 4,553
Content Analysis in Communication Research.
Pool, I. D. S., & Berelson, B. (1952) · American Sociological Review · doi:10.2307/2088029 · Cited by 3,106
Content Analysis for the Social Sciences and Humanities
Parker, S. R. (1970) · Sociology · doi:10.1177/003803857000400231 · Cited by 2,178
Handbook of Research Methods in Social and Personality Psychology
West, T., Reis, H. T., Judd, C. M., Banaji, M. R., Crandall, C., Giner-Sorolla, R., Biernat, M., Fiedler, K., Ermark, F., Tepper, S. J., et al. (2024) · Cambridge University Press eBooks · doi:10.1017/9781009170123 · Cited by 2,134
Reliability of Content Analysis: The Case of Nominal Scale Coding
Scott, W. A. (1955) · Public Opinion Quarterly · doi:10.1086/266577 · Cited by 1,853
Coefficient Kappa: Some Uses, Misuses, and Alternatives
Brennan, R. L., & Prediger, D. J. (1981) · Educational and Psychological Measurement · doi:10.1177/001316448104100307 · Cited by 1,676
Rethinking validity and reliability in content analysis
Potter, W. J., & Levine‐Donnerstein, D. (1999) · Journal of Applied Communication Research · doi:10.1080/00909889909365539 · Cited by 1,662
Reliability of Nominal Data Based on Qualitative Judgments
Perreault, W. D., & Leigh, L. E. (1989) · Journal of Marketing Research · doi:10.2307/3172601 · Cited by 1,575
Kolbe, R. H., & Burnett, M. S. (1991) · Journal of Consumer Research · doi:10.1086/209256 · Cited by 1,516
Content Analysis in Communication Research
Burns, T., & Berelson, B. (1953) · British Journal of Sociology · doi:10.2307/587175 · Cited by 1,433
Handbook of Research Methods in Social and Personality Psychology: Author Index
Reis, H. T., & Judd, C. M. (2013) · Cambridge University Press eBooks · doi:10.1017/cbo9780511996481 · Cited by 1,080
Reliability Measures for Qualitative Data: Theory and Implications
Rust, R. T., & Cooil, B. (1994) · Journal of Marketing Research · doi:10.1177/002224379403100101 · Cited by 978
Beyond kappa: a review of interrater agreement measures
Banerjee, M., Capozzoli, M., McSweeney, L., & Sinha, D. (1999) · Canadian Journal of Statistics · doi:10.2307/3315487 · Cited by 976
Investigating Communication: An Introduction to Research Methods
Frey, L. R., Botan, C. H., & Kreps, G. L. (1999) · Cited by 939
Interrater reliability and agreement of subjective judgments.
Tinsley, H. E. A., & Weiss, D. J. (1975) · Journal of Counseling Psychology · doi:10.1037/h0076640 · Cited by 883
Handbook of Applied Multivariate Statistics and Mathematical Modeling
Tinsley, H. E. A., & Brown, S. D. (2000) · Elsevier eBooks · doi:10.1016/b978-0-12-691360-6.x5000-9 · Cited by 842
Riffe, D., Lacy, S. R., Watson, B. R., & Fico, F. G. (2019) · Routledge eBooks · doi:10.4324/9780429464287 · Cited by 469
Interrater Reliability and Agreement
Tinsley, H. E. A., & Weiss, D. J. (2000) · Elsevier eBooks · doi:10.1016/b978-012691360-6/50005-7 · Cited by 411
Communications Through Limited Response Questioning
Bennett, E., Alpert, R., & Goldstein, A. C. (1954) · Public Opinion Quarterly · doi:10.1086/266520 · Cited by 254
Sampling Error and Selecting Intercoder Reliability Samples for Nominal Content Categories
Lacy, S. R., & Riffe, D. (1996) · Journalism & Mass Communication Quarterly · doi:10.1177/107769909607300414 · Cited by 243
A Content Analysis of Content Analyses: Twenty-Five Years of Journalism Quarterly
Riffe, D., & Freitag, A. R. (1997) · Journalism & Mass Communication Quarterly · doi:10.1177/107769909707400306 · Cited by 206
Hughes, M. A., & Garrett, D. E. (1990) · Journal of Marketing Research · doi:10.1177/002224379002700206 · Cited by 177
Judgment of counseling process: Reliability, agreement, and error.
Lawlis, G. F., & Lu, E. (1972) · Psychological Bulletin · doi:10.1037/h0032935 · Cited by 174
Behavioral observation and coding.
Bakeman, R. A. (2000) · Cited by 154
Dewey, M. E. (1983) · The British Journal of Psychiatry · doi:10.1192/bjp.143.5.487 · Cited by 102
Research Methods in the Social Sciences
Ellis, L. (1994) · Medical Entomology and Zoology · Cited by 93
Mass Communication Research: Contemporary Methods and Applications
Singletary, M. W. (1993) · Medical Entomology and Zoology · Cited by 92
On Agreement Indices for Nominal Data
Popping, R. (1988) · Palgrave Macmillan UK eBooks · doi:10.1007/978-1-349-19051-5_6 · Cited by 89
A Content Analysis of Content Analyses: Twenty-Five Years of Journalism Quarterly
Riffe, D., & Freitag, A. R. (1997) · Journalism & Mass Communication Quarterly · doi:10.1177/107769909707400414 · Cited by 89
Generalization of Scott's Index of Intercoder Agreement
Craig, R. T. (1981) · Public Opinion Quarterly · doi:10.1086/268657 · Cited by 73
A SAS MACRO for Calculating Intercoder Agreement in Content Analysis
Kang, N., Kara, A., Laskey, H. A., & Seaton, F. B. (1993) · Journal of Advertising · doi:10.1080/00913367.1993.10673400 · Cited by 34
Effects of the use of percentage agreement on behavioral observation reliabilities: A reassessment
Suen, H. K., & Lee, P. S. C. (1985) · Journal of Psychopathology and Behavioral Assessment · doi:10.1007/bf00960754 · Cited by 33
Does Hollywood Hate Business or Money?
Lichter, S. R., Lichter, L. S., & Amundson, A. (1997) · Journal of Communication · doi:10.1111/j.1460-2466.1997.tb02693.x · Cited by 18
AGREE, a package for computing nominal scale agreement
Popping, R. (1984) · Computational Statistics & Data Analysis · doi:10.1016/0167-9473(84)90008-2 · Cited by 13
Ethical Advertising Research Standards: Three Case Studies
Martin, C. R. (1994) · Journal of Advertising · doi:10.1080/00913367.1994.10673447 · Cited by 10
Computer Studies Measuring the Joint Agreement between Multiple Raters and a Standard
Berry, K. J., & Mielke, P. W. (1997) · Educational and Psychological Measurement · doi:10.1177/0013164497057003013 · Cited by 6
Research works printed but not linked: 2
- Lacy S., 1996, Journalism & Mass Communication Quarterly, 73, 969
- Perreault W. D., 1989, Journal of Marketing Research, 26, 135, 10.1177/002224378902600201 · DOI · Paper page
Cited by
Sorted by citations
Answering the Call for a Standard Reliability Measure for Coding Data
Hayes, A. F., & Krippendorff, K. (2007) · Communication Methods and Measures · doi:10.1080/19312450709336664 · Cited by 4,225
Intercoder Reliability in Qualitative Research: Debates and Practical Guidelines
O’Connor, C., & Joffe, H. (2020) · International Journal of Qualitative Methods · doi:10.1177/1609406919899220 · Cited by 3,871
“Studies that influence important medical, policy, or financial decisions arguably merit a higher ICR threshold than exploratory academic research (Hruschka et al., 2004; Lombard et al., 2002).”
BackgroundMethodInfluentialCampbell, J. L., Quincy, C., Osserman, J., & Pedersen, O. K. (2013) · Sociological Methods & Research · doi:10.1177/0049124113500475 · Cited by 2,974
Reliability in content analysis: some common misconceptions and recommendations.
Krippendorff, K. (2004) · Human Communication Research · doi:10.1093/hcr/30.3.411 · Cited by 1,027
Structural Topic Models for Open‐Ended Survey Responses
Roberts, M. E., Stewart, B., Tingley, D. H., Lucas, C., Leder-Luis, J., Gadarian, S. K., Albertson, B. L., & Rand, D. G. (2014) · American Journal of Political Science · doi:10.1111/ajps.12103 · Cited by 2,103
Metadata
- Authors
- Lombard, M., Snyder-Duch, J., & Bracken, C. C.
- Affiliations
- Temple University (US)
- Cleveland State University (US)
- Venue
- Human Communication Research
- Publisher
- Oxford University Press (OUP)
- Published
- 2002-10
- Volume
- 28
- Issue
- 4
- Pages
- 587-604
- ISSN
- 0360-3989 (ISSN-L)
- 1468-2958
- OpenAlex
- W2160375265
- Cited by
- 3,068
- References
- 42
- Citations by year
- 2026: 234 · 2025: 173 · 2024: 136 · 2023: 143 · 2022: 212 · 2021: 238 · 2020: 206 · 2019: 206 · 2018: 174 · 2017: 168 · 2016: 174 · 2015: 153 · 2014: 168 · 2013: 158 · 2012: 117
- FWCI
- 19.75
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Human Communication Research (publishedVersion)
- EngagedScholarship @ Cleveland State University (Cleveland State University) (submittedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Reliability and Agreement in Measurement
- Social Sciences › Communication › News Media Framing and Political Communication
- Social Sciences › General Social Sciences › Computational and Text Analysis Methods
- Keywords
- intercoder reliability, inter-coder agreement, content analysis, mass communication research, reporting practices, percent agreement, reliability index, coding procedures, research guidelines
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
