Rethinking validity and reliability in content analysis
Abstract
The central thesis in this essay is that validity and reliability should be conceptualized differently across the various forms of content and the various uses of theory. This is especially true with applied communication research where a theory is not always available to guide the design. A distinction needs to made between manifest and latent (pattern and projective) content. Also, we argue that content analyses need not be limited to theory‐based coding schemes and standards set by experts. When researchers are clear about what kind of content they want to analyze and the role of theory in their studies, they are in a better position to select the most appropriate strategies for demonstrating validity and reliability.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Objectivity was relatively easy to obtain by a single coder because the coding task dealt with manifest content.
“Given that the coding task dealt with manifest content, which can be observed in the nature of the coding questions, objectivity was relatively easy to obtain by a single coder (Potter and LevineDonnerstein 1999).”
From Digital product passport for sustainable and circular supply chain management: a structured review of use cases (2024) · cites this paper as Potter and Levine-Donnerstein 1999
No theory was used in designing the coding categories.
“No theory was used in designing the coding categories as we aimed to obtain descriptive results as a starting point for the later analysis (Potter and Levine-Donnerstein 1999).”
From Digital product passport for sustainable and circular supply chain management: a structured review of use cases (2024) · cites this paper as Potter and Levine-Donnerstein 1999
Potter and Levine-Donnerstein (1999) underscored the two-step process in content analysis: developing a coding scheme congruent with the theoretical framework of the study, and assessing coders' correct decision making according to certain guidelines and standards.
“At this point, we refer to Potter and LevineDonnerstein's (1999) concept of validity and reliability incontent analysis. They underscored the two-step process in content analysis: developing a coding scheme congruent with the theoretical framework of the study, and assessing coders' correct decision making according to certain guidelines and standards.”
From Investigating Critical Thinking in ELT Textbooks: A Systematic Literature Review of Textbook Evaluation Studies (2024) · cites this paper as Potter and LevineDonnerstein's (1999)
Cites
Reliability of Nominal Data Based on Qualitative Judgments
Perreault, W. D., & Leigh, L. E. (1989) · Journal of Marketing Research · doi:10.2307/3172601 · Cited by 1,575 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Three Approaches to Qualitative Content Analysis
Hsieh, H.-F., & Shannon, S. E. (2005) · Qualitative Health Research · doi:10.1177/1049732305276687 · Cited by 44,601 · see citing works
Qualitative Content Analysis: A Focus on Trustworthiness
Elo, S., Kääriäinen, M., Kanste, O., Pölkki, T., Utriainen, K., & Kyngäs, H. (2014) · Sage Open · doi:10.1177/2158244014522633 · Cited by 4,395 · see citing works
Intercoder Reliability in Qualitative Research: Debates and Practical Guidelines
O’Connor, C., & Joffe, H. (2020) · International Journal of Qualitative Methods · doi:10.1177/1609406919899220 · Cited by 3,871 · see citing works
“Inarguably, complete objectivity is not a realistic expectation while coding latent content (Potter & Levine-Donnerstein, 1999) and indeed may not be a desirable one.”
BackgroundCritical thinking, cognitive presence, and computer conferencing in distance education
Garrison, D. R., Anderson, T., & Archer, W. (2001) · American Journal of Distance Education · doi:10.1080/08923640109527071 · Cited by 3,044 · see citing works
Content Analysis in Mass Communication: Assessment and Reporting of Intercoder Reliability
Lombard, M., Snyder-Duch, J., & Bracken, C. C. (2002) · Human Communication Research · doi:10.1111/j.1468-2958.2002.tb00826.x · Cited by 3,068 · see citing works
Kleinheksel, A. J., Rockich-Winston, N., Tawfik, H., & Wyatt, T. R. (2020) · American Journal of Pharmaceutical Education · doi:10.5688/ajpe7113 · Cited by 1,054 · see citing works
Reliability in Coding Open-Ended Data: Lessons Learned from HIV Behavioral Research
Hruschka, D. J., Schwartz, D., St.John, D. C., Picone-Decaro, E., Jenkins, R. A., & Carey, J. W. (2004) · Field Methods · doi:10.1177/1525822x04266540 · Cited by 829 · see citing works
Belur, J., Tompson, L., Thornton, A., & Simon, M. (2021) · Sociological Methods & Research · doi:10.1177/0049124118799372 · Cited by 753 · see citing works
Michie, S., Wood, C. E., Johnston, M., Abraham, C., Francis, J. J., & Hardeman, W. (2015) · Health Technology Assessment · doi:10.3310/hta19990 · Cited by 680 · see citing works
Towards a theory of multi-tier sustainable supply chains: A systematic literature review
M. Tachizawa, E., & Yew Wong, C. (2014) · Supply Chain Management: An International Journal · doi:10.1108/scm-02-2014-0070 · Cited by 419 · see citing works
Acquah, E. O., & Katz, H. T. (2020) · Computers & Education · doi:10.1016/j.compedu.2019.103667 · Cited by 280 · see citing works
Best practices for the human evaluation of automatically generated text
van der Lee, C., Gatt, A., van Miltenburg, E., Wubben, S., & Krahmer, E. (2019) · Proceedings of the 12th International Conference on Natural Language Generation · doi:10.18653/v1/w19-8643 · Cited by 205 · see citing works
“…size For expert-focused evaluations, good advice is provided by Van Enschot et al. (2017): difficult coding tasks (which most NLG evaluations are) require three or more annotators (though preferably more; see Potter and Levine-Donnerstein, 1999), more straightforward tasks can do with two to three.”
MethodZhang, A., & Seuring, S. (2024) · International Journal of Logistics Research and Applications · doi:10.1080/13675567.2024.2374256 · Cited by 158 · see citing works
Bussolari, C., Currin-McCulloch, J., Packman, W., Kogan, L., & Erdman, P. (2021) · Animals · doi:10.3390/ani11020330 · Cited by 113 · see citing works
Gender as a Variable in Natural-Language Processing: Ethical Considerations
Larson, B. (2017) · Proceedings of the First ACL Workshop on Ethics in Natural Language Processing · doi:10.18653/v1/w17-1601 · Cited by 111 · see citing works
Applying the behaviour change technique (BCT) taxonomy v1: A study of coder training
Wood, C. E., Richardson, M., Johnston, M., Abraham, C., Francis, J., Hardeman, W., & Michie, S. (2015) · Translational Behavioral Medicine · doi:10.1007/s13142-014-0290-z · Cited by 100 · see citing works
Potential use of ayahuasca in grief therapy
González, D., Carvalho, M., Cantillo, J., Aixalá, M., & Farré, M. (2019) · OMEGA - Journal of Death and Dying · doi:10.1177/0030222817710879 · Cited by 44 · see citing works
Bussolari, C. J., Habarth, J., Katz, R., Phillips, S., Carmack, B., & Packman, W. (2018) · Bereavement Care · doi:10.1080/02682621.2018.1542571 · Cited by 36 · see citing works
Wang, J., Zhang, L., Xu, L., & Qian, X. (2024) · Journal of Criminal Justice · doi:10.1016/j.jcrimjus.2024.102231 · Cited by 28 · see citing works
Caregiving and Help Seeking in First Episode Psychosis: A Qualitative Study
Sadath, A., Muralidhar, D., Varambally, S., Jose, J. P., & Gangadhar, B. N. (2014) · Journal of Psychosocial Rehabilitation and Mental Health · doi:10.1007/s40737-014-0013-2 · Cited by 26 · see citing works
Flink, I. J. E., Beirens, T. M. J., Butte, D., & Raat, H. (2014) · Ethnicity & Health · doi:10.1080/13557858.2013.801402 · Cited by 17 · see citing works
Vukušić Rukavina, T., Machala Poplašen, L., Majer, M., Relić, D., Viskić, J., & Marelić, M. (2022) · JMIR Medical Education · doi:10.2196/35585 · Cited by 12 · see citing works
Villagrana, M., & Lee, S.-Y. (2020) · Journal of Public Child Welfare · doi:10.1080/15548732.2019.1657548 · Cited by 9 · see citing works
Soe, T. (2024) · Theory and Practice of Second Language Acquisition · doi:10.31261/tapsla.13882 · Cited by 5 · see citing works
The therapeutic use of ayahuasca in grief
González, D., Aronovich, A. A., & Carvalho, M. (2021) · Ayahuasca Healing and Science · doi:10.1007/978-3-030-55688-4_12 · Cited by 5 · see citing works
- Authors
- Potter, W. J., & Levine‐Donnerstein, D.
- Affiliations
- Florida State University (US)
- University of California, Santa Barbara (US)
- Venue
- Journal of Applied Communication Research
- Publisher
- Informa UK Limited
- Published
- 1999-08-01
- Volume
- 27
- Issue
- 3
- Pages
- 258-284
- ISSN
- 0090-9882 (ISSN-L)
- 1479-5752
- OpenAlex
- W2166410494
- Cited by
- 1,662
- References
- 1
- Citations by year
- 2026: 53 · 2025: 93 · 2024: 121 · 2023: 89 · 2022: 113 · 2021: 136 · 2020: 130 · 2019: 128 · 2018: 98 · 2017: 107 · 2016: 82 · 2015: 82 · 2014: 95 · 2013: 69 · 2012: 54
- FWCI
- 46.29
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Journal of Applied Communication Research (publishedVersion)
- Topics
- Social Sciences › Education › Qualitative Research Methods and Applications
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Reliability and Agreement in Measurement
- Social Sciences › General Social Sciences › Computational and Text Analysis Methods
- Keywords
- content analysis, validity, reliability statistics, manifest content, latent content, coding scheme, applied communication research, research design
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
