The DOI matches its Crossref record; 70 of the 70 references it cites were found in a registry or an index.
Reliability in Coding Open-Ended Data: Lessons Learned from HIV Behavioral Research
Abstract
Analysis of text from open-ended interviews has become an important research tool in numerous fields, including business, education, and health research. Coding is an essential part of such analysis, but questions of quality control in the coding process have generally received little attention. This article examines the text coding process applied to three HIV-related studies conducted with the Centers for Disease Control and Prevention considering populations in the United States and Zimbabwe. Based on experience coding data from these studies, we conclude that (1) a team of coders will initially produce very different codings, but (2) it is possible, through a process of codebook revision and recoding, to establish strong levels of intercoder reliability (e.g., most codes with kappa 0.8). Furthermore, steps can be taken to improve initially poor intercoder reliability and to reduce the number of iterations required to generate stronger intercoder reliability.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Three research team members were involved in forming the initial codebook, which was validated by using a multi-step coding process [31].”
From Use of Intelligent Voice Assistants by Older Adults with Low Technology Use (2020) · cites this paper as [31]
In the citing paper
“The nature of each these primary commands was coded as “polite”, “rude” or “neutral” using a using a multi-step process by two research team members [24], achieving a Cohen's Kappa of 0.91.”
From "Phantom Friend" or "Just a Box with Information": Personification and Ontological Categorization of Smart Speaker-based Voice Assistants by Older Adults (2019) · cites this paper as [24]
Citations
Cites
The Measurement of Observer Agreement for Categorical Data
Landis, J. R., & Koch, G. G. (1977) · Biometrics · doi:10.2307/2529310 · Cited by 81,148
A Coefficient of Agreement for Nominal Scales
Cohen, J. (1960) · Educational and Psychological Measurement · doi:10.1177/001316446002000104 · Cited by 42,398
Handbook of Qualitative Research
Crozier, G., Denzin, N., & Lincoln, Y. (1994) · British Journal of Educational Studies · doi:10.2307/3121684 · Cited by 26,666
Content Analysis: An Introduction to its Methodology.
Shelley, M., & Krippendorff, K. (1984) · Journal of the American Statistical Association · doi:10.2307/2288384 · Cited by 24,312
Judgment under Uncertainty: Heuristics and Biases
Tversky, A., & Kahneman, D. (1975) · Utility, Probability, and Human Decision Making · doi:10.1007/978-94-010-1834-0_8 · Cited by 22,650
Statistical Methods for Rates and Proportions
Fleiss, J. L., Levin, B., & Paik, M. C. (2003) · Wiley Series in Probability and Statistics · doi:10.1002/0471445428 · Cited by 18,646
(2008) · Biostatistics and Microbiology: A Survival Manual · doi:10.1007/978-0-387-77282-0_6 · Cited by 17,371
Content Analysis: An Introduction to Its Methodology
Krippendorff, K. (2019) · doi:10.4135/9781071878781 · Cited by 15,448
(1998) · Cited by 13,501
(1994) · Cited by 13,126
Research works printed but not linked: 11
- Bell, P., 2001, Journal of Information, Communication and Library Sciences, 7, 81
- Carey, J., 1998, Cultural Anthropology Methods Journal, 10, 14
- Carey, J. W., 1996, Cultural Anthropology Methods, 8, 1
- 10.1037/0033-2909.86.5.974 · doi:10.1037/0033-2909.86.5.974 · Paper page
- General Accounting Office (GAO), 1991, Using structured interview techniques
- Guba, E. G., 1994, Handbook of qualitative research, 105
- MacQueen, K. M., 1998, Cultural Anthropology Methods, 10, 31
- Mutchler, M. G., Journal of Homosexuality
- Nunnally, J. C., 1994, Psychometric theory
- Simon, H. A., 1981, The sciences of the artificial
- Strotman, R., 2002, AnSWR: Analysis software for Word-based records, version 6.2
Cited by
Conceptualizing the circular economy: An analysis of 114 definitions
Kirchherr, J., Reike, D., & Hekkert, M. (2017) · Resources, Conservation and Recycling · doi:10.1016/j.resconrec.2017.09.005 · Cited by 8,550
Intercoder Reliability in Qualitative Research: Debates and Practical Guidelines
O’Connor, C., & Joffe, H. (2020) · International Journal of Qualitative Methods · doi:10.1177/1609406919899220 · Cited by 3,871
“Theoretical relevance or meaning should never be sacrificed for reliability (Hruschka et al., 2004).”
BackgroundInfluentialCampbell, J. L., Quincy, C., Osserman, J., & Pedersen, O. K. (2013) · Sociological Methods & Research · doi:10.1177/0049124113500475 · Cited by 2,974
Startups in times of crisis – A rapid response to the COVID-19 pandemic
Kuckertz, A., Brändle, L., Gaudig, A., Hinderer, S., Morales Reyes, C. A., Prochotta, A., Steinbrink, K. M., & Berger, E. S. C. (2020) · Journal of Business Venturing Insights · doi:10.1016/j.jbvi.2020.e00169 · Cited by 1,178
Qualitative research and content validity: developing best practices based on science and experience
Brod, M., Tesler, L. E., & Christensen, T. L. (2009) · Quality of Life Research · doi:10.1007/s11136-009-9540-9 · Cited by 935
Metadata
- Authors
- Hruschka, D. J., Schwartz, D., St.John, D. C., Picone-Decaro, E., Jenkins, R. A., & Carey, J. W.
- Affiliations
- Centers for Disease Control and Prevention (US)
- ORCID
- Hruschka, Daniel J.
- Venue
- Field Methods
- Publisher
- SAGE Publications
- Published
- 2004-08
- Volume
- 16
- Issue
- 3
- Pages
- 307-331
- ISSN
- 1064-8631 (ISSN-L)
- 1525-822X
- 1552-3969
- OpenAlex
- W2138996324
- Cited by
- 829
- References
- 70
- Citations by year
- 2026: 31 · 2025: 53 · 2024: 54 · 2023: 30 · 2022: 66 · 2021: 52 · 2020: 78 · 2019: 61 · 2018: 62 · 2017: 57 · 2016: 76 · 2015: 60 · 2014: 35 · 2013: 43 · 2012: 18
- FWCI
- 3.39
- Type
- journal article
- Language
- en
- Open access
- closed
- Licence
- journals.sagepub.com/page/policies/text-and-data-mining-license (tdm)
- Locations
- Field Methods (publishedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Reliability and Agreement in Measurement
- Social Sciences › General Social Sciences › Computational and Text Analysis Methods
- Social Sciences › Sociology and Political Science › Survey Methodology and Nonresponse
- Keywords
- intercoder reliability, text coding, codebook generation, interview coding, kappa statistic
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Reliability in Coding Open-Ended Data: Lessons Learned from HIV Behavioral Research. https://app.cochat.ai/papers/10.1177/1525822x04266540
- The paper
- Hruschka, D. J., Schwartz, D., St.John, D. C., Picone-Decaro, E., Jenkins, R. A., & Carey, J. W. (2004). Reliability in Coding Open-Ended Data: Lessons Learned from HIV Behavioral Research. Field Methods. https://doi.org/10.1177/1525822x04266540
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.