Reliability in Coding Open-Ended Data: Lessons Learned from HIV Behavioral Research
The DOI matches its Crossref record; 70 of the 70 references it cites were found in a registry or an index.
Abstract
Analysis of text from open-ended interviews has become an important research tool in numerous fields, including business, education, and health research. Coding is an essential part of such analysis, but questions of quality control in the coding process have generally received little attention. This article examines the text coding process applied to three HIV-related studies conducted with the Centers for Disease Control and Prevention considering populations in the United States and Zimbabwe. Based on experience coding data from these studies, we conclude that (1) a team of coders will initially produce very different codings, but (2) it is possible, through a process of codebook revision and recoding, to establish strong levels of intercoder reliability (e.g., most codes with kappa 0.8). Furthermore, steps can be taken to improve initially poor intercoder reliability and to reduce the number of iterations required to generate stronger intercoder reliability.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Three research team members were involved in forming the initial codebook, which was validated by using a multi-step coding process [31].”
From Use of Intelligent Voice Assistants by Older Adults with Low Technology Use (2020) · cites this paper as [31]
In the citing paper
“The nature of each these primary commands was coded as “polite”, “rude” or “neutral” using a using a multi-step process by two research team members [24], achieving a Cohen's Kappa of 0.91.”
From "Phantom Friend" or "Just a Box with Information": Personification and Ontological Categorization of Smart Speaker-based Voice Assistants by Older Adults (2019) · cites this paper as [24]
Citations
Cites
The Measurement of Observer Agreement for Categorical Data
Landis, J. R., & Koch, G. G. (1977) · Biometrics · doi:10.2307/2529310 · Cited by 81,148
A Coefficient of Agreement for Nominal Scales
Cohen, J. (1960) · Educational and Psychological Measurement · doi:10.1177/001316446002000104 · Cited by 42,398
Handbook of Qualitative Research
Crozier, G., Denzin, N., & Lincoln, Y. (1994) · British Journal of Educational Studies · doi:10.2307/3121684 · Cited by 26,666
Content Analysis: An Introduction to its Methodology.
Shelley, M., & Krippendorff, K. (1984) · Journal of the American Statistical Association · doi:10.2307/2288384 · Cited by 24,312
Judgment under Uncertainty: Heuristics and Biases
Tversky, A., & Kahneman, D. (1975) · Utility, Probability, and Human Decision Making · doi:10.1007/978-94-010-1834-0_8 · Cited by 22,650
Statistical Methods for Rates and Proportions
Fleiss, J. L., Levin, B., & Paik, M. C. (2003) · Wiley Series in Probability and Statistics · doi:10.1002/0471445428 · Cited by 18,646
(2008) · Biostatistics and Microbiology: A Survival Manual · doi:10.1007/978-0-387-77282-0_6 · Cited by 17,371
Content Analysis: An Introduction to Its Methodology
Krippendorff, K. (2019) · doi:10.4135/9781071878781 · Cited by 15,448
(1998) · Cited by 13,501
(1994) · Cited by 13,126
Hutchins, E. (1995) · doi:10.7551/mitpress/1881.001.0001 · Cited by 10,801
Cicchetti, D. V. (1994) · Psychological Assessment · doi:10.1037/1040-3590.6.4.284 · Cited by 9,349
Statistical Methods for Rates and Proportions.
Everitt, B. S., & Fleiss, J. L. (1981) · Biometrics · doi:10.2307/2530193 · Cited by 8,990
Measuring nominal scale agreement among many raters.
Fleiss, J. L. (1971) · Psychological Bulletin · doi:10.1037/h0031619 · Cited by 8,733
Judgment under Uncertainty: Heuristics and Biases
(1982) · doi:10.1017/cbo9780511809477 · Cited by 8,240
Weber, R. (1990) · doi:10.4135/9781412983488 · Cited by 7,750
Handbook of Qualitative Research
(1994) · Journal of Environmental Psychology · doi:10.1016/s0272-4944(05)80230-0 · Cited by 5,715
Judgment Under Uncertainty: Heuristics and Biases.
Shafer, G. R., Kahnerman, D., Slovic, P., & Tversky, A. (1984) · Journal of the American Statistical Association · doi:10.2307/2288362 · Cited by 5,657
Research methods in anthropology: qualitative and quantitative approaches
(2002) · Choice Reviews Online · doi:10.5860/choice.39-4047 · Cited by 5,638
Ryan, G. W., & Bernard, H. R. (2003) · Field Methods · doi:10.1177/1525822x02239569 · Cited by 5,475
Judgment under uncertainty: List of contributors
Kahneman, D. J., Slovic, P., & Tversky, A. (1982) · Cited by 4,670
Judgment under uncertainty: Causality and attribution
Kahneman, D. J., Slovic, P., & Tversky, A. (1982) · Cited by 4,670
Essentials of Psychological Testing
Rosner, B., & Cronbach, L. J. (1960) · The American Journal of Psychology · doi:10.2307/1419921 · Cited by 4,513
Qualitative research in health care: Assessing quality in qualitative research
Mays, N. (2000) · BMJ · doi:10.1136/bmj.320.7226.50 · Cited by 3,742
Content Analysis in Mass Communication: Assessment and Reporting of Intercoder Reliability
Lombard, M., Snyder-Duch, J., & Bracken, C. C. (2002) · Human Communication Research · doi:10.1111/j.1468-2958.2002.tb00826.x · Cited by 3,068
Qualitative Research: Rigour and qualitative research
Mays, N., & Pope, C. (1995) · BMJ · doi:10.1136/bmj.311.6997.109 · Cited by 2,715
Byrt, T., Bishop, J., & Carlin, J. B. (1993) · Journal of Clinical Epidemiology · doi:10.1016/0895-4356(93)90018-v · Cited by 2,179
Codebook Development for Team-Based Qualitative Analysis
MacQueen, K. M., McLellan, E., Kay, K., & Milstein, B. (1998) · CAM Journal · doi:10.1177/1525822x980100020301 · Cited by 1,675
Rethinking validity and reliability in content analysis
Potter, W. J., & Levine‐Donnerstein, D. (1999) · Journal of Applied Communication Research · doi:10.1080/00909889909365539 · Cited by 1,662
Coefficient Kappa: Some Uses, Misuses, and Alternatives
Brennan, R. L., & Prediger, D. J. (1981) · Educational and Psychological Measurement · doi:10.1177/001316448104100307 · Cited by 1,570
Kolbe, R. H., & Burnett, M. S. (1991) · Journal of Consumer Research · doi:10.1086/209256 · Cited by 1,520
Nursing research : the application of qualitative approaches
Morse, J. M., & Field, P. (1996) · Medical Entomology and Zoology · Cited by 1,475
(2017) · doi:10.4135/9781473960275 · Cited by 1,474
Madill, A., Jordan, A., & Shirley, C. (2000) · British Journal of Psychology · doi:10.1348/000712600161646 · Cited by 1,105
(1994) · Cited by 1,079
Simon, W., & Cecco, J. P. D. (1980) · Journal of Marriage and the Family · doi:10.2307/351847 · Cited by 1,017
The Place of Inter-Rater Reliability in Qualitative Research: An Empirical Study
Armstrong, D., Gosling, A., Weinman, J., & Marteau, T. (1997) · Sociology · doi:10.1177/0038038597031003015 · Cited by 1,008
Beyond kappa: a review of interrater agreement measures
Banerjee, M., Capozzoli, M., McSweeney, L., & Sinha, D. (1999) · Canadian Journal of Statistics · doi:10.2307/3315487 · Cited by 976
Essentials of Psychological Testing
(2005) · PsyPag Quarterly · doi:10.53841/bpspag.2005.1.54.23 · Cited by 737
Integration and generalization of kappas for multiple raters
Conger, A. J. (1980) · Psychological Bulletin · doi:10.1037/0033-2909.88.2.322 · Cited by 545
Liebetrau, A. (1983) · Sage eBooks · doi:10.4135/9781412984942 · Cited by 479
Large sample variance of kappa in the case of different sets of raters.
Fleiss, J. L., Nee, J. C., & Landis, J. R. (1979) · Psychological Bulletin · doi:10.1037//0033-2909.86.5.974 · Cited by 398
Nursing research: The application of qualitative approaches
Redfern, S. J. (1986) · Nurse Education Today · doi:10.1016/0260-6917(86)90011-0 · Cited by 364
A systematic approach for using qualitative methods in primary prevention research
Willms, D. G., Best, J. A., Taylor, D. W., Gilbert, J. R., Wilson, D. M. C., Lindsay, E. A., & Singer, J. (1990) · Medical Anthropology Quarterly · doi:10.1525/maq.1990.4.4.02a00020 · Cited by 310 · cited 3×
“…stopped at the first round of coding (Weinberger et al. 1998), this article finds that a few coding iterations can generate acceptable agreement between coders, even when dealing with text data that has high interpretive burden (Willms et al. 1990; Carey, Morgan, and Oxtoby 1996; Hagelin 1999).”
BackgroundAnalysing qualitative interview data: addressing issues of validity andreliability
Appleton, J. V. (1995) · Journal of Advanced Nursing · doi:10.1111/j.1365-2648.1995.tb02653.x · Cited by 309
Complex Information Processing
(2013) · Psychology Press eBooks · doi:10.4324/9780203761618 · Cited by 265
Carey, J. W., Morgan, M., & Oxtoby, M. J. (1996) · CAM Journal · doi:10.1177/1525822x960080030101 · Cited by 258
Self-citationStrengthening Qualitative Studies and Reports: Standards to Promote Academic Integrity
Drisko, J. W. (1997) · Journal of Social Work Education · doi:10.1080/10437797.1997.10778862 · Cited by 258 · cited 1×
“Field Methods, Vol. 16, No. 3, August 2004 307–331 DOI: 10.1177/1525822X04266540 © 2004 Sage Publications 307 Boulton 1996), social work (Drisko 1997), and health fields in general (Mays and Pope 1995).”
Influential"Perfectly Healthy, but Dead": The Myth of Inter-Rater Reliability
Morse, J. M. (1997) · Qualitative Health Research · doi:10.1177/104973239700700401 · Cited by 245
INTERRATER CORRELATIONS DO NOT ESTIMATE THE RELIABILITY OF JOB PERFORMANCE RATINGS
MURPHY, K. R., & DESHON, R. (2000) · Personnel Psychology · doi:10.1111/j.1744-6570.2000.tb02421.x · Cited by 183
Gorden, R. L. (1992) · Cited by 129
On the Reliability of Unitizing Continuous Data
Krippendorff, K. (1995) · Sociological Methodology · doi:10.2307/271061 · Cited by 128
Mayadas, N. S. (1991) · PsycEXTRA Dataset · doi:10.1037/e539172004-001 · Cited by 93
HOW reliable are change‐corrected measures of agreement?
Guggenmoos-Holzmann, I. (1993) · Statistics in Medicine · doi:10.1002/sim.4780122305 · Cited by 87
Qualitative research in health care: I. The scope and validity of methods
Fitzpatrick, R. M., & Boulton, M. G. (1996) · Journal of Evaluation in Clinical Practice · doi:10.1111/j.1365-2753.1996.tb00036.x · Cited by 81
Analyzing Data: Maintaining Rigor in a Qualitative Study
Rose, K. E., & Webb, C. (1998) · Qualitative Health Research · doi:10.1177/104973239800800409 · Cited by 79
Taking Stock: Issues from the Literature on Validity and Reliability in Qualitative Research
Sykes, W. (1991) · Market Research Society Journal · doi:10.1177/147078539103300101 · Cited by 64
How much observational data is enough? An empirical test using marital interaction coding
Heyman, R. E., Chaudhry, B. R., Treboux, D., Crowell, J. A., Lord, C., Vivian, D., & Waters, E. (2001) · Behavior Therapy · doi:10.1016/s0005-7894(01)80047-2 · Cited by 49
Coder training: theoretical training or practical socialization?
Hak, T., & Bernts, T. (1996) · Qualitative Sociology · doi:10.1007/bf02393420 · Cited by 46
Can raters consistently evaluate the content of focus groups?
Weinberger, M., Ferguson, J. A., Westmoreland, G. R., Mamlin, L. A., Segar, D. S., Eckert, G. J., Greene, J. Y., Martin, D., & Tierney, W. M. (1998) · Social Science & Medicine · doi:10.1016/s0277-9536(97)10028-4 · Cited by 42
Evaluating HIV Prevention Interventions
Mantell, J. E., DiVittis, A. T., & Auerbach, M. I. (1997) · AIDS prevention and mental health · doi:10.1007/978-1-4899-1840-6 · Cited by 36
Recommendations from Overweight Youth Regarding School‐Based Weight Control Programs
Neumark‐Sztainer, D., & Story, M. (1997) · Journal of School Health · doi:10.1111/j.1746-1561.1997.tb01289.x · Cited by 30
CDC EZ-Text: Software for Management and Analysis of Semistructured Qualitative Data Sets
Carey, J. W., Wenzel, P. H., Reilly, C., Sheridan, J. F., & Steinberg, J. M. (1998) · Field Methods · doi:10.1177/1525822x980100010501 · Cited by 28
Self-citationParsons, J. T., Huszti, H. C., Crudder, S. O., Gage, B., Jarvis, D., Mendoza, J. L., & Parish, K. L. (1998) · AIDS and Behavior · doi:10.1023/a:1022325805088 · Cited by 19
Hagelin, E. (1999) · Journal of Pediatric Nursing · doi:10.1016/s0882-5963(99)80031-1 · Cited by 19
Fixed-Choice and Open-Ended Response Formats: A Comparison from HIV Prevention Research in Zimbabwe
Hruschka, D. J., Cummings, B., John, D. C. S., Moore, J. S., Khumalo-Sakutukwa, G. N., & Carey, J. W. (2004) · Field Methods · doi:10.1177/1525822x03262663 · Cited by 14
Self-citationO’Leary, A., Moore, J. S., Khumalo-Sakutukwa, G. N., et al. (2003) · AIDS · doi:10.1097/00002030-200307250-00020 · Cited by 10
Comparing Sexual Behavioral Patterns Between Two Bathhouses
Mutchler, M. G., Bingham, T. A., Chión, M., Jenkins, R. A., Klosinski, L. E., & Secura, G. M. (2003) · Journal of Homosexuality · doi:10.1300/j082v44n03_10 · Cited by 9
Self-citationHohnloser, J. H., Kadlec, P., & Puerner, F. (1995) · Computers and Biomedical Research · doi:10.1006/cbmr.1995.1026 · Cited by 5
The Health Care Needs of Hospitalized Patients with AIDS in Taiwan
WANG, S. S. L., LIN, L.-C., & Kuo, B. I. (1997) · AIDS Patient Care and STDs · doi:10.1089/apc.1997.11.179 · Cited by 2
Research works printed but not linked: 11
- Bell, P., 2001, Journal of Information, Communication and Library Sciences, 7, 81
- Carey, J., 1998, Cultural Anthropology Methods Journal, 10, 14
- Carey, J. W., 1996, Cultural Anthropology Methods, 8, 1
- 10.1037/0033-2909.86.5.974 · doi:10.1037/0033-2909.86.5.974 · Paper page
- General Accounting Office (GAO), 1991, Using structured interview techniques
- Guba, E. G., 1994, Handbook of qualitative research, 105
- MacQueen, K. M., 1998, Cultural Anthropology Methods, 10, 31
- Mutchler, M. G., Journal of Homosexuality
- Nunnally, J. C., 1994, Psychometric theory
- Simon, H. A., 1981, The sciences of the artificial
- Strotman, R., 2002, AnSWR: Analysis software for Word-based records, version 6.2
Cited by
Conceptualizing the circular economy: An analysis of 114 definitions
Kirchherr, J., Reike, D., & Hekkert, M. (2017) · Resources, Conservation and Recycling · doi:10.1016/j.resconrec.2017.09.005 · Cited by 8,550
Intercoder Reliability in Qualitative Research: Debates and Practical Guidelines
O’Connor, C., & Joffe, H. (2020) · International Journal of Qualitative Methods · doi:10.1177/1609406919899220 · Cited by 3,871
“Theoretical relevance or meaning should never be sacrificed for reliability (Hruschka et al., 2004).”
BackgroundInfluentialCampbell, J. L., Quincy, C., Osserman, J., & Pedersen, O. K. (2013) · Sociological Methods & Research · doi:10.1177/0049124113500475 · Cited by 2,974
Startups in times of crisis – A rapid response to the COVID-19 pandemic
Kuckertz, A., Brändle, L., Gaudig, A., Hinderer, S., Morales Reyes, C. A., Prochotta, A., Steinbrink, K. M., & Berger, E. S. C. (2020) · Journal of Business Venturing Insights · doi:10.1016/j.jbvi.2020.e00169 · Cited by 1,178
Qualitative research and content validity: developing best practices based on science and experience
Brod, M., Tesler, L. E., & Christensen, T. L. (2009) · Quality of Life Research · doi:10.1007/s11136-009-9540-9 · Cited by 935
Metadata
- Authors
- Hruschka, D. J., Schwartz, D., St.John, D. C., Picone-Decaro, E., Jenkins, R. A., & Carey, J. W.
- Affiliations
- Centers for Disease Control and Prevention (US)
- ORCID
- Hruschka, Daniel J.
- Venue
- Field Methods
- Publisher
- SAGE Publications
- Published
- 2004-08
- Volume
- 16
- Issue
- 3
- Pages
- 307-331
- ISSN
- 1064-8631 (ISSN-L)
- 1525-822X
- 1552-3969
- OpenAlex
- W2138996324
- Cited by
- 829
- References
- 70
- Citations by year
- 2026: 31 · 2025: 53 · 2024: 54 · 2023: 30 · 2022: 66 · 2021: 52 · 2020: 78 · 2019: 61 · 2018: 62 · 2017: 57 · 2016: 76 · 2015: 60 · 2014: 35 · 2013: 43 · 2012: 18
- FWCI
- 3.39
- Type
- journal article
- Language
- en
- Open access
- closed
- Licence
- journals.sagepub.com/page/policies/text-and-data-mining-license (tdm)
- Locations
- Field Methods (publishedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Reliability and Agreement in Measurement
- Social Sciences › General Social Sciences › Computational and Text Analysis Methods
- Social Sciences › Sociology and Political Science › Survey Methodology and Nonresponse
- Keywords
- intercoder reliability, text coding, codebook generation, interview coding, kappa statistic
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.