Coding In-depth Semistructured Interviews: Problems of Unitization and Intercoder Reliability and Agreement
Abstract
Many social science studies are based on coded in-depth semistructured interview transcripts. But researchers rarely report or discuss coding reliability in this work. Nor is there much literature on the subject for this type of data. This article presents a procedure for developing coding schemes for such data. It involves standardizing the units of text on which coders work and then improving the coding scheme’s discriminant capability (i.e., reducing coding errors) to an acceptable point as indicated by measures of either intercoder reliability or intercoder agreement. This approach is especially useful for situations where a single knowledgeable coder will code all the transcripts once the coding scheme has been established. This approach can also be used with other types of qualitative data and in other circumstances.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Intercoder reliability was assessed using Cohen's Kappa statistic on a representative 20% subsample of reflections (n = 22), yielding κ = 0.82, confirming high reliability.
“Intercoder reliability was assessed using Cohen's Kappa statistic on a representative 20% subsample of reflections (n = 22), yielding κ = 0.82, thus confirming high reliability (Campbell et al., 2013).”
From Generative AI in Design Thinking Pedagogy: Enhancing Creativity, Critical Thinking, and Ethical Reasoning in Higher Education (2025) · cites this paper as Campbell et al., 2013
The coding used an inductive method with no pre-given codes and constant comparison.
“That coding used the inductive method, with no pre-given codes (Campbell et al., 2013), with constant comparison of codes as initial coding proceeded.”
From The four-day workweek in Australia: insights from early adopters of the 100:80:100 model (2026) · cites this paper as Campbell et al., 2013
This sole-coder approach aligns with the interpretative nature of qualitative research, and senior researchers were involved at all stages to enhance the reliability of the findings.
“This sole-coder approach aligns with the interpretative nature of qualitative research, and senior researchers were involved at all stages to enhance the reliability of the findings [17].”
From Disability-First AI Dataset Annotation: Co-designing Stuttered Speech Annotation Guidelines with People Who Stutter (2026) · cites this paper as [17]
The second and third authors verified the initial analysis results, and conflicts were resolved using a negotiated agreement approach.
“3) The initial version of the analysis results was further verified by the second and third authors, and the negotiated agreement approach (Campbell et al., 2013) was employed to address the conflicts.”
From Exploring the problems, their causes and solutions of AI pair programming: A study on GitHub and Stack Overflow (2025) · cites this paper as Campbell et al., 2013
Cites
The Discovery of Grounded Theory: Strategies for Qualitative Research
Glaser, B. G., & Strauss, A. L. (2017) · Aldine (Routledge reprint) · doi:10.4324/9780203793206 · Cited by 25,789 · see citing works
Weber, R. (1990) · doi:10.4135/9781412983488 · Cited by 7,750 · see citing works
Toward a Conceptual Framework for Mixed-Method Evaluation Designs
Greene, J. C., Caracelli, V. J., & Graham, W. F. (1989) · Educational Evaluation and Policy Analysis · doi:10.3102/01623737011003255 · Cited by 5,687 · see citing works
Content Analysis in Mass Communication: Assessment and Reporting of Intercoder Reliability
Lombard, M., Snyder-Duch, J., & Bracken, C. C. (2002) · Human Communication Research · doi:10.1111/j.1468-2958.2002.tb00826.x · Cited by 3,068 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Intercoder Reliability in Qualitative Research: Debates and Practical Guidelines
O’Connor, C., & Joffe, H. (2020) · International Journal of Qualitative Methods · doi:10.1177/1609406919899220 · Cited by 3,871 · see citing works
Smith, B., & McGannon, K. R. (2018) · International Review of Sport and Exercise Psychology · doi:10.1080/1750984x.2017.1317357 · Cited by 3,074 · see citing works
Flexible Coding of In-depth Interviews: A Twenty-first-century Approach
Deterding, N. M., & Waters, M. C. (2021) · Sociological Methods & Research · doi:10.1177/0049124118799377 · Cited by 1,626 · see citing works
McDonald, N., Schoenebeck, S., & Forte, A. (2019) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3359174 · Cited by 1,206 · see citing works
Guidelines for establishing reliability when coding narrative data
Syed, M., & Nelson, S. C. (2015) · Emerging Adulthood · doi:10.1177/2167696815587648 · Cited by 801 · see citing works
Belur, J., Tompson, L., Thornton, A., & Simon, M. (2021) · Sociological Methods & Research · doi:10.1177/0049124118799372 · Cited by 753 · see citing works
A team-based approach to open coding: considerations for creating intercoder consensus
Cascio, M. A., Lee, E., Vaudrin, N., & Freedman, D. A. (2019) · Field Methods · doi:10.1177/1525822x19838237 · Cited by 492 · see citing works
Rose, J., & Johnson, C. W. (2020) · Journal of Leisure Research · doi:10.1080/00222216.2020.1722042 · Cited by 419 · see citing works
“Sharenting,” parent blogging, and the boundaries of the digital self
Blum-Ross, A., & Livingstone, S. (2017) · Popular Communication · doi:10.1080/15405702.2016.1223300 · Cited by 418 · see citing works
“Is your gig any good?” Analysing job quality in the Australian platform-based food-delivery sector
Goods, C., Veen, A., & Barratt, T. (2019) · Journal of Industrial Relations · doi:10.1177/0022185618817069 · Cited by 365 · see citing works
Big data, big decisions: The impact of big data on board level decision-making
Merendino, A., Dibb, S., Meadows, M., Quinn, L., Wilson, D., Simkin, L., & Canhoto, A. (2018) · Journal of Business Research · doi:10.1016/j.jbusres.2018.08.029 · Cited by 239 · see citing works
Inter-Coder Agreement in Qualitative Coding: Considerations for its Use
Halpin, S. N. (2024) · American Journal of Qualitative Research · doi:10.29333/ajqr/14887 · Cited by 149 · see citing works
Comparing Generative AI and teacher feedback: student perceptions of usefulness and trustworthiness
Henderson, M., Bearman, M., Chung, J., Fawns, T., Buckingham Shum, S., Matthews, K. E., & de Mello Heredia, J. (2026) · Assessment & Evaluation in Higher Education · doi:10.1080/02602938.2025.2502582 · Cited by 145 · see citing works
“The coders (JH, XL) then continued with the coding, consulting with each other as a process of forming consensus and sustaining inter-coder agreement as described by Campbell et al. (2013).”
Parenting with Alexa: Exploring the Introduction of Smart Speakers on Family Dynamics
Beneteau, E., Boone, A., Wu, Y., Kientz, J. A., Yip, J., & Hiniker, A. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376344 · Cited by 140 · see citing works
Spreadsheet Comprehension: Guesswork, Giving Up and Going Back to the Author
Srinivasa Ragavan, S., Sarkar, A., & Gordon, A. D. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445634 · Cited by 67 · see citing works
Kharbat, F. F., Alshawabkeh, A., & Woolsey, M. L. (2020) · Aslib Journal of Information Management · doi:10.1108/ajim-02-2020-0054 · Cited by 63 · see citing works
Challenges of Females with Autism: A Parental Perspective
Mademtzi, M., Singh, P., Shic, F., & Koenig, K. (2018) · Journal of Autism and Developmental Disorders · doi:10.1007/s10803-017-3341-8 · Cited by 54 · see citing works
Catuogno, S., Arena, C., Saggese, S., & Sarto, F. (2017) · BMC Health Services Research · doi:10.1186/s12913-017-2479-6 · Cited by 45 · see citing works
Rana, V., Verhoeven, B., & Sharma, M. (2025) · Journal of University Teaching and Learning Practice · doi:10.53761/tjse2f36 · Cited by 48 · see citing works
Balbim, G. M., Marques, I. G., Cortez, C., Magallanes, M., Rocha, J., & Marquez, D. X. (2019) · Journal of Cross-Cultural Gerontology · doi:10.1007/s10823-019-09390-8 · Cited by 37 · see citing works
Howard, B. N., Van Dorn, R., Myers, B. J., Zule, W. A., Browne, F. A., Carney, T., & Wechsberg, W. M. (2017) · BMC Health Services Research · doi:10.1186/s12913-017-2669-2 · Cited by 27 · see citing works
Zhou, X., Liang, P., Zhang, B., Li, Z., Ahmad, A., Shahin, M., & Waseem, M. (2025) · Journal of Systems and Software · doi:10.1016/j.jss.2024.112204 · Cited by 30 · see citing works
“To help understand the taxonomies of problems, causes, and solutions of using Copilot, we provide examples with the “#” symbol, which indicates the “GitHub Issue ID”, “GitHub Discussion ID”, or “SO Post ID” in the dataset (Zhou et al., 2024).”
MethodDanilova, A., Naiakshina, A., & Smith, M. (2020) · Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering · doi:10.1145/3377811.3380387 · Cited by 25 · see citing works
Desai, S., Lundy, M., & Chin, J. (2023) · Proceedings of the 5th International Conference on Conversational User Interfaces · doi:10.1145/3571884.3597141 · Cited by 25 · see citing works
Exploring the effects of Grammarly on EFL students’ foreign language anxiety and learner autonomy
Dizon, G., & Gold, J. (2023) · The JALT CALL Journal · doi:10.29140/jaltcall.v19n3.1049 · Cited by 24 · see citing works
Beahm, J. D., McCall, H. C., Carleton, R. N., Titov, N., Dear, B., & Hadjistavropoulos, H. D. (2021) · Internet Interventions · doi:10.1016/j.invent.2021.100481 · Cited by 23 · see citing works
Desai, S., Hu, X., Lundy, M., & Chin, J. (2023) · International Conference on Human-Agent Interaction · doi:10.1145/3623809.3623827 · Cited by 13 · see citing works
Integrating security controls in DevSecOps: Challenges, solutions, and future research directions
Sinan, M., Shahin, M., & Gondal, I. (2025) · Journal of Software: Evolution and Process · doi:10.1002/smr.70029 · Cited by 13 · see citing works
AI assistance in peer feedback provision: Pedagogically sound, but minimally adopted
Pozdniakov, S., Brazil, J., Banihashem, S. K., Noroozi, O., Gašević, D., Sadiq, S., & Khosravi, H. (2026) · Computers & Education · doi:10.1016/j.compedu.2026.105591 · Cited by 11 · see citing works
Li, X., & Awang, M. M. B. (2026) · European Journal of Education · doi:10.1111/ejed.70490 · Cited by 7 · see citing works
Tang, X., Li, J., & Wu, S. (2026) · Proceedings of the 2026 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3772318.3790405 · Cited by 4 · see citing works
The four-day workweek in Australia: insights from early adopters of the 100:80:100 model
Hopkins, J., Bardoel, E. A., & Djurkovic, N. (2026) · Humanities and Social Sciences Communications · doi:10.1057/s41599-026-07536-x · Cited by 2 · see citing works
- Authors
- Campbell, J. L., Quincy, C., Osserman, J., & Pedersen, O. K.
- Affiliations
- Dartmouth College (US)
- IBM (United States) (US)
- University of London (GB)
- Copenhagen Business School (DK)
- Corresponding author
- Campbell, John L.
- ORCID
- Osserman, Jordan
- Venue
- Sociological Methods & Research
- Publisher
- SAGE Publications
- Published
- 2013-08
- Volume
- 42
- Issue
- 3
- Pages
- 294-320
- ISSN
- 0049-1241 (ISSN-L)
- 1552-8294
- OpenAlex
- W1981425990
- Cited by
- 2,974
- References
- 4
- Citations by year
- 2026: 438 · 2025: 370 · 2024: 320 · 2023: 263 · 2022: 331 · 2021: 341 · 2020: 317 · 2019: 241 · 2018: 147 · 2017: 103 · 2016: 53 · 2015: 35 · 2014: 13 · 2013: 1 · 2012: 1
- FWCI
- 37.02
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Sociological Methods & Research (publishedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- BIROn (Birkbeck, University of London) (submittedVersion)
- CBS Research Portal (Copenhagen Business School) (submittedVersion)
- Funders
- National Science Foundation
- Dartmouth College
- Copenhagen Business School
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Reliability and Agreement in Measurement
- Social Sciences › Education › Qualitative Research Methods and Applications
- Social Sciences › General Social Sciences › Computational and Text Analysis Methods
- Keywords
- coding scheme, interview transcripts, coding reliability, intercoder reliability, inter-coder agreement, coding errors, qualitative data analysis, coding scheme development
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
