Process guidelines for establishing Intercoder Reliability in qualitative studies
Open accessAbstract
Qualitative interviews are increasingly being utilized within the context of intervention trials. While there is emerging assistance for conducting and reporting qualitative analysis, there are limited practical resources available for researchers engaging in a group coding process and interested in ensuring adequate Intercoder Reliability (ICR); the amount of agreement between two or more coders for the codes applied to qualitative text. Assessing the reliability of the coding helps establish the credibility of qualitative findings. We discuss our experience calculating ICR in the context of a behavioural HIV prevention trial for young women in South Africa which involves multiple rounds of longitudinal qualitative data collection. We document the steps that we took to improve ICR in this study, the challenges to improving ICR, and the value of the process to qualitative data analysis. As a result, we provide guidelines for other researchers to consider as they embark on large qualitative projects.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The iterative process of calculating inter-coder reliability identified places where coding was divergent and was used to guide consensus meetings between coders, increasing the trustworthiness of the analyses.
“This iterative process of calculating inter-coder reliability identified places where coding was divergent (e.g., disagreement about the application of a code), and was used to guide consensus meetings between coders (Macphail et al., 2016), increasing the trustworthiness of the analyses.”
From Exposure Therapy Acceptability Among Pregnant Latinas With Anxiety: A Qualitative Content Analysis (2025) · cites this paper as Macphail et al., 2016
Cites
How Many Interviews Are Enough?: An Experiment with Data Saturation and Variability
Guest, G., Bunce, A., & Johnson, L. (2006) · Field Methods · doi:10.1177/1525822x05279903 · Cited by 20,808 · see citing works
Qualitative research in health care: Analysing qualitative data
Pope, C. (2000) · BMJ · doi:10.1136/bmj.320.7227.114 · Cited by 8,131 · see citing works
Answering the Call for a Standard Reliability Measure for Coding Data
Hayes, A. F., & Krippendorff, K. (2007) · Communication Methods and Measures · doi:10.1080/19312450709336664 · Cited by 4,225 · cited 1× · see citing works
“Krippendorf’s alpha is better able to correct for chance and is equipped to handle multiple coders (Hayes and Krippendorff, 2007), but is computationally and conceptually difficult.”
BackgroundContent Analysis in Mass Communication: Assessment and Reporting of Intercoder Reliability
Lombard, M., Snyder-Duch, J., & Bracken, C. C. (2002) · Human Communication Research · doi:10.1111/j.1468-2958.2002.tb00826.x · Cited by 3,068 · see citing works
Codebook Development for Team-Based Qualitative Analysis
MacQueen, K. M., McLellan, E., Kay, K., & Milstein, B. (1998) · CAM Journal · doi:10.1177/1525822x980100020301 · Cited by 1,675 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Intercoder Reliability in Qualitative Research: Debates and Practical Guidelines
O’Connor, C., & Joffe, H. (2020) · International Journal of Qualitative Methods · doi:10.1177/1609406919899220 · Cited by 3,871 · see citing works
“This can cause major difficulty in any form of automated ICR calculation: When coders’ selection of text segments varies by a mere digit or punctuation mark, automated systems can record these as different data units and the calculated ICR coefficient is compromised (MacPhail et al., 2016).”
BackgroundMethodInfluentialSmith, B., & McGannon, K. R. (2018) · International Review of Sport and Exercise Psychology · doi:10.1080/1750984x.2017.1317357 · Cited by 3,074 · see citing works
McDonald, N., Schoenebeck, S., & Forte, A. (2019) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3359174 · Cited by 1,206 · see citing works
Qualitative research in nursing and health professions regulation
Squires, A., & Dorsen, C. (2018) · Journal of Nursing Regulation · doi:10.1016/s2155-8256(18)30150-9 · Cited by 108 · see citing works
Kodish, S. R., Bio, F., Oemcke, R., Conteh, J., Beauliere, J. M., Pyne-Bailey, S., Rohner, F., Ngnie-Teta, I., Jalloh, M. B., & Wirth, J. P. (2019) · PLOS Neglected Tropical Diseases · doi:10.1371/journal.pntd.0007645 · Cited by 44 · see citing works
Exposure Therapy Acceptability Among Pregnant Latinas With Anxiety: A Qualitative Content Analysis
Ponting, C., Ong, E., Dunkel Schetter, C., & Chavira, D. A. (2025) · Cultural Diversity & Ethnic Minority Psychology · doi:10.1037/cdp0000589 · Cited by 3 · see citing works
Landwehr, E., Roberts, L., Garratt-Reed, D., & Maxwell-Smith, C. (2025) · International Journal of Offender Therapy and Comparative Criminology · doi:10.1177/0306624x241228977 · Cited by 2 · see citing works
- Authors
- MacPhail, C., Khoza, N., Abler, L., & Ranganathan, M.
- Affiliations
- University of New England (AU)
- University of the Witwatersrand (ZA)
- Duke University (US)
- London School of Hygiene & Tropical Medicine (GB)
- Corresponding author
- MacPhail, Catherine
- Venue
- Qualitative Research
- Publisher
- SAGE Publications
- Published
- 2015-04-20
- Volume
- 16
- Issue
- 2
- Pages
- 198-212
- ISSN
- 1468-7941 (ISSN-L)
- 1741-3109
- OpenAlex
- W1985181430
- Cited by
- 534
- References
- 5
- Citations by year
- 2026: 85 · 2025: 78 · 2024: 62 · 2023: 70 · 2022: 73 · 2021: 57 · 2020: 56 · 2019: 31 · 2018: 13 · 2017: 6 · 2016: 2
- FWCI
- 6.39
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Locations
- Qualitative Research (publishedVersion)
- Research Online (University of Wollongong) (submittedVersion)
- LSHTM Research Online (London School of Hygiene and Tropical Medicine) (submittedVersion)
- RUNE (Research UNE) (submittedVersion)
- Funders
- University of New England
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Reliability and Agreement in Measurement
- Social Sciences › Education › Qualitative Research Methods and Applications
- Social Sciences › General Social Sciences › Computational and Text Analysis Methods
- Keywords
- intercoder reliability, qualitative data analysis, coding agreement, HIV prevention, young women, process guidelines
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
