The Place of Inter-Rater Reliability in Qualitative Research: An Empirical Study
Abstract
Assessing inter-rater reliability, whereby data are independently coded and the codings compared for agreements, is a recognised process in quantitative research. However, its applicability to qualitative research is less clear: should researchers be expected to identify the same codes or themes in a transcript or should they be expected to produce different accounts? Some qualitative researchers argue that assessing inter-rater reliability is an important method for ensuring rigour, others that it is unimportant; and yet it has never been formally examined in an empirical qualitative study. Accordingly, to explore the degree of inter-rater reliability that might be expected, six researchers were asked to identify themes in the same focus group transcript. The results showed close agreement on the basic themes but each analyst `packaged' the themes differently.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Armstrong et al found considerable variation in the ways that six experienced researchers who independently coded one focus group transcript packaged coding frameworks, including the language used, despite showing substantial agreement.
“Although six experienced researchers who independently coded one focus group transcript showed substantial agreement, Armstrong et al found considerable variation in the ways that they packaged coding frameworks (including the language used).16”
From Checklists for improving rigour in qualitative research: a case of the tail wagging the dog? (2001) · cites this paper as [16]
Cites
Wagner, H. R., Glaser, B. G., & Strauss, A. L. (1968) · Social Forces · doi:10.2307/2575405 · Cited by 46,470 · see citing works
Designing qualitative research
Shaliha, F., Mozaffari, M., Ramezani, F., Hajnasiri, H., & Moafi, F. (1989) · Choice Reviews Online · doi:10.5860/choice.27-1232 · Cited by 9,914 · see citing works
Handbook of qualitative research
Griffin, T. F. (1999) · Psychology and Marketing · doi:10.1002/(sici)1520-6793(199907)16:4 · Cited by 9,061 · see citing works
The Research Act: A Theoretical Introduction to Sociological Methods
Fox, W. S., & Denzin, N. K. (1979) · Contemporary Sociology A Journal of Reviews · doi:10.2307/2065439 · Cited by 6,291 · see citing works
Writing Culture: The Poetics and Politics of Ethnography
Clifford, J., & Marcus, G. E. (1986) · Cited by 6,160 · see citing works
Introduction : Entering the field of qualitative research
Denzin, N. (1994) · Medical Entomology and Zoology · Cited by 2,910 · see citing works
Qualitative Research: Rigour and qualitative research
Mays, N., & Pope, C. (1995) · BMJ · doi:10.1136/bmj.311.6997.109 · Cited by 2,715 · see citing works
Designing funded qualitative research.
Morse, J. M. (1994) · Cited by 1,981 · see citing works
In the Field: An Introduction to Field Research
Burgess, R. G. (1984) · Medical Entomology and Zoology · Cited by 1,552 · see citing works
(2002) · Routledge eBooks · doi:10.4324/9780203413081 · Cited by 1,459 · see citing works
Post-Modern Ethnography: From Document of the Occult to Occult Document
Tyler, S. A. (1986) · University of California Press eBooks · doi:10.1525/9780520946286-008 · Cited by 691 · see citing works
Burgess, R. G. (2002) · Routledge eBooks · doi:10.4324/9780203418161 · Cited by 661 · see citing works
The Research Act: A Theoretical Introduction to Sociological Methods.
Bylund, H. B., & Denzin, N. K. (1972) · Contemporary Sociology A Journal of Reviews · doi:10.2307/2063497 · Cited by 493 · see citing works
Designing Qualitative Research
Farr, B. C. (2008) · Transformation An International Journal of Holistic Mission Studies · doi:10.1177/026537880802500310 · Cited by 389 · see citing works
Hammersley, M. (1991) · doi:10.4324/9781315538457 · Cited by 338 · see citing works
Harris, I. B. (2006) · MedEdPORTAL · doi:10.15766/mep_2374-8265.227 · Cited by 175 · see citing works
Moscovici, S. (2015) · doi:10.5040/9781666984972.ch-006 · Cited by 175 · see citing works
Analyzing together: recollections of a team approach
Turner, B. Α. (2002) · Routledge eBooks · doi:10.4324/9780203413081-11 · Cited by 54 · see citing works
Researching health care. Designs, dilemmas, disciplines
Black, N. D. (1993) · Journal of Epidemiology & Community Health · doi:10.1136/jech.47.4.339 · Cited by 52 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Qualitative research in health care: Analysing qualitative data
Pope, C. (2000) · BMJ · doi:10.1136/bmj.320.7227.114 · Cited by 8,131 · see citing works
Intercoder Reliability in Qualitative Research: Debates and Practical Guidelines
O’Connor, C., & Joffe, H. (2020) · International Journal of Qualitative Methods · doi:10.1177/1609406919899220 · Cited by 3,871 · see citing works
“It is generally accepted that different analysts, with different theoretical commitments, will organize codes into themes in different ways (Armstrong et al., 1997).”
BackgroundChecklists for improving rigour in qualitative research: a case of the tail wagging the dog?
Barbour, R. S. (2001) · BMJ · doi:10.1136/bmj.322.7294.1115 · Cited by 2,613 · see citing works
McDonald, N., Schoenebeck, S., & Forte, A. (2019) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3359174 · Cited by 1,206 · see citing works
Reliability in Coding Open-Ended Data: Lessons Learned from HIV Behavioral Research
Hruschka, D. J., Schwartz, D., St.John, D. C., Picone-Decaro, E., Jenkins, R. A., & Carey, J. W. (2004) · Field Methods · doi:10.1177/1525822x04266540 · Cited by 829 · see citing works
Belur, J., Tompson, L., Thornton, A., & Simon, M. (2021) · Sociological Methods & Research · doi:10.1177/0049124118799372 · Cited by 753 · see citing works
Lau, R., Stevenson, F., Ong, B. N., Dziedzic, K., Treweek, S., Eldridge, S., Everitt, H., Kennedy, A., Qureshi, N., Rogers, A., et al. (2015) · Implementation Science · doi:10.1186/s13012-016-0396-4 · Cited by 533 · see citing works
“controversial [98, 99].”
A team-based approach to open coding: considerations for creating intercoder consensus
Cascio, M. A., Lee, E., Vaudrin, N., & Freedman, D. A. (2019) · Field Methods · doi:10.1177/1525822x19838237 · Cited by 492 · see citing works
Is social media bad for mental health and wellbeing? Exploring the perspectives of adolescents
O’Reilly, M., Dogra, N., Whiteman, N., Hughes, J., Eruyar, S., & Reilly, P. (2018) · Clinical Child Psychology and Psychiatry · doi:10.1177/1359104518775154 · Cited by 446 · see citing works
Rose, J., & Johnson, C. W. (2020) · Journal of Leisure Research · doi:10.1080/00222216.2020.1722042 · Cited by 419 · see citing works
Examining menstrual tracking to inform the design of personal informatics tools
Epstein, D. A., Lee, N. B., Kang, J. H., Agapie, E., Schroeder, J., Pina, L. R., Fogarty, J., Kientz, J. A., & Munson, S. (2017) · Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3025453.3025635 · Cited by 346 · see citing works
The impact of caring for a child with cerebral palsy: quality of life for mothers and fathers
Davis, E., Shelly, A., Waters, E., Boyd, R., Cook, K., & Davern, M. (2010) · Child: Care, Health and Development · doi:10.1111/j.1365-2214.2009.00989.x · Cited by 313 · see citing works
“Thinking too much”: a systematic review of a common idiom of distress
Kaiser, B. N., Haroz, E. E., Kohrt, B. A., Bolton, P. A., Bass, J. K., & Hinton, D. E. (2015) · Social Science & Medicine · doi:10.1016/j.socscimed.2015.10.044 · Cited by 287 · see citing works
Throuvala, M. A., Griffiths, M. D., Rennoldson, M., & Kuss, D. J. (2019) · Computers in Human Behavior · doi:10.1016/j.chb.2018.12.012 · Cited by 272 · see citing works
Inter-Coder Agreement in Qualitative Coding: Considerations for its Use
Halpin, S. N. (2024) · American Journal of Qualitative Research · doi:10.29333/ajqr/14887 · Cited by 149 · see citing works
Noonan, E. J., Sawning, S., Combs, R., Weingartner, L. A., Martin, L. J., Jones, V. F., & Holthouser, A. (2018) · Teaching and Learning in Medicine · doi:10.1080/10401334.2017.1365718 · Cited by 119 · see citing works
The newfound credibility of qualitative research? tales of technical essentialism and co-option
Barbour, R. S. (2003) · Qualitative Health Research · doi:10.1177/1049732303253331 · Cited by 108 · see citing works
Jones, V. A., & Reddick, R. J. (2017) · The Journal of Negro Education · doi:10.7709/jnegroeducation.86.3.0204 · Cited by 62 · see citing works
Emoji Accessibility for Visually Impaired People
Tigwell, G. W., Gorman, B. M., & Menzies, R. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376267 · Cited by 42 · see citing works
McCormack, L., & Thomson, S. (2017) · Psychological Trauma: Theory, Research, Practice, and Policy · doi:10.1037/tra0000193 · Cited by 29 · see citing works
Examining How Students with Visual Impairments Navigate Accessible Documents
Singleton, K. J., & Neuber, K. S. (2020) · Journal of Visual Impairment & Blindness · doi:10.1177/0145482x20953268 · Cited by 18 · see citing works
Kim, N., Homer, M., & Jang, H. (2025) · American Journal of Speech-Language Pathology · doi:10.1044/2025_ajslp-24-00464 · Cited by 9 · see citing works
Soe, T. (2024) · Theory and Practice of Second Language Acquisition · doi:10.31261/tapsla.13882 · Cited by 5 · see citing works
“By inter-rater reliability, the same content is independently coded by the two or more coders and the findings are compared for agreement (Armstrong et al., 1997).”
Background
- Authors
- Armstrong, D., Gosling, A., Weinman, J., & Marteau, T.
- Venue
- Sociology
- Publisher
- SAGE Publications
- Published
- 1997-08
- Volume
- 31
- Issue
- 3
- Pages
- 597-606
- ISSN
- 0038-0385 (ISSN-L)
- 1469-8684
- OpenAlex
- W2021023566
- Cited by
- 1,008
- References
- 19
- Citations by year
- 2026: 57 · 2025: 50 · 2024: 52 · 2023: 60 · 2022: 84 · 2021: 83 · 2020: 81 · 2019: 81 · 2018: 68 · 2017: 67 · 2016: 62 · 2015: 44 · 2014: 51 · 2013: 36 · 2012: 22
- FWCI
- 14.95
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Sociology (publishedVersion)
- Topics
- Social Sciences › Education › Qualitative Research Methods and Applications
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Reliability and Agreement in Measurement
- Social Sciences › General Social Sciences › Computational and Text Analysis Methods
- Keywords
- inter-rater reliability, qualitative research, coding agreement, theme identification, scientific rigor
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
