CoChat
Sign Up
Cited by 1,206Qualitative Research Methods and ApplicationsReliability and Agreement in MeasurementComputational and Text Analysis Methods

Reliability and Inter-rater Reliability in Qualitative Research: Norms and Guidelines for CSCW and HCI Practice

Open access
McDonald, Nora, Schoenebeck, Sarita, Forte, Andrea·2019·Proceedings of the ACM on Human-Computer Interaction
DOI 10.1145/3359174
Open this paper in CoChat — free accountOpen PDF

Abstract

What does reliability mean for building a grounded theory? What about when writing an auto-ethnography? When is it appropriate to use measures like inter-rater reliability (IRR)? Reliability is a familiar concept in traditional scientific practice, but how, and even whether to establish reliability in qualitative research is an oft-debated question. For researchers in highly interdisciplinary fields like computer-supported cooperative work (CSCW) and human-computer interaction (HCI), the question is particularly complex as collaborators bring diverse epistemologies and training to their research. In this article, we use two approaches to understand reliability in qualitative research. We first investigate and describe local norms in the CSCW and HCI literature, then we combine examples from these findings with guidelines from methods literature to help researchers answer questions like: "should I calculate IRR?" Drawing on a meta-analysis of a representative sample of CSCW and HCI papers from 2016-2018, we find that authors use a variety of approaches to communicate reliability; notably, IRR is rare, occurring in around 1/9 of qualitative papers. We reflect on current practices and propose guidelines for reporting on reliability in qualitative research using IRR as a central example of a form of agreement. The guidelines are designed to generate discussion and orient new CSCW and HCI scholars and reviewers to reliability in qualitative research.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1145/3359174

Cites

95 of 95 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • To ensure reliability in the analysis, the authors held several group meetings where they compared codes and discussed differences as themes emerged.

    “To ensure reliability in our analysis, we held several group meetings where authors compared codes and discussed differences as the themes began to emerge, as is appropriate during reflexive thematic analysis [61].”

    From “It’s Weird That it Knows What I Want”: Usability and Interactions with Copilot for Novice Programmers (2024) · cites this paper as [61]

    Section 5.1.1 describes group meetings to compare codes, discuss differences, and develop emergent themes; §5.2.3 says discussing disagreements can strengthen the coding process. These passages support the cited rationale for the meetings, but do not specifically establish the practice as appropriate for reflexive thematic analysis. The citing authors’ statement that they held meetings is their own report, not a claim about the source paper. (LLM, conf=0.83)

    Experimental — accuracy not yet measured

  • Iterative coding and comparing quotations to resolve disagreements helped maintain codebook reliability.

    “Iterative coding and comparing quotations to resolve disagreements helped maintain codebook reliability [19].”

    From Studying Exploration & Long-Term Use of Voice Assistants by Older Adults (2023) · cites this paper as [19]

    Section 4.1 describes iterative discussions to resolve coding disagreements and refine codebook definitions, and §5.2.3 says this can strengthen the codebook and coding process. The paper does not specifically describe comparing quotations as the means of resolving disagreements or say that this maintains codebook reliability. (LLM, conf=0.82)

    Experimental — accuracy not yet measured

  • The first author coded all transcripts and developed themes through discussion with the second author, who validated that themes represented participants' views.

    “The first author coded all transcripts and developed themes through discussion with the second author, who validated that themes represented participants’ views [22].”

    From 'AI-Induced Delusional Spirals': Understanding Lived Experiences During Maladaptive Human-Chatbot Interactions (2026) · cites this paper as [22]

    Section 4.1 describes qualitative studies in which one researcher codes data and then meets with coauthors to discuss and refine themes, supporting the cited methodological approach. It does not establish that the citing paper’s first author coded all transcripts or that its second author validated the themes against participants’ views; those are details of the citing authors’ own work. (LLM, conf=0.85)

    Experimental — accuracy not yet measured

  • Disagreements were negotiated and resolved at each stage, using negotiated agreement best practices.

    “Disagreements were negotiated and resolved at each stage, using negotiated agreement best practices [87].”

    From The Impact of Generative AI on Critical Thinking: Self-Reported Reductions in Cognitive Effort and Confidence Effects From a Survey of Knowledge Workers (2025) · cites this paper as [87]

  • The qualitative data analysis used reflexive Thematic Analysis following Clarke and Braun [9] and McDonald et al. [52].

    “For qualitative data, we conducted a reflexive Thematic Analysis (TA) following Clarke and Braun [9] and McDonald et al. [52] to identify patterns and understand parental perspectives.”

    From SET-PAiREd: Designing for Parental Involvement in Learning with an AI-Assisted Educational Robot (2025) · cites this paper as [52]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 4

  1. [31]31. Robert M. Emerson, Rachel I. Fretz, and Linda L. Shaw. 2011. Writing Ethnographic Fieldnotes, Second Edition (Second edition ed.). University of Chicago Press, Chicago.
  2. [37]37. Lisa Given. 2008. Natural Setting. In The Sage encyclopedia of qualitative research methods. SAGE, London.
  3. [74]74. Judith S. Olson and Wendy A. Kellogg (Eds.). 2014. Ways of Knowing in HCI. Springer-Verlag, New York.
  4. [90]90. G. Cochran William. 1977. Sampling Techniques, 3rd Edition (3rd edition ed.). John Wiley & Sons, New York.

Cited by

Sorted by citations

Related

Authors
McDonald, N., Schoenebeck, S., & Forte, A.
Affiliations
  • Drexel University (US)
  • University of Michigan (US)
ORCID
Venue
Proceedings of the ACM on Human-Computer Interaction
Publisher
Association for Computing Machinery (ACM)
Published
2019-11-07
Volume
3
Issue
CSCW
Pages
1-23
ISSN
2573-0142 (ISSN-L)
DOI
10.1145/3359174
OpenAlex
W3017863658
Cited by
1,206
References
95
Citations by year
2026: 235 · 2025: 278 · 2024: 209 · 2023: 176 · 2022: 136 · 2021: 130 · 2020: 42
FWCI
207.91
Type
journal article
Language
en
Open access
Licence
www.acm.org/publications/policies/copyright_policy#Background (vor)
Locations
Proceedings of the ACM on Human-Computer Interaction (publishedVersion)
Funders
NSF (CNS-1703736)
Topics
  • Social Sciences › Education › Qualitative Research Methods and Applications
  • Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Reliability and Agreement in Measurement
  • Social Sciences › General Social Sciences › Computational and Text Analysis Methods
Keywords
reliability statistics, inter-rater reliability, qualitative research, grounded theory, autoethnography, reporting guidelines, computer-supported cooperative work, human-computer interaction, epistemology, research reproducibility

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review