Using Pooled Kappa to Summarize Interrater Agreement across Many Items
Abstract
We propose the pooled estimator of kappa, an efficient estimator when summarizing the interrater agreement for qualitative data with many items but few subjects. We evaluate this estimator through a simulation of proposed and alternative (average kappa) estimators and subsequently apply our method to calculate pooled and average kappas over 2,176 rated items from six semistructured interviews with sponsors of the CAHPS. The proposed pooled kappa estimator efficiently summarizes interrater agreement by domain. It is more widely applicable and makes better use of scarce subjects than simply averaging item-level kappas.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Two authors independently coded the qualitative responses, resulting in moderate interrater reliability (pooled kappa = .74).
“Third, two authors independently coded the qualitative responses, resulting in moderate interrater reliability (pooled kappa = .74;De Vries et al., 2008).”
From Billboard-style posters preferred overall at two conferences, but should include more methods and limitations (2024) · cites this paper as De Vries et al., 2008
Cited by
Sorted by citations
Psychedelic therapy for smoking cessation: qualitative analysis of participant accounts
Noorani, T., Garcia-Romeu, A., Swift, T. C., Griffiths, R. R., & Johnson, M. W. (2018) · Journal of Psychopharmacology · doi:10.1177/0269881118780612 · Cited by 285 · see citing works
Intensive meditation training influences emotional responses to suffering
Rosenberg, E. L., Zanesco, A. P., King, B. G., Aichele, S. R., Jacobs, T. L., Bridwell, D. A., MacLean, K. A., Shaver, P. R., Ferrer, E., Sahdra, B. K., et al. (2015) · Emotion · doi:10.1037/emo0000080 · Cited by 70 · see citing works
Uncertainty-aware deep learning in healthcare: A scoping review
Loftus, T. J., Shickel, B., Ruppert, M. M., Balch, J. A., Ozrazgat-Baslanti, T., Tighe, P. J., Efron, P. A., Hogan, W. R., Rashidi, P., Upchurch, G. R., et al. (2022) · PLOS Digital Health · doi:10.1371/journal.pdig.0000085 · Cited by 73 · see citing works
Barone, W., Beck, J., Mitsunaga-Whitten, M., & Perl, P. (2019) · Journal of Psychoactive Drugs · doi:10.1080/02791072.2019.1580805 · Cited by 61 · see citing works
Learning to design for social justice in community‐engaged engineering
Reynante, B. (2022) · Journal of Engineering Education · doi:10.1002/jee.20444 · Cited by 55 · see citing works
Kodish, S. R., Bio, F., Oemcke, R., Conteh, J., Beauliere, J. M., Pyne-Bailey, S., Rohner, F., Ngnie-Teta, I., Jalloh, M. B., & Wirth, J. P. (2019) · PLOS Neglected Tropical Diseases · doi:10.1371/journal.pntd.0007645 · Cited by 44 · see citing works
Anderson, A., Kollmann, E. K., Beyer, M., Weitzman, O., Bequette, M., Haupt, G., & Velázquez, H. (2021) · Journal of Chemical Education · doi:10.1021/acs.jchemed.1c00193 · Cited by 26 · see citing works
Pimentel, D. R. (2025) · Journal of Research in Science Teaching · doi:10.1002/tea.21974 · Cited by 28 · see citing works
Oosterhoff, B., Wray-Lake, L., & Harden, K. P. (2022) · Developmental Psychology · doi:10.1037/dev0001366 · Cited by 7 · see citing works
Oronje, B., Morrison, M., Suharlim, C., Folkman, K., Glaude-Hosh, A., & Jeisy-Scott, V. (2024) · doi:10.32388/p7n5bo.2 · Cited by 2 · see citing works
- Authors
- De Vries, H., Elliott, M. N., Kanouse, D. E., & Teleki, S. S.
- Affiliations
- RAND Europe
- RAND Corporation (US)
- ORCID
- Elliott, Marc N.
- Venue
- Field Methods
- Publisher
- SAGE Publications
- Published
- 2008-03-03
- Volume
- 20
- Issue
- 3
- Pages
- 272-282
- ISSN
- 1064-8631 (ISSN-L)
- 1525-822X
- 1552-3969
- OpenAlex
- W2160134066
- Cited by
- 254
- References
- 0
- Citations by year
- 2026: 21 · 2025: 17 · 2024: 24 · 2023: 24 · 2022: 34 · 2021: 31 · 2020: 25 · 2019: 16 · 2018: 17 · 2017: 14 · 2016: 5 · 2015: 5 · 2014: 9 · 2013: 5 · 2012: 3
- FWCI
- 1.41
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Field Methods (publishedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Reliability and Agreement in Measurement
- Social Sciences › Sociology and Political Science › Survey Methodology and Nonresponse
- Social Sciences › Decision Sciences › Management Science and Operations Research › Psychometric Methodologies and Testing
- Keywords
- interobserver agreement, inter-rater reliability, qualitative data, semi-structured interviews, CAHPS, customer satisfaction measurement, small sample size
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
