Interrater Reliability in Systematic Review Methodology: Exploring Variation in Coder Decision-Making
Open accessAbstract
A methodologically sound systematic review is characterized by transparency, replicability, and a clear inclusion criterion. However, little attention has been paid to reporting the details of interrater reliability (IRR) when multiple coders are used to make decisions at various points in the screening and data extraction stages of a study. Prior research has mentioned the paucity of information on IRR including number of coders involved, at what stages and how IRR tests were conducted, and how disagreements were resolved. This article examines and reflects on the human factors that affect decision-making in systematic reviews via reporting on three IRR tests, conducted at three different points in the screening process, for two distinct reviews. Results of the two studies are discussed in the context of IRR and intrarater reliability in terms of the accuracy, precision, and reliability of coding behavior of multiple coders. Findings indicated that coding behavior changes both between and within individuals over time, emphasizing the importance of conducting regular and systematic IRR and intrarater reliability tests, especially when multiple coders are involved, to ensure consistency and clarity at the screening and coding stages. Implications for good practice while screening/coding for systematic reviews are discussed.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Two researchers independently and in duplicate searched, reviewed titles and abstracts, checked full texts, and selected articles, with inter-rater reliability k of at least 0.9 in all data extraction steps.
“Two researchers (RZ, FC) searched the papers, reviewed the titles and abstracts of the retrieved articles separately and in duplicate, checked the full texts, and selected the articles for inclusion in the study.”
Most systematic reviews do not measure or report inter-coder reliability (ICR), nor do they provide transparency in how their coding processes are executed.
“It is worth noting that most systematic reviews do not measure or report ICR, nor do they provide transparency in how their coding processes are executed [165].”
From A systematic review of echo chamber research: comparative analysis of conceptualizations, operationalizations, and varying outcomes (2025) · cites this paper as [165]
Cites
The Measurement of Observer Agreement for Categorical Data
Landis, J. R., & Koch, G. G. (1977) · Biometrics · doi:10.2307/2529310 · Cited by 81,148 · see citing works
Qualitative Data Analysis: An Expanded Sourcebook
(1994) · Journal of Environmental Psychology · doi:10.1016/s0272-4944(05)80231-2 · Cited by 48,420 · see citing works
Content Analysis: An Introduction to its Methodology.
Shelley, M., & Krippendorff, K. (1984) · Journal of the American Statistical Association · doi:10.2307/2288384 · Cited by 24,305 · see citing works
Interrater reliability: The kappa statistic
McHugh, M. L. (2012) · Biochemia Medica · doi:10.11613/bm.2012.031 · Cited by 19,877 · see citing works
(2005) · PubMed · Cited by 7,011 · see citing works
Content Analysis in Mass Communication: Assessment and Reporting of Intercoder Reliability
Lombard, M., Snyder-Duch, J., & Bracken, C. C. (2002) · Human Communication Research · doi:10.1111/j.1468-2958.2002.tb00826.x · Cited by 3,068 · see citing works
Campbell, J. L., Quincy, C., Osserman, J., & Pedersen, O. K. (2013) · Sociological Methods & Research · doi:10.1177/0049124113500475 · Cited by 2,974 · see citing works
All Frames Are Not Created Equal: A Typology and Critical Analysis of Framing Effects
Levin, I. P., Schneider, S. L., & Gaeth, G. J. (1998) · Organizational Behavior and Human Decision Processes · doi:10.1006/obhd.1998.2804 · Cited by 2,592 · see citing works
Computing inter‐rater reliability and its variance in the presence of high agreement
Gwet, K. L. (2008) · British Journal of Mathematical and Statistical Psychology · doi:10.1348/000711006x126600 · Cited by 2,167 · see citing works
High agreement but low kappa: II. Resolving the paradoxes
Cicchetti, D. V., & Feinstein, A. R. (1990) · Journal of Clinical Epidemiology · doi:10.1016/0895-4356(90)90159-m · Cited by 1,838 · see citing works
Kahneman, D. J., & Tversky, A. (1982) · Scientific American · doi:10.1038/scientificamerican0182-160 · Cited by 1,699 · see citing works
Rethinking validity and reliability in content analysis
Potter, W. J., & Levine‐Donnerstein, D. (1999) · Journal of Applied Communication Research · doi:10.1080/00909889909365539 · Cited by 1,662 · see citing works
Kolbe, R. H., & Burnett, M. S. (1991) · Journal of Consumer Research · doi:10.1086/209256 · Cited by 1,516 · see citing works
Introduction to Systematic Reviews
(2013) · Active Learning Exercises · doi:10.21019/ale.2000.252 · Cited by 1,239 · see citing works
The Place of Inter-Rater Reliability in Qualitative Research: An Empirical Study
Armstrong, D., Gosling, A., Weinman, J., & Marteau, T. (1997) · Sociology · doi:10.1177/0038038597031003015 · Cited by 1,008 · see citing works
Borah, R., Brown, A. W., Capers, P. L., & Kaiser, K. A. (2017) · BMJ Open · doi:10.1136/bmjopen-2016-012545 · Cited by 737 · cited 3× · see citing works
“A recent study (Borah et al. 2017) on the resources required to conduct systematic reviews indicated that the mean number of members involved in conducting a systematic review was 5 (SD 1⁄4 3) and the average time period required to conduct and publish a review was 67.”
BackgroundInfluentialAssessing intrarater, interrater, and test-retest reliability of continuous measurements
Rousson, V., Gasser, T., & Seifert, B. (2002) · Statistics in Medicine · doi:10.1002/sim.1253 · Cited by 415 · see citing works
Clark, T. (2008) · Sociology · doi:10.1177/0038038508094573 · Cited by 367 · see citing works
THE DATA ARE OUT THERE, OR ARE THEY? IMPLICATIONS FOR ARCHIVING AND REVISITING QUALITATIVE DATA
Mauthner, N. S., Parry, O., & Backett-Milburn, K. C. (1998) · Sociology · doi:10.1177/0038038598032004006 · Cited by 308 · see citing works
"Perfectly Healthy, but Dead": The Myth of Inter-Rater Reliability
Morse, J. M. (1997) · Qualitative Health Research · doi:10.1177/104973239700700401 · Cited by 246 · see citing works
Deep thoughts and shallow frames: on the susceptibility to framing effects
LeBoeuf, R. A., & Shafir, E. B. (2003) · Journal of Behavioral Decision Making · doi:10.1002/bdm.433 · Cited by 239 · see citing works
Intercoder reliability indices: disuse, misuse, and abuse
Feng, G. C. (2013) · Quality & Quantity · doi:10.1007/s11135-013-9956-8 · Cited by 117 · see citing works
Negotiating Expectations in the Field: Gatekeepers, Research Fatigue and Cultural Biases
Mandel, J. L. (2003) · Singapore Journal of Tropical Geography · doi:10.1111/1467-9493.00152 · Cited by 73 · see citing works
Information retrieval in systematic reviews: a case study of the crime prevention literature
Tompson, L., & Belur, J. (2015) · Journal of Experimental Criminology · doi:10.1007/s11292-015-9243-x · Cited by 26 · see citing works
Self-citationPetrosino, A. J. (1995) · Journal of Crime and Justice · doi:10.1080/0735648x.1995.9721049 · Cited by 24 · see citing works
Ordóñez (2000) · doi:10.1002/1099-0771(200007/09)13:3
Flagged first, then by citations
Cited by
Sorted by citations
Artificial intelligence in higher education: the state of the field
Crompton, H., & Burke, D. (2023) · International Journal of Educational Technology in Higher Education · doi:10.1186/s41239-023-00392-8 · Cited by 2,010 · see citing works
“The inter-rater reliability was calculated by percentage agreement (Belur et al., 2018).”
MethodAffordances and challenges of artificial intelligence in K-12 education: a systematic review
Crompton, H., Jones, M. V., & Burke, D. (2024) · Journal of Research on Technology in Education · doi:10.1080/15391523.2022.2121344 · Cited by 213 · see citing works
Inter-Coder Agreement in Qualitative Coding: Considerations for its Use
Halpin, S. N. (2024) · American Journal of Qualitative Research · doi:10.29333/ajqr/14887 · Cited by 149 · see citing works
Singh, J., Karanika-Murray, M., Baguley, T., & Hudson, J. (2020) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph17196987 · Cited by 132 · see citing works
The Educational Affordances and Challenges of ChatGPT: State of the Field
Crompton, H., & Burke, D. (2024) · TechTrends · doi:10.1007/s11528-024-00939-0 · Cited by 141 · see citing works
“Calculated using percentage agreement (Belur et al., 2021), researchers arrived at a 97% inter-rater agreement.”
Paulsen, L., & Lindsay, E. (2024) · Education and Information Technologies · doi:10.1007/s10639-023-12401-4 · Cited by 111 · see citing works
Pallarés, J. G., Hernández‐Belmonte, A., Martínez‐Cava, A., Vetrovsky, T., Steffl, M., & Courel‐Ibáñez, J. (2021) · Scandinavian Journal of Medicine & Science in Sports · doi:10.1111/sms.14006 · Cited by 80 · see citing works
A scoping review of school-based efforts to support students who have experienced trauma.
Stratford, B., Cook, E., Hanneke, R., Katz, E., Seok, D., Steed, H., Fulks, E., Lessans, A., & Temkin, D. (2020) · School Mental Health · doi:10.1007/s12310-020-09368-9 · Cited by 83 · see citing works
Staying on target: A systematic literature review on learner‐facing learning analytics dashboards
Valle, N., Antonenko, P., Dawson, K., & Huggins‐Manley, A. C. (2021) · British Journal of Educational Technology · doi:10.1111/bjet.13089 · Cited by 79 · see citing works
Rodríguez-Rivas, M. E., Cangas, A. J., Cariola, L. A., Varela, J. J., & Valdebenito, S. (2022) · JMIR Serious Games · doi:10.2196/35099 · Cited by 68 · see citing works
Hartmann, D., Wang, S. M., Pohlmann, L., & Berendt, B. (2025) · Journal of Computational Social Science · doi:10.1007/s42001-025-00381-z · Cited by 59 · see citing works
Sommet, N., Weissman, D. L., Cheutin, N., & Elliot, A. (2022) · doi:10.31219/osf.io/xhe3u · Cited by 36 · see citing works
Zupo, R., Castellana, F., Crupi, P., Desantis, A., Rondanelli, M., Corbo, F., & Clodoveo, M. L. (2023) · Metabolites · doi:10.3390/metabo13121187 · Cited by 30 · see citing works
“According to PRISMA concepts and the quality assessment steps, a coefficient k of at least 0.9 was obtained in all data extraction steps [28].”
MethodInfluentialMachine learning to optimize literature screening in medical guideline development
Harmsen, W., de Groot, J., Harkema, A., van Dusseldorp, I., de Bruin, J., van den Brand, S., & van de Schoot, R. (2024) · Systematic Reviews · doi:10.1186/s13643-024-02590-5 · Cited by 21 · see citing works
Decoding machine learning in nursing research: A scoping review of effective algorithms
Choi, J., Lee, H., & Kim‐Godwin, Y. (2025) · Journal of Nursing Scholarship · doi:10.1111/jnu.13026 · Cited by 15 · see citing works
Arendsen, J. L., Brugman, B. C., van Koningsbruggen, G. M., & Fransen, M. L. (2025) · International Journal of Consumer Studies · doi:10.1111/ijcs.70056 · Cited by 4 · see citing works
A noise audit of the peer review of a scientific article: a WPOM journal case study
Bonavia, T., & Marin-Garcia, J. A. (2023) · WPOM-Working Papers on Operations Management · doi:10.4995/wpom.19631 · Cited by 3 · see citing works
- Authors
- Belur, J., Tompson, L., Thornton, A., & Simon, M.
- Affiliations
- University College London (GB)
- Corresponding author
- Belur, Jyoti
- Venue
- Sociological Methods & Research
- Publisher
- SAGE Publications
- Published
- 2018-09-24
- Volume
- 50
- Issue
- 2
- Pages
- 837-865
- ISSN
- 0049-1241 (ISSN-L)
- 1552-8294
- OpenAlex
- W2893300601
- Cited by
- 753
- References
- 26
- Citations by year
- 2026: 148 · 2025: 151 · 2024: 129 · 2023: 120 · 2022: 93 · 2021: 72 · 2020: 33 · 2019: 7
- FWCI
- 5.57
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Sociological Methods & Research (publishedVersion, cc-by-nc)
- RePEc: Research Papers in Economics (submittedVersion)
- UCL Discovery (University College London) (submittedVersion)
- Funders
- College of Policing (ES/L007223/1)
- Economic and Social Research Council (ES/L007223/1)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Reliability and Agreement in Measurement
- Social Sciences › Sociology and Political Science › Delphi Technique in Research
- Keywords
- inter-rater reliability, systematic reviews, data extraction, study screening, intra-rater reliability, reporting practices, disagreement resolution
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
