Interrater Reliability in Systematic Review Methodology: Exploring Variation in Coder Decision-Making
Open accessThe DOI matches its Crossref record.
Abstract
A methodologically sound systematic review is characterized by transparency, replicability and a clear inclusion criteria. However, little attention has been paid to reporting the details of inter-rater reliability (IRR) when multiple coders are used to make decisions at various points in the screening and data extraction stages of a study. Prior research has mentioned the paucity of information on IRR, including number of coders involved, at what stages and how IRR tests were conducted, and how disagreements were resolved. This paper examines and reflects on the human factors that affect decision-making in systematic reviews via reporting on three IRR tests, conducted at three different points in the screening process, for two distinct reviews. Results of the two studies are discussed in the context of inter rater and intra rater reliability in terms of the accuracy, precision and reliability of coding behaviour of multiple coders. Findings indicated that coding behaviour changes both between and within individuals over time, emphasising the importance of conducting regular and systematic inter and intra-rater reliability tests, especially when multiple coders are involved, to ensure consistency and clarity at the screening and coding stages. Implications for good practice while screening/coding for systematic reviews are discussed.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cited by
Security and the smart city: A systematic review
Laufs, J., Borrion, H., & Bradford, B. (2020) · Sustainable Cities and Society · doi:10.1016/j.scs.2020.102023 · Cited by 296
The impact of hospital accreditation on the quality of healthcare: a systematic literature review
Hussein, M., Pavlova, M., Ghalwash, M., & Groot, W. (2021) · BMC Health Services Research · doi:10.1186/s12913-021-07097-6 · Cited by 239
Singh, J., Karanika-Murray, M., Baguley, T., & Hudson, J. (2020) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph17196987 · Cited by 132
A scoping review of school-based efforts to support students who have experienced trauma.
Stratford, B., Cook, E., Hanneke, R., Katz, E., Seok, D., Steed, H., Fulks, E., Lessans, A., & Temkin, D. (2020) · School Mental Health · doi:10.1007/s12310-020-09368-9 · Cited by 83
Metadata
- Authors
- Belur, J., Tompson, L., Thornton, A., & Simon, M.
- Venue
- CrimRxiv
- Publisher
- PubPub
- Published
- 2018-09-24
- OpenAlex
- W2949170298
- Cited by
- 58
- References
- 0
- Type
- posted content
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Interrater Reliability in Systematic Review Methodology: Exploring Variation in Coder Decision-Making. https://app.cochat.ai/papers/10.21428/cb6ab371.b8fc485a
- The paper
- Belur, J., Tompson, L., Thornton, A., & Simon, M. (2018). Interrater Reliability in Systematic Review Methodology: Exploring Variation in Coder Decision-Making. CrimRxiv. https://doi.org/10.21428/cb6ab371.b8fc485a
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.