CoChat

Automating risk of bias assessment in systematic reviews: a real-time mixed methods comparison of human researchers to a machine learning system

Open access
Jardim, Patricia Sofia Jacobsen, Rose, Christopher James, Ames, Heather Melanie, Echavez, Jose Francisco Meneses, Van de Velde, Stijn, Muller, Ashley Elizabeth · 2022 · BMC Medical Research Methodology

The DOI matches its Crossref record; the references it cites are not checked yet; 10 of the 20 sources its claims cite were checked for support.

Abstract

Background: Machine learning and automation are increasingly used to make the evidence synthesis process faster and more responsive to policymakers’ needs. In systematic reviews of randomized controlled trials (RCTs), risk of bias assessment is a resource-intensive task that typically requires two trained reviewers. One function of RobotReviewer, an off-the-shelf machine learning system, is an automated risk of bias assessment. Methods: We assessed the feasibility of adopting RobotReviewer within a national public health institute using a randomized, real-time, user-centered study. The study included 26 RCTs and six reviewers from two projects examining health and social interventions. We randomized these studies to one of two RobotReviewer platforms. We operationalized feasibility as accuracy, time use, and reviewer acceptability. We measured accuracy by the number of corrections made by human reviewers (either to automated assessments or another human reviewer’s assessments). We explored acceptability through group discussions and individual email responses after presenting the quantitative results. Results: Reviewers were equally likely to accept judgment by RobotReviewer as each other’s judgement during the consensus process when measured dichotomously; risk ratio 1.02 (95% CI 0.92 to 1.13; p = 0.33). We were not able to compare time use. The acceptability of the program by researchers was mixed. Less experienced reviewers were generally more positive, and they saw more benefits and were able to use the tool more flexibly. Reviewers positioned human input and human-to-human interaction as superior to even a semi-automation of this process. Conclusion: Despite being presented with evidence of RobotReviewer’s equal performance to humans, participating reviewers were not interested in modifying standard procedures to include automation. If further studies confirm equal accuracy and reduced time compared to manual practices, we suggest that the benefits of RobotReviewer may support its future implementation as one of two assessors, despite reviewer ambivalence. Future research should study barriers to adopting automated tools and how highly educated and experienced researchers can adapt to a job market that is increasingly challenged by new technologies.

Overview

CoChat Summary

TL;DR

In a real-time mixed-methods study, RobotReviewer performed as accurately as any single human reviewer in Cochrane RoB1 assessment, but reviewers remained unwilling to replace one of two human assessors. Time use could not be compared because neither project leader chose a fully manual comparison arm.

Key findings

  • Humans accepted 83% of RobotReviewer’s assessments (133/160) and 81% of each other’s assessments (130/160); risk ratio 1.02 (95% CI 0.92 to 1.13; p = 0.33).
  • There was a statistically significant difference in the distributions of correction frequency: humans corrected each other more heavily than they corrected RobotReviewer (χ²₁₂ = 31.260, p < 0.001); zero corrections occurred in 55% of studies for human-to-human assessments versus 48% for human-to-RobotReviewer assessments.
  • Full agreement between RobotReviewer, one human, and the final consensus assessment occurred in 79% of questions (126/160); RobotReviewer under-estimated risk of bias in 4% of questions (8/160).
  • Acceptability was mixed: less experienced reviewers were generally more positive and saw RobotReviewer as a pedagogical or text-finding aid, while reviewers overall did not want to modify the standard two-human procedure because they valued human-to-human discussion.

Methods

  • Randomized, real-time, user-centered mixed-methods study within two ongoing systematic reviews; 26 RCTs and six reviewers from two projects examining health and social interventions.
  • Project A contributed 23 RCTs randomized to RR-EPPI or RR-web; Project B contributed 3 RCTs that were not randomized. Accuracy was measured as human corrections to RobotReviewer versus corrections to another human, using χ² and Mann-Whitney tests.
  • Acceptability was explored through group discussions and individual email responses after presenting quantitative results; qualitative data were thematically analyzed, with translation from Norwegian to English.

Limitations

  • The paper states that feasibility was operationalized as accuracy, resource use, and acceptability, but structural adoption factors were not systematically measured; no manual arm meant relative time use could not be estimated; the sample was small (26 RCTs); no formal published protocol was used; and RobotReviewer’s dichotomized assessments may have made human assessments more likely to be corrected.
  • The paper also notes possible social desirability bias because study administrators worked closely with participants and encouraged ML use.
  • Evident from the text: data were missing for the second assessor for 12 studies, limiting paired human assessments for those RCTs.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1186/s12874-022-01649-y

Cites

Statistics

Recomputing p-values needs the paper’s full text.

Figures

Fig. 1 Flowchart of included studies and their domaine assesement available for analysis
Fig. 1 Flowchart of included studies and their domaine assesement available for analysisCC BY · PubMed Central
Fig. 2 Human corrections to RobotReviewer’s assessments and to other humans’ assessments
Fig. 2 Human corrections to RobotReviewer’s assessments and to other humans’ assessmentsCC BY · PubMed Central
Fig. 3 Corrections between individual human assessments and RobotReviewer (left) and between individual human assessments and final consensus (right), in the assessments of all 26 RCTs
Fig. 3 Corrections between individual human assessments and RobotReviewer (left) and between individual human assessments and final consensus (right), in the assessments of all 26 RCTsCC BY · PubMed Central
Fig. 4 Corrections from RobotReviewer to any one human assessment to the final consensus assessment
Fig. 4 Corrections from RobotReviewer to any one human assessment to the final consensus assessmentCC BY · PubMed Central

Verification

Claims the paper makes

8 Valid · 0 Not valid · 7 Not checked · 73 Own results and interpretations

35 more claims

Create a free account to see all 88 claims and to verify the 10 references still open.

See all claims — free account
  • Probably not a substantial claim. 34
  • It may state two separate claims as one. 30

Create a free account to see these claims.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

5 of 6 · the rest aren’t loaded

1 more finding

Create a free account to see every finding of this check.

See all findings — free account

Citations

The citation lists are queued

They are fetched later and appear here then.

References 30 · Cited by 68

Research works printed but not linked: 4

  1. Schunemann H. GRADE handbook for grading quality of evidence and strength of recommendation. Version 3.2. http://www.cc-imsnet/gradepro. 2008.
  2. Thomas J, Graziosi S, Brunton J, Ghouze Z, O’Driscoll P, Bond M. EPPI-Reviewer: advanced software for systematic reviews, maps and evidence synthesis. London: UCL Social Research Institute; 2020.
  3. Tingulstad A, Meneses-Echavez J, Evensen LH, Bjerk M, Holte HH. Arbeidsrettede rehabiliteringstiltak ved langtidssykmelding: en systematisk oversikt [Work-related interventions for people on long-term sick leave: a systematic review] Rapport −2021. Oslo: Folkehelseinstituttet; 2021.
  4. Hestevik CH, Müller AE, Forsetlund SL. Behandlingstiltak for personer som utøver seksuell vold i nære relasjoner: en systematisk oversikt [Treatment for perpetrators of sexual violence in close relationships: a systematic review]. Rapport−2021. Folkehelseinstituttet: Oslo; 2021.

Metadata

Authors
Jardim, P. S. J., Rose, C. J., Ames, H. M., Echavez, J. F. M., Van de Velde, S., & Muller, A. E.
Affiliations
Norwegian Institute of Public Health (NO)
Corresponding author
Jardim, Patricia Sofia Jacobsen
Venue
BMC Medical Research Methodology
Published
2022-06-08
Volume
22
Issue
1
Pages
167-167
ISSN
1471-2288 (ISSN-L)
OpenAlex
W4281827825
Cited by
68
References
30
Citations by year
2026: 14 · 2025: 13 · 2024: 23 · 2023: 15 · 2022: 3
FWCI
4.43
Type
journal article
Language
en
Open access
Licence
Locations
  • BMC Medical Research Methodology (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
  • NORA - Norwegian Open Research Archives (submittedVersion, other-oa)
Funders
Norwegian Institute of Public Health, Norway
Topics
  • Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
  • Social Sciences › Sociology and Political Science › Delphi Technique in Research
  • Physical Sciences › Mathematics › Statistics and Probability › Advanced Causal Inference Techniques
Keywords
risk of bias assessment, machine learning, systematic reviews, randomized controlled trials, human-to-human interaction, evidence synthesis
MeSH terms
Machine Learning, Systematic Reviews as Topic, Humans, Bias, Randomized Controlled Trials as Topic, Risk Assessment

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-10-05, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-05 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (gpt-6-sol).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Automating risk of bias assessment in systematic reviews: a real-time mixed methods comparison of human researchers to a machine learning system. https://app.cochat.ai/papers/10.1186/s12874-022-01649-y
The paper
Jardim, P. S. J., Rose, C. J., Ames, H. M., Echavez, J. F. M., Van de Velde, S., & Muller, A. E. (2022). Automating risk of bias assessment in systematic reviews: a real-time mixed methods comparison of human researchers to a machine learning system. BMC Medical Research Methodology. https://doi.org/10.1186/s12874-022-01649-y

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.