Automating risk of bias assessment in systematic reviews: a real-time mixed methods comparison of human researchers to a machine learning system
Open accessThe DOI matches its Crossref record; the references it cites are not checked yet; 10 of the 20 sources its claims cite were checked for support.
Abstract
Background: Machine learning and automation are increasingly used to make the evidence synthesis process faster and more responsive to policymakers’ needs. In systematic reviews of randomized controlled trials (RCTs), risk of bias assessment is a resource-intensive task that typically requires two trained reviewers. One function of RobotReviewer, an off-the-shelf machine learning system, is an automated risk of bias assessment. Methods: We assessed the feasibility of adopting RobotReviewer within a national public health institute using a randomized, real-time, user-centered study. The study included 26 RCTs and six reviewers from two projects examining health and social interventions. We randomized these studies to one of two RobotReviewer platforms. We operationalized feasibility as accuracy, time use, and reviewer acceptability. We measured accuracy by the number of corrections made by human reviewers (either to automated assessments or another human reviewer’s assessments). We explored acceptability through group discussions and individual email responses after presenting the quantitative results. Results: Reviewers were equally likely to accept judgment by RobotReviewer as each other’s judgement during the consensus process when measured dichotomously; risk ratio 1.02 (95% CI 0.92 to 1.13; p = 0.33). We were not able to compare time use. The acceptability of the program by researchers was mixed. Less experienced reviewers were generally more positive, and they saw more benefits and were able to use the tool more flexibly. Reviewers positioned human input and human-to-human interaction as superior to even a semi-automation of this process. Conclusion: Despite being presented with evidence of RobotReviewer’s equal performance to humans, participating reviewers were not interested in modifying standard procedures to include automation. If further studies confirm equal accuracy and reduced time compared to manual practices, we suggest that the benefits of RobotReviewer may support its future implementation as one of two assessors, despite reviewer ambivalence. Future research should study barriers to adopting automated tools and how highly educated and experienced researchers can adapt to a job market that is increasingly challenged by new technologies.
Overview
CoChat Summary
TL;DR
In a real-time mixed-methods study, RobotReviewer performed as accurately as any single human reviewer in Cochrane RoB1 assessment, but reviewers remained unwilling to replace one of two human assessors. Time use could not be compared because neither project leader chose a fully manual comparison arm.
Key findings
- Humans accepted 83% of RobotReviewer’s assessments (133/160) and 81% of each other’s assessments (130/160); risk ratio 1.02 (95% CI 0.92 to 1.13; p = 0.33).
- There was a statistically significant difference in the distributions of correction frequency: humans corrected each other more heavily than they corrected RobotReviewer (χ²₁₂ = 31.260, p < 0.001); zero corrections occurred in 55% of studies for human-to-human assessments versus 48% for human-to-RobotReviewer assessments.
- Full agreement between RobotReviewer, one human, and the final consensus assessment occurred in 79% of questions (126/160); RobotReviewer under-estimated risk of bias in 4% of questions (8/160).
- Acceptability was mixed: less experienced reviewers were generally more positive and saw RobotReviewer as a pedagogical or text-finding aid, while reviewers overall did not want to modify the standard two-human procedure because they valued human-to-human discussion.
Methods
- Randomized, real-time, user-centered mixed-methods study within two ongoing systematic reviews; 26 RCTs and six reviewers from two projects examining health and social interventions.
- Project A contributed 23 RCTs randomized to RR-EPPI or RR-web; Project B contributed 3 RCTs that were not randomized. Accuracy was measured as human corrections to RobotReviewer versus corrections to another human, using χ² and Mann-Whitney tests.
- Acceptability was explored through group discussions and individual email responses after presenting quantitative results; qualitative data were thematically analyzed, with translation from Norwegian to English.
Limitations
- The paper states that feasibility was operationalized as accuracy, resource use, and acceptability, but structural adoption factors were not systematically measured; no manual arm meant relative time use could not be estimated; the sample was small (26 RCTs); no formal published protocol was used; and RobotReviewer’s dichotomized assessments may have made human assessments more likely to be corrected.
- The paper also notes possible social desirability bias because study administrators worked closely with participants and encouraged ML use.
- Evident from the text: data were missing for the second assessor for 12 studies, limiting paired human assessments for those RCTs.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures




Verification
Claims the paper makes
8 Valid · 0 Not valid · 7 Not checked · 73 Own results and interpretations
Every source the paper cites for this claim was checked and supports it.
In the paper
“However, intensive standardized training on risk of bias assessment may significantly improve the reliability of the RoB1 tool [12].”
§ Automating risk of bias assessment in systematic reviews: a real-time mixed methods comparison of human researchers to a machine learning system > Introduction > Assessing risk of bias
What the sources say
- Supportedcites [12] da Costa BR, Beckett B, Diaz A, Resta NM, Johnston BC, Egger M, et al. Effect of standardized training on the reliability of the Cochrane risk of bias assessment tool: a prospective study. Syst Rev. 2017;6(1):44. source
The Abstract Conclusions state almost exactly the cited claim. The full Conclusions qualify the finding: training significantly improved within-group agreement for all assessed items, while evidence of significant between-group improvement was limited to allocation concealment and incomplete outcome data. The citing sentence’s “may” preserves the paper’s overall caution. (LLM, conf=0.94)
hedgedstates a possible cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“It has been estimated that the average health or medical-related systematic review takes more than thirteen months to complete [1].”
§ Automating risk of bias assessment in systematic reviews: a real-time mixed methods comparison of human researchers to a machine learning system > Introduction > Evidence synthesis and machine learning
What the sources say
- Supportedcites [1] Borah R, Brown AW, Capers PL, Kaiser KA. Analysis of the time and workers needed to conduct systematic reviews of medical interventions using data from the PROSPERO registry. BMJ Open. 2017;7(2):e012545. source
The Results report a mean of 67.3 weeks—about 15.5 months—from registered project start to publication. The Abstract describes the sample as systematic reviews of medical interventions, so the estimate supports the claim that such reviews take more than thirteen months on average. (LLM, conf=0.93)
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“Supervised techniques, such as an ML system designed to classify studies as randomized controlled trials (RCTs) versus other study designs, have sufficient precision and recall to be used to help produce Cochrane systematic reviews [6].”
§ Automating risk of bias assessment in systematic reviews: a real-time mixed methods comparison of human researchers to a machine learning system > Introduction > Evidence synthesis and machine learning
What the sources say
- Supportedcites [6] Thomas J, McDonald S, Noel-Storr A, Shemilt I, Elliott J, Mavergames C, et al. Machine learning reduced workload with minimal risk of missing studies: development and evaluation of a randomized controlled trial classifier for Cochrane Reviews. J Clin Epidemiol. 2021;133:140–51. source
The Background describes a supervised classifier for distinguishing RCT reports from other records and says it was deployed within Cochrane. The Results report 99% recall and 8% precision in calibration; the Discussion acknowledges that precision appears low but reports substantial savings in manual screening. This supports the claim that the system could help produce Cochrane Reviews, rather than a claim that its precision was high. (LLM, conf=0.90)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Reviewers were equally likely to accept judgment by RobotReviewer as each other’s judgement during the consensus process when measured dichotomously; risk ratio 1.02 (95% CI 0.92 to 1.13; p = 0.33).”
- Also stated in: § Accuracy · plain
“RobotReviewer performed as accurately as any one reviewer: humans accepted 83% of RobotReviewer’s assessments (133 of 160), and 81% (130 of 160) of each other’s assessments; risk ratio 1.02 (95% CI 0.92 to 1.13; p = 0.33).”
§ Automating risk of bias assessment in systematic reviews: a real-time mixed methods comparison of human researchers to a machine learning system > Abstract > Results
plainn = 160- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § Accuracy · plain
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The acceptability of the program by researchers was mixed.”
§ Automating risk of bias assessment in systematic reviews: a real-time mixed methods comparison of human researchers to a machine learning system > Abstract > Results
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
35 more claims
Create a free account to see all 88 claims and to verify the 10 references still open.
- Probably not a substantial claim. 34
- It may state two separate claims as one. 30
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“To change the current perceptions on automated data extraction and RoB assessments they suggest that there need to be further quantifications of how much time can be saved using automation, and that more independent studies are needed to assess how RobotReviewer or other such (semi)automated systems perform in practice [36].”
From A narrative review of recent tools and innovations toward automating living systematic reviews and evidence syntheses (2023) · cites this paper as [36]
Experimental — accuracy not yet measuredIn the citing paper
“In 2022, Jardim et al. conducted a user-centred, randomised study comparing human bias judgements with RobotReviewer and assessed acceptability of the system via group discussions and feedback.”
From A narrative review of recent tools and innovations toward automating living systematic reviews and evidence syntheses (2023) · cites this paper as [36]
In the citing paper
“When replacing a human with RobotReviewer, they found that reviewers are as likely to be in consensus with the machine's prediction as they are with assessments from other humans.”
From A narrative review of recent tools and innovations toward automating living systematic reviews and evidence syntheses (2023) · cites this paper as [36]
In the citing paper
“They did however also find that reviewers still preferred human-human interactions and that there is no significant interest in integrating RobotReviewer's automation into standard SR processes.”
From A narrative review of recent tools and innovations toward automating living systematic reviews and evidence syntheses (2023) · cites this paper as [36]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Automating risk of bias assessment in systematic reviews: a real-time mixed methods comparison of human researchers to a machine learning system > Results
“For each study, four RobotReviewer assessments were available for analysis, four assessments from the first reviewer, four assessments from the second reviewer, and the final consensus set of four assessments.”
§ Automating risk of bias assessment in systematic reviews: a real-time mixed methods comparison of human researchers to a machine learning system > Results
“Data was missing for the second assessor for twelve studies.”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Automating risk of bias assessment in systematic reviews: a real-time mixed methods comparison of human researchers to a machine learning system > Results
“There were 160 human corrections made to RobotReviewer’s assessments before consensus, and identically 160 human corrections made to each other, that is, differences between the final consensus assessment and each individual human assessment.”
§ Automating risk of bias assessment in systematic reviews: a real-time mixed methods comparison of human researchers to a machine learning system > Results > Accuracy
“humans accepted 83% of RobotReviewer’s assessments (133 of 160), and 81% (130 of 160) of each other’s assessments”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Automating risk of bias assessment in systematic reviews: a real-time mixed methods comparison of human researchers to a machine learning system > Abstract > Methods
“We randomized these studies to one of two RobotReviewer platforms.”
§ Automating risk of bias assessment in systematic reviews: a real-time mixed methods comparison of human researchers to a machine learning system > Methods > Procedures and study design
“Project B [29] contributed 3 RCTs, and we did not randomize these studies.”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
“human corrections to each other were evenly distributed among the first three questions (9, 10, and 8, retrospectively, of 31), and humans corrected each other’s assessment to this fourth question least often (5).”
§ Automating risk of bias assessment in systematic reviews: a real-time mixed methods comparison of human researchers to a machine learning system > Results > Accuracy
Reported 31 · recomputed 32
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
“In 4% (8 of 160) of questions, RobotReviewer under-estimated risk of bias”
§ Automating risk of bias assessment in systematic reviews: a real-time mixed methods comparison of human researchers to a machine learning system > Results > Accuracy
Reported 4 · recomputed 5
1 more finding
Create a free account to see every finding of this check.
Citations
The citation lists are queued
They are fetched later and appear here then.
References 30 · Cited by 68
Research works printed but not linked: 4
- Schunemann H. GRADE handbook for grading quality of evidence and strength of recommendation. Version 3.2. http://www.cc-imsnet/gradepro. 2008.
- Thomas J, Graziosi S, Brunton J, Ghouze Z, O’Driscoll P, Bond M. EPPI-Reviewer: advanced software for systematic reviews, maps and evidence synthesis. London: UCL Social Research Institute; 2020.
- Tingulstad A, Meneses-Echavez J, Evensen LH, Bjerk M, Holte HH. Arbeidsrettede rehabiliteringstiltak ved langtidssykmelding: en systematisk oversikt [Work-related interventions for people on long-term sick leave: a systematic review] Rapport −2021. Oslo: Folkehelseinstituttet; 2021.
- Hestevik CH, Müller AE, Forsetlund SL. Behandlingstiltak for personer som utøver seksuell vold i nære relasjoner: en systematisk oversikt [Treatment for perpetrators of sexual violence in close relationships: a systematic review]. Rapport−2021. Folkehelseinstituttet: Oslo; 2021.
Metadata
- Authors
- Jardim, P. S. J., Rose, C. J., Ames, H. M., Echavez, J. F. M., Van de Velde, S., & Muller, A. E.
- Affiliations
- Norwegian Institute of Public Health (NO)
- Corresponding author
- Jardim, Patricia Sofia Jacobsen
- Venue
- BMC Medical Research Methodology
- Published
- 2022-06-08
- Volume
- 22
- Issue
- 1
- Pages
- 167-167
- ISSN
- 1471-2288 (ISSN-L)
- PMC
- PMC9174024
- OpenAlex
- W4281827825
- Semantic Scholar
- 515804f633dfecf2d4dd34f5d44aa64f99236d62
- Cited by
- 68
- References
- 30
- Citations by year
- 2026: 14 · 2025: 13 · 2024: 23 · 2023: 15 · 2022: 3
- FWCI
- 4.43
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- creativecommons.org/licenses/by/4.0 (tdm)
- creativecommons.org/licenses/by/4.0 (vor)
- Locations
- BMC Medical Research Methodology (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- NORA - Norwegian Open Research Archives (submittedVersion, other-oa)
- Funders
- Norwegian Institute of Public Health, Norway
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Sociology and Political Science › Delphi Technique in Research
- Physical Sciences › Mathematics › Statistics and Probability › Advanced Causal Inference Techniques
- Keywords
- risk of bias assessment, machine learning, systematic reviews, randomized controlled trials, human-to-human interaction, evidence synthesis
- MeSH terms
- Machine Learning, Systematic Reviews as Topic, Humans, Bias, Randomized Controlled Trials as Topic, Risk Assessment
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-05, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-05 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Automating risk of bias assessment in systematic reviews: a real-time mixed methods comparison of human researchers to a machine learning system. https://app.cochat.ai/papers/10.1186/s12874-022-01649-y
- The paper
- Jardim, P. S. J., Rose, C. J., Ames, H. M., Echavez, J. F. M., Van de Velde, S., & Muller, A. E. (2022). Automating risk of bias assessment in systematic reviews: a real-time mixed methods comparison of human researchers to a machine learning system. BMC Medical Research Methodology. https://doi.org/10.1186/s12874-022-01649-y
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.