CoChat

Machine learning to help researchers evaluate biases in clinical trials: a prospective, randomized user study

Open access

The DOI matches its Crossref record; 26 of the 28 references it cites were found in a registry or an index; 3 of the 8 sources its claims cite were checked for support.

Abstract

OBJECTIVE: Assessing risks of bias in randomized controlled trials (RCTs) is an important but laborious task when conducting systematic reviews. RobotReviewer (RR), an open-source machine learning (ML) system, semi-automates bias assessments. We conducted a user study of RobotReviewer, evaluating time saved and usability of the tool. MATERIALS AND METHODS: Systematic reviewers applied the Cochrane Risk of Bias tool to four randomly selected RCT articles. Reviewers judged: whether an RCT was at low, or high/unclear risk of bias for each bias domain in the Cochrane tool (Version 1); and highlighted article text justifying their decision. For a random two of the four articles, the process was semi-automated: users were provided with ML-suggested bias judgments and text highlights. Participants could amend the suggestions if necessary. We measured time taken for the task, ML suggestions, usability via the System Usability Scale (SUS) and collected qualitative feedback. RESULTS: For 41 volunteers, semi-automation was quicker than manual assessment (mean 755 vs. 824 s; relative time 0.75, 95% CI 0.62-0.92). Reviewers accepted 301/328 (91%) of the ML Risk of Bias (RoB) judgments, and 202/328 (62%) of text highlights without change. Overall, ML suggested text highlights had a recall of 0.90 (SD 0.14) and precision of 0.87 (SD 0.21) with respect to the users' final versions. Reviewers assigned the system a mean 77.7 SUS score, corresponding to a rating between "good" and "excellent". CONCLUSIONS: Semi-automation (where humans validate machine learning suggestions) can improve the efficiency of evidence synthesis. Our system was rated highly usable, and expedited bias assessment of RCTs.

Overview

CoChat Summary

TL;DR

In a prospective randomized user study, semi-automated RobotReviewer bias assessment was 25% faster than manual assessment for systematic reviewers. Reviewers accepted most ML bias judgments and text highlights, and rated the system highly usable.

Key findings

  • Semi-automation reduced mean time per article to 755 s (SD 868) vs 824 s (SD 717) manual; mixed model relative time 0.75 (95% CI 0.62–0.92).
  • Reviewers made no change to 301/328 (91%) ML bias judgments and to 202/328 (62%) ML text highlights.
  • ML text highlights had recall 0.90 (SD 0.14) and precision 0.87 (SD 0.21) against users’ final edits.
  • Usability was high: results report mean SUS 77.4 (SD 14.2), described as between “good” and “excellent”; the abstract states 77.7.

Methods

  • Prospective, randomized within-subject user study: 41 volunteers with systematic review experience each assessed four RCT articles, two with ML suggestions and two manual, with randomized article selection/order and task order.
  • Participants assigned low vs high/unclear risk of bias for four Cochrane domains (random sequence generation, allocation concealment, blinding of participants/personnel, blinding of outcome assessors) and marked supporting text.
  • Primary outcome was time taken; analyses used Wilcoxon signed rank tests and a mixed linear model with participant as random effect, plus precision/recall of ML rationales and SUS usability scores.

Limitations

  • The paper states participants were recruited via email/Twitter, completed anonymously, and were not extensively verified; those open to new technologies may be overrepresented, so usability/acceptability findings may be optimistic.
  • The paper states RobotReviewer did not provide recommendations for selective reporting or incomplete outcome data, and the Cochrane RoB tool was evolving at the time of writing.
  • Evident from the text: only four bias domains were assessed, judgments were dichotomized into low vs high/unclear, and article difficulty/order affected assessment times; sensitivity analyses are in the online Appendix rather than fully reported in the main text.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1186/s12911-019-0814-z

Cites

26 of 26 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Figures

Fig. 1 A schematic of our approach to automated risk of bias prediction and supportive rationale extraction. This is an ensemble composed of two models that we have described at length elsewhere: a linear kernel multi-task SVM variant (MT-SVM; bottom) [10], and a rationale-augmented CNN (RA-CNN; top-k) [11]
Fig. 1 A schematic of our approach to automated risk of bias prediction and supportive rationale extraction. This is an ensemble composed of two models that we have described at length elsewhere: a linear kernel multi-task SVM variant (MT-SVM; bottom) [10], and a rationale-augmented CNN (RA-CNN; top-k) [11]CC BY · PubMed Central
Fig. 2 The RobotReviewer web interface. Screenshot of the RobotReviewer web interface, as used by study participants, with this example showing machine learning suggestions. Participants viewed the article (left), and completed the risk of bias assessment (right)
Fig. 2 The RobotReviewer web interface. Screenshot of the RobotReviewer web interface, as used by study participants, with this example showing machine learning suggestions. Participants viewed the article (left), and completed the risk of bias assessment (right)CC BY · PubMed Central
Fig. 3 Randomization strategy. For each participant, we randomized: the selection and order of test articles, and the sequence which the two semi-automation and two manual tasks were presented
Fig. 3 Randomization strategy. For each participant, we randomized: the selection and order of test articles, and the sequence which the two semi-automation and two manual tasks were presentedCC BY · PubMed Central
Fig. 4 The user study pipeline overview
Fig. 4 The user study pipeline overviewCC BY · PubMed Central
Fig. 5 Violin plots of total time (in seconds) taken using the machine learning aided versus standard (fully manual) approach
Fig. 5 Violin plots of total time (in seconds) taken using the machine learning aided versus standard (fully manual) approachCC BY · PubMed Central
Fig. 6 Flow diagram of user interaction with the machine learning suggested rationales (text highlights)
Fig. 6 Flow diagram of user interaction with the machine learning suggested rationales (text highlights)CC BY · PubMed Central
Fig. 7 Question-by-question breakdown of the SUS questionnaire responses; *supplementary task-specific questions
Fig. 7 Question-by-question breakdown of the SUS questionnaire responses; *supplementary task-specific questionsCC BY · PubMed Central

Verification

Claims the paper makes

2 Valid · 0 Not valid · 4 Not checked · 28 Own results and interpretations

20 more claims

Create a free account to see all 34 claims and every source verdict.

See all claims — free account
  • It may state two separate claims as one. 3
  • Probably not a substantial claim. 7

Create a free account to see these claims.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

4 of 4

Citations

Cites

Show all 26 references

Research works printed but not linked: 1

  1. Sauro J. Measuring usability with the system usability scale (SUS). 2011. http://www.measuringusability.com/sus.php

Cited by

See all 53

Metadata

Authors
Soboczenski, F., Trikalinos, T. A., Kuiper, J., Bias, R. G., Wallace, B. C., & Marshall, I. J.
Affiliations
  • King's College London (GB)
  • Brown University (US)
  • The University of Texas at Austin (US)
  • Northeastern University (US)
Corresponding author
Soboczenski, Frank
Venue
BMC Medical Informatics and Decision Making
Published
2019-05-08
Volume
19
Issue
1
Pages
96-96
ISSN
1472-6947 (ISSN-L)
OpenAlex
W2902105843
Cited by
53
References
28
Citations by year
2026: 5 · 2025: 9 · 2024: 7 · 2023: 10 · 2022: 11 · 2021: 3 · 2020: 7 · 2019: 1
FWCI
3.67
Type
journal article
Language
en
Open access
Licence
Locations
  • BMC Medical Informatics and Decision Making (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
Funders
  • U.S. National Library of Medicine (R01LM012086)
  • Medical Research Council (MR/N015185/1)
Topics
  • Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
  • Social Sciences › General Social Sciences › Computational and Text Analysis Methods
  • Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
Keywords
risk of bias assessment, randomized controlled trials, Cochrane risk of bias tool, machine learning, semi-automation, usability, systematic reviews, evidence synthesis, text highlighting, user study
MeSH terms
Machine Learning, Feedback, Humans, Prospective Studies, Bias, Randomized Controlled Trials as Topic, Risk Assessment

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (gpt-6-sol).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Machine learning to help researchers evaluate biases in clinical trials: a prospective, randomized user study. https://app.cochat.ai/papers/10.1186/s12911-019-0814-z
The paper
Soboczenski, F., Trikalinos, T. A., Kuiper, J., Bias, R. G., Wallace, B. C., & Marshall, I. J. (2019). Machine learning to help researchers evaluate biases in clinical trials: a prospective, randomized user study. BMC Medical Informatics and Decision Making. https://doi.org/10.1186/s12911-019-0814-z

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.