Machine learning to help researchers evaluate biases in clinical trials: a prospective, randomized user study
Open accessThe DOI matches its Crossref record; 26 of the 28 references it cites were found in a registry or an index; 3 of the 8 sources its claims cite were checked for support.
Abstract
OBJECTIVE: Assessing risks of bias in randomized controlled trials (RCTs) is an important but laborious task when conducting systematic reviews. RobotReviewer (RR), an open-source machine learning (ML) system, semi-automates bias assessments. We conducted a user study of RobotReviewer, evaluating time saved and usability of the tool. MATERIALS AND METHODS: Systematic reviewers applied the Cochrane Risk of Bias tool to four randomly selected RCT articles. Reviewers judged: whether an RCT was at low, or high/unclear risk of bias for each bias domain in the Cochrane tool (Version 1); and highlighted article text justifying their decision. For a random two of the four articles, the process was semi-automated: users were provided with ML-suggested bias judgments and text highlights. Participants could amend the suggestions if necessary. We measured time taken for the task, ML suggestions, usability via the System Usability Scale (SUS) and collected qualitative feedback. RESULTS: For 41 volunteers, semi-automation was quicker than manual assessment (mean 755 vs. 824 s; relative time 0.75, 95% CI 0.62-0.92). Reviewers accepted 301/328 (91%) of the ML Risk of Bias (RoB) judgments, and 202/328 (62%) of text highlights without change. Overall, ML suggested text highlights had a recall of 0.90 (SD 0.14) and precision of 0.87 (SD 0.21) with respect to the users' final versions. Reviewers assigned the system a mean 77.7 SUS score, corresponding to a rating between "good" and "excellent". CONCLUSIONS: Semi-automation (where humans validate machine learning suggestions) can improve the efficiency of evidence synthesis. Our system was rated highly usable, and expedited bias assessment of RCTs.
Overview
CoChat Summary
TL;DR
In a prospective randomized user study, semi-automated RobotReviewer bias assessment was 25% faster than manual assessment for systematic reviewers. Reviewers accepted most ML bias judgments and text highlights, and rated the system highly usable.
Key findings
- Semi-automation reduced mean time per article to 755 s (SD 868) vs 824 s (SD 717) manual; mixed model relative time 0.75 (95% CI 0.62–0.92).
- Reviewers made no change to 301/328 (91%) ML bias judgments and to 202/328 (62%) ML text highlights.
- ML text highlights had recall 0.90 (SD 0.14) and precision 0.87 (SD 0.21) against users’ final edits.
- Usability was high: results report mean SUS 77.4 (SD 14.2), described as between “good” and “excellent”; the abstract states 77.7.
Methods
- Prospective, randomized within-subject user study: 41 volunteers with systematic review experience each assessed four RCT articles, two with ML suggestions and two manual, with randomized article selection/order and task order.
- Participants assigned low vs high/unclear risk of bias for four Cochrane domains (random sequence generation, allocation concealment, blinding of participants/personnel, blinding of outcome assessors) and marked supporting text.
- Primary outcome was time taken; analyses used Wilcoxon signed rank tests and a mixed linear model with participant as random effect, plus precision/recall of ML rationales and SUS usability scores.
Limitations
- The paper states participants were recruited via email/Twitter, completed anonymously, and were not extensively verified; those open to new technologies may be overrepresented, so usability/acceptability findings may be optimistic.
- The paper states RobotReviewer did not provide recommendations for selective reporting or incomplete outcome data, and the Cochrane RoB tool was evolving at the time of writing.
- Evident from the text: only four bias domains were assessed, judgments were dichotomized into low vs high/unclear, and article difficulty/order affected assessment times; sensitivity analyses are in the online Appendix rather than fully reported in the main text.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures
![Fig. 1 A schematic of our approach to automated risk of bias prediction and supportive rationale extraction. This is an ensemble composed of two models that we have described at length elsewhere: a linear kernel multi-task SVM variant (MT-SVM; bottom) [10], and a rationale-augmented CNN (RA-CNN; top-k) [11]](https://figures.cochat.ai/paper-figures/5a2557b2f2f64b0c98f311013739b75a/machine-learning-to-help-researchers-evaluate-biases-in-clinical-trials-a-figure-1.jpg)






Verification
Claims the paper makes
2 Valid · 0 Not valid · 4 Not checked · 28 Own results and interpretations
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“However, almost all of this work has considered retrospective evaluation of ML/NLP methods applied to existing datasets, providing only indirect evidence of the utility of such methods for EBM.”
§ Machine learning to help researchers evaluate biases in clinical trials: a prospective, randomized user study > Background and significance
What the sources say
- Partly supportedcites [6] Jonnalagadda SR, Goyal P, Huffman MD. Automating data extraction in systematic reviews: a systematic review. Syst Rev. 2015;4:78. sourceExperimental — accuracy not yet measured
The Results (Results of individual studies, no page given) summarize evaluations of automated data-extraction methods by dataset and accuracy, and the Discussion (Research implications, no page given) says the 26 studies used differing datasets and gold standards for evaluation. This supports the claim's characterization of the data-extraction studies, but the review does not establish that almost all ML/NLP work across evidence synthesis was retrospective or explicitly assess whether it provided only indirect evidence of utility for EBM. (LLM, conf=0.82)
- Partly supportedcites [7] O’Connor AM, Tsafnat G, Gilbert SB, et al. Moving toward the automation of the systematic review process: a summary of discussions at the second meeting of International Collaboration for the Automation of Systematic Reviews (ICASR). Syst Rev. 2018;7:3. sourceExperimental — accuracy not yet measured
The Background sections on Validation of tools and Common needs and challenges describe limited-purpose validation datasets and a shortage of studies showing the benefits of screening systems in different disciplines. That supports the concern that practical utility is not well established, but the paper does not say that almost all prior ML/NLP work used retrospective evaluations of existing datasets. The indexed copy provides no page numbers for these passages. (LLM, conf=0.84)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Regarding our supplementary task-specific questions, 69% of participants “strongly agreed” and “agreed” that they would use our system for systematic reviews.”
§ Machine learning to help researchers evaluate biases in clinical trials: a prospective, randomized user study > Results > System usability scale (SUS) scores
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“56% “strongly agreed” or “agreed” that the ML suggested text was helpful.”
§ Machine learning to help researchers evaluate biases in clinical trials: a prospective, randomized user study > Results > System usability scale (SUS) scores
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For 41 volunteers, semi-automation was quicker than manual assessment (mean 755 vs. 824 s; relative time 0.75, 95% CI 0.62–0.92).”
- Also stated in: § Results · plain
“The mean time taken to assess an article was 755 s (SD 868) for semi-automated assessment, and 824 s (SD 717) for manual assessment (see Fig. 5 and Table 3 P < 0.001). In our mixed model adjusting for participant identifier as a random effect, semi-automation was 25% faster than fully manual bias assessment: relative time for semi-automation vs manual 0.75, 95% CI 0.62 to 0.92.”
§ Machine learning to help researchers evaluate biases in clinical trials: a prospective, randomized user study > Abstract > Results
plainstates a causen = 41- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § Results · plain
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“72% “strongly disagreed” or “disagreed” that the suggested text was confusing.”
§ Machine learning to help researchers evaluate biases in clinical trials: a prospective, randomized user study > Results > System usability scale (SUS) scores
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
20 more claims
Create a free account to see all 34 claims and every source verdict.
- It may state two separate claims as one. 3
- Probably not a substantial claim. 7
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“In a recent simulated validation study, volunteer reviewers agreed with 91% of RobotReviewer’s assessments, and RoB1 assessment by one reviewer with RobotReviewer was 25% faster than fully manual assessments [16].”
From Automating risk of bias assessment in systematic reviews: a real-time mixed methods comparison of human researchers to a machine learning system (2022) · cites this paper as [16]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Machine learning to help researchers evaluate biases in clinical trials: a prospective, randomized user study > Results
“824 s (SD 717) for manual assessment”
§ Machine learning to help researchers evaluate biases in clinical trials: a prospective, randomized user study > Results
“| (seconds) | (SD = 868) | (SD = 727) | | 0.92) |”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Machine learning to help researchers evaluate biases in clinical trials: a prospective, randomized user study > Abstract > Results
“Reviewers assigned the system a mean 77.7 SUS score”
§ Machine learning to help researchers evaluate biases in clinical trials: a prospective, randomized user study > Results > System usability scale (SUS) scores
“For the standard SUS, participants gave a mean score of 77.4 (SD 14.2).”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
“made partial changes to 113 (35%) of the annotations”
§ Machine learning to help researchers evaluate biases in clinical trials: a prospective, randomized user study > Results > User edits to the machine learning suggestions
Reported 35 · recomputed 34.45122
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
“| Judgements changed | 27 (9%) | 7 (9%) | 7 (9%) | 6 (7%) | 7 (9%) |”
§ Machine learning to help researchers evaluate biases in clinical trials: a prospective, randomized user study > Results > User edits to the machine learning suggestions
Reported 9 · recomputed 8.231707
Citations
Cites
Fitting Linear Mixed-Effects Models Using lme4
Bates, D., Mächler, M., Bolker, B., & Walker, S. (2015) · Journal of Statistical Software · doi:10.18637/jss.v067.i01 · Cited by 86,835
The Cochrane Collaboration's tool for assessing risk of bias in randomised trials
Higgins, J. P. T., Altman, D. G., Gotzsche, P. C., Juni, P., Moher, D., Oxman, A. D., Savovic, J., Schulz, K. F., Weeks, L., Sterne, J. A. C., et al. (2011) · BMJ · doi:10.1136/bmj.d5928 · Cited by 33,817
Dermyshi, E., Wang, Y., Yan, C., Hong, W., Qiu, G., Gong, X., & Zhang, T. (2017) · Neonatology · doi:10.1159/000454668 · Cited by 24,451
Evidence based medicine: what it is and what it isn't
Sackett, D. L., Rosenberg, W. M. C., Gray, J. A. M., Haynes, R. B., & Richardson, W. S. (1996) · BMJ · doi:10.1136/bmj.312.7023.71 · Cited by 15,769 · cited 1×
“Evidence-Based Medicine (EBM) aspires to inform patient care using the entirety of the available evidence [1].”
BackgroundAn Empirical Evaluation of the System Usability Scale
Bangor, A., Kortum, P. T., & Miller, J. T. (2008) · International Journal of Human–Computer Interaction · doi:10.1080/10447310802205776 · Cited by 5,181
Yeh, M.-L. (2014) · Journal of Nursing Research · doi:10.1097/jnr.0000000000000065 · Cited by 3,829
Distant supervision for relation extraction without labeled data
Mintz, M., Bills, S., Snow, R., & Jurafsky, D. (2009) · Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP: Volume 2 - ACL-IJCNLP '09 · doi:10.3115/1690219.1690287 · Cited by 2,752
(2013) · Journal of Usability Studies archive · Cited by 1,357
Seventy-five trials and eleven systematic reviews a day: how will we ever keep up?
Bastian, H., Glasziou, P., & Chalmers, I. (2010) · PLoS Medicine · doi:10.1371/journal.pmed.1000326 · Cited by 1,188
(2016) · Cochrane Database of Systematic Reviews · Cited by 985
Research works printed but not linked: 1
- Sauro J. Measuring usability with the system usability scale (SUS). 2011. http://www.measuringusability.com/sus.php
Cited by
Gwinnutt, J. M., Wieczorek, M., Balanescu, A., Bischoff-Ferrari, H. A., Boonen, A., Cavalli, G., de Souza, S., de Thurah, A., Dorner, T. E., Moe, R. H., et al. (2023) · Annals of the Rheumatic Diseases · doi:10.1136/annrheumdis-2021-222020 · Cited by 263
Abdelwanis, M., Alarafati, H. K., Tammam, M. M. S., & Simsekler, M. C. E. (2024) · Journal of Safety Science and Resilience · doi:10.1016/j.jnlssr.2024.06.001 · Cited by 177
Nußbaumer-Streit, B., Sommer, I., Hamel, C., Devane, D., Noel-Storr, A. H., Puljak, L., Trivella, M., & Gartlehner, G. (2023) · BMJ Evidence-Based Medicine · doi:10.1136/bmjebm-2022-112185 · Cited by 130
Gwinnutt, J. M., Wieczorek, M., Cavalli, G., Balanescu, A. R., Bischoff-Ferrari, H. A., Boonen, A. E. R. C. H., de Souza, S., de Thurah, A. L., Dorner, T. E., Moe, R. H., et al. (2022) · RMD Open · doi:10.1136/rmdopen-2021-002168 · Cited by 118
Bazoukis, G., Stavrakis, S., Zhou, J., Bollepalli, S. C., Tse, G., Zhang, Q., Singh, J. P., & Armoundas, A. A. (2020) · Heart Failure Reviews · doi:10.1007/s10741-020-10007-3 · Cited by 106
Metadata
- Authors
- Soboczenski, F., Trikalinos, T. A., Kuiper, J., Bias, R. G., Wallace, B. C., & Marshall, I. J.
- Affiliations
- King's College London (GB)
- Brown University (US)
- The University of Texas at Austin (US)
- Northeastern University (US)
- Corresponding author
- Soboczenski, Frank
- Venue
- BMC Medical Informatics and Decision Making
- Published
- 2019-05-08
- Volume
- 19
- Issue
- 1
- Pages
- 96-96
- ISSN
- 1472-6947 (ISSN-L)
- PMC
- PMC6505190
- OpenAlex
- W2902105843
- Cited by
- 53
- References
- 28
- Citations by year
- 2026: 5 · 2025: 9 · 2024: 7 · 2023: 10 · 2022: 11 · 2021: 3 · 2020: 7 · 2019: 1
- FWCI
- 3.67
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- creativecommons.org/licenses/by/4.0/ (tdm)
- creativecommons.org/licenses/by/4.0/ (vor)
- Locations
- BMC Medical Informatics and Decision Making (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Funders
- U.S. National Library of Medicine (R01LM012086)
- Medical Research Council (MR/N015185/1)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › General Social Sciences › Computational and Text Analysis Methods
- Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
- Keywords
- risk of bias assessment, randomized controlled trials, Cochrane risk of bias tool, machine learning, semi-automation, usability, systematic reviews, evidence synthesis, text highlighting, user study
- MeSH terms
- Machine Learning, Feedback, Humans, Prospective Studies, Bias, Randomized Controlled Trials as Topic, Risk Assessment
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Machine learning to help researchers evaluate biases in clinical trials: a prospective, randomized user study. https://app.cochat.ai/papers/10.1186/s12911-019-0814-z
- The paper
- Soboczenski, F., Trikalinos, T. A., Kuiper, J., Bias, R. G., Wallace, B. C., & Marshall, I. J. (2019). Machine learning to help researchers evaluate biases in clinical trials: a prospective, randomized user study. BMC Medical Informatics and Decision Making. https://doi.org/10.1186/s12911-019-0814-z
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.