Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat

The DOI matches its Crossref record; 28 of the 28 references it cites were found in a registry or an index; 4 of the 21 sources its claims cite were checked for support.

Performance and usability of machine learning for screening in systematic reviews: a comparative evaluation of three tools

Open access

Abstract

Background: We explored the performance of three machine learning tools designed to facilitate title and abstract screening in systematic reviews (SRs) when used to (a) eliminate irrelevant records (automated simulation) and (b) complement the work of a single reviewer (semi-automated simulation). We evaluated user experiences for each tool. Methods: We subjected three SRs to two retrospective screening simulations. In each tool (Abstrackr, DistillerSR, RobotAnalyst), we screened a 200-record training set and downloaded the predicted relevance of the remaining records. We calculated the proportion missed and workload and time savings compared to dual independent screening. To test user experiences, eight research staff tried each tool and completed a survey. Results: Using Abstrackr, DistillerSR, and RobotAnalyst, respectively, the median (range) proportion missed was 5 (0 to 28) percent, 97 (96 to 100) percent, and 70 (23 to 100) percent for the automated simulation and 1 (0 to 2) percent, 2 (0 to 7) percent, and 2 (0 to 4) percent for the semi-automated simulation. The median (range) workload savings was 90 (82 to 93) percent, 99 (98 to 99) percent, and 85 (85 to 88) percent for the automated simulation and 40 (32 to 43) percent, 49 (48 to 49) percent, and 35 (34 to 38) percent for the semi-automated simulation. The median (range) time savings was 154 (91 to 183), 185 (95 to 201), and 157 (86 to 172) hours for the automated simulation and 61 (42 to 82), 92 (46 to 100), and 64 (37 to 71) hours for the semi-automated simulation. Abstrackr identified 33–90% of records missed by a single reviewer. RobotAnalyst performed less well and DistillerSR provided no relative advantage. User experiences depended on user friendliness, qualities of the user interface, features and functions, trustworthiness, ease and speed of obtaining predictions, and practicality of the export file(s). Conclusions: The workload savings afforded in the automated simulation came with increased risk of missing relevant records. Supplementing a single reviewer’s decisions with relevance predictions (semi-automated simulation) sometimes reduced the proportion missed, but performance varied by tool and SR. Designing tools based on reviewers’ self-identified preferences may improve their compatibility with present workflows. Systematic review registration: Not applicable.

Overview

CoChat Summary

TL;DR

In retrospective simulations across three systematic reviews, three machine-learning screening tools differed substantially in performance and usability. Abstrackr was most usable and, when used to complement a single reviewer, sometimes reduced missed records, but automated exclusion carried a high risk of missing relevant studies.

Key findings

  • In the automated simulation, median (range) proportion missed was 5% (0–28) for Abstrackr, 97% (96–100) for DistillerSR, and 70% (23–100) for RobotAnalyst; workload savings were 90% (82–93), 99% (98–99), and 85% (84–88), respectively, and estimated time savings were 154 (91–183), 185 (95–201), and 157 (86–172) hours.
  • In the semi-automated simulation, median (range) proportion missed was 1% (0–2) for Abstrackr, 2% (0–7) for DistillerSR, and 2% (0–4) for RobotAnalyst; workload savings were 40% (32–43), 49% (48–49), and 35% (34–38), and time savings were 61 (42–82), 92 (46–100), and 64 (37–71) hours. Abstrackr identified 33–90% of records missed by a single reviewer; RobotAnalyst performed less well and DistillerSR provided no relative advantage.
  • Among 8 research staff (73% response rate), median (IQR) System Usability Scale scores were 79 (23) for Abstrackr, 64 (31) for DistillerSR, and 31 (8) for RobotAnalyst; 62% chose Abstrackr as first choice, 38% chose DistillerSR as first choice, and 88% ranked RobotAnalyst last.
  • In a post hoc 500-record training set tested only in Abstrackr, automated-simulation median proportion missed increased from 28 to 41% for Antipsychotics and 5 to 9% for Bronchiolitis, with no change for Visual Acuity; workload and estimated time savings increased.

Methods

  • Retrospective screening simulations using three systematic reviews (Antipsychotics, Bronchiolitis, Visual Acuity; screening workloads 12,156, 5,861, and 11,229 records) and three tools (Abstrackr, DistillerSR/DistillerAI, RobotAnalyst). A 200-record training set was used, except Visual Acuity required 300 records in Abstrackr before predictions were produced.
  • Two simulations were tested: automated exclusion of records predicted irrelevant, and semi-automated screening where predictions complemented one human reviewer. Performance metrics were calculated from 2 × 2 cross-tabulations: proportion missed, workload savings, and estimated time savings assuming 0.5 min per record and an 8-h work day.
  • User experience testing involved 8 research staff who completed a screening exercise in each tool and a REDCap survey including the System Usability Scale; quantitative data were summarized and qualitative data analyzed thematically.

Limitations

  • The paper states that training sets differed across tools and reviews, standard tool settings were used, the training set was 200 records, and the sample was only three systematic reviews; findings should not be generalized to other tools, reviews, or semi-automated screening approaches. It also states that time savings did not account for troubleshooting usability issues, variability in screening time, or differences between obviously excluded and uncertain records, and that the impact of missed studies on review results was not investigated.
  • Evident from the text: the systematic reviews and user sample were convenient samples, and RobotAnalyst failed to upload full RIS record lists for two reviews, requiring manual screening assumptions for 170 Bronchiolitis and 183 Visual Acuity records.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1186/s13643-019-1222-2

Cites

28 of 28 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Figures

Fig. 1 Proportion missed (percent) by tool and systematic review, automated simulation
Fig. 1 Proportion missed (percent) by tool and systematic review, automated simulationCC BY · PubMed Central
Fig. 2 Workload savings (percent) by tool and systematic review, automated simulation
Fig. 2 Workload savings (percent) by tool and systematic review, automated simulationCC BY · PubMed Central
Fig. 3 Estimated time savings (days) by tool and systematic review, automated simulation
Fig. 3 Estimated time savings (days) by tool and systematic review, automated simulationCC BY · PubMed Central
Fig. 4 Proportion missed (percent) by tool and systematic review, semi-automated simulation
Fig. 4 Proportion missed (percent) by tool and systematic review, semi-automated simulationCC BY · PubMed Central
Fig. 5 Workload savings (percent) by tool and systematic review, semi-automated simulation
Fig. 5 Workload savings (percent) by tool and systematic review, semi-automated simulationCC BY · PubMed Central
Fig. 6 Estimated time savings (days) by tool and systematic review, semi-automated simulation
Fig. 6 Estimated time savings (days) by tool and systematic review, semi-automated simulationCC BY · PubMed Central

Verification

Claims the paper makes

2 Valid · 0 Not valid · 10 Not checked · 53 Own results and interpretations

15 more claims

Create a free account to see all 65 claims and to verify the 17 references still open.

See all claims — free account
  • It may state two separate claims as one. 34
  • Probably not a substantial claim. 19
  • The quote only partly states this claim. 1

Create a free account to see these claims.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

2 more claims about this paper

Create a free account to see every claim other papers make about this one.

See all claims — free account

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

3 of 3

Citations

Cites

Show all 28 references

Research works printed but not linked: 2

  1. Pillay J, Freeman EE, Hodge W, MacGregor T, Featherstone R, Vandermeer B, et al. Screening for impaired visual acuity and vision-related functional limitations in adults 65 years and older in primary health care: systematic review. Edmonton: Evidence Review Synthesis Centre; 2017. http://canadiantaskforce.ca/ctfphc-guidelines/overview/.
  2. Chen Y. Developing stopping rules for a ML system in citation screening [dissertation]. Providence: Brown University; 2019.

Cited by

See all 134

Metadata

Authors
Gates, A., Guitard, S., Pillay, J., Elliott, S. A., Dyson, M. P., Newton, A. S., & Hartling, L.
Affiliations
University of Alberta (CA)
Corresponding author
Hartling, Lisa
Venue
Systematic Reviews
Published
2019-11-15
Volume
8
Issue
1
Pages
278-278
ISSN
2046-4053 (ISSN-L)
OpenAlex
W2982683456
Cited by
133
References
28
Citations by year
2026: 24 · 2025: 19 · 2024: 22 · 2023: 23 · 2022: 12 · 2021: 14 · 2020: 18
FWCI
2.23
Type
journal article
Language
en
Open access
Licence
Locations
  • Systematic Reviews (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
Funders
Agency for Healthcare Research and Quality (290-2015-00001-I)
Topics
  • Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
  • Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
  • Social Sciences › General Social Sciences › Computational and Text Analysis Methods
Keywords
title and abstract screening, machine learning, automated screening, semi-automated screening, reviewer workload, user experience, user interface, user credibility, time savings
MeSH terms
Abstracting and Indexing, Machine Learning, Systematic Reviews as Topic, Humans, Software, Time Factors, Reproducibility of Results, Information Storage and Retrieval, Workload

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

A language model drew the claims from the paper on 2026-10-04, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-04 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (qwen3.8-omni-flash).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Performance and usability of machine learning for screening in systematic reviews: a comparative evaluation of three tools. https://app.cochat.ai/papers/10.1186/s13643-019-1222-2
The paper
Gates, A., Guitard, S., Pillay, J., Elliott, S. A., Dyson, M. P., Newton, A. S., & Hartling, L. (2019). Performance and usability of machine learning for screening in systematic reviews: a comparative evaluation of three tools. Systematic Reviews. https://doi.org/10.1186/s13643-019-1222-2

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.