Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat

The DOI matches its Crossref record; 43 of the 45 references it cites were found in a registry or an index; 17 of the 23 sources its claims cite were checked for support.

Toward systematic review automation: a practical guide to using machine learning tools in research synthesis

Open access
Marshall, Iain J., Wallace, Byron C. · 2019 · Systematic Reviews

Abstract

Technologies and methods to speed up the production of systematic reviews by reducing the manual labour involved have recently emerged. Automation has been proposed or used to expedite most steps of the systematic review process, including search, screening, and data extraction. However, how these technologies work in practice and when (and when not) to use them is often not clear to practitioners. In this practical guide, we provide an overview of current machine learning methods that have been proposed to expedite evidence synthesis. We also offer guidance on which of these are ready for use, their strengths and weaknesses, and how a systematic review team might go about using them in practice.

Overview

CoChat Summary

TL;DR

This practical guide surveys machine learning and natural language processing methods proposed to semi-automate systematic review steps, especially search, screening, and data extraction. It assesses which tools are ready for use, their strengths and weaknesses, and how review teams might adopt them in practice.

Key findings

  • Machine learning for identifying RCTs is the most mature and ready for practice: RCT tagger achieved AUROC = 0.973, and RobotSearch achieved AUROC = 0.987 while reducing irrelevant articles retrieved by roughly half compared with the Cochrane Highly Sensitive Search Strategy without losing additional RCTs.
  • Abstract screening systems are mature but semi-automatic and human-in-the-loop; available systems include Abstrackr, SWIFT-Review, EPPI reviewer, and RobotAnalyst, but the safe stopping point for manual screening is unclear and varies across reviews.
  • Data extraction remains less mature and not readily accessible: ExaCT retrieves the five most likely sentences and had recall of 72% to 100% across variables; RobotReviewer’s bias classifications are reasonable but less accurate than published Cochrane reviews, though its identified sentences were similarly relevant to bias decisions.
  • Synthesis automation is beyond current ML/NLP capabilities; full automation remains distant, and most tools are designed as human-in-the-loop systems.

Methods

  • Design: practical guide and non-exhaustive survey aimed at non-computer scientists, providing an overview of current ML/NLP methods and guidance on practical use.
  • Data/sample: ML systems available for use in practice were identified by manually screening records in SR Toolbox on January 3, 2019; the authors excluded ML methods from academic papers unless a system was available and excluded software tools unless they used ML.
  • Analysis: narrative synthesis of methods and tools, with assessments of readiness, strengths, weaknesses, and suggested workflows for systematic review teams.

Limitations

  • Stated: the review is not exhaustive; it covers tools available at the time of writing and excludes methods without an available system.
  • Stated: many tools are academic prototypes that do not age well, with broken links, difficult or slow interfaces, and server errors; maintaining professional-grade software is difficult under fixed-term grant funding.
  • Stated: automated abstract screening has an unclear and review-specific stopping point, and data extraction technologies remain in formative stages; full automation is a distant goal.
  • Evident from the text: the paper does not report a new empirical evaluation or quantitative synthesis of tool performance; its readiness recommendations are expert assessments based on the described literature and available systems.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1186/s13643-019-1074-9

Cites

43 of 43 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Figures

Fig. 1 Classifying text using machine learning, in this example logistic regression with a ‘bag of words’ representation of the texts. The system is ‘trained’, learning a coefficient (or weight) for each unique word in a manually labelled set of documents (typically in the 1000s). In use, the learned coefficients are used to predict a probability for an unknown document
Fig. 1 Classifying text using machine learning, in this example logistic regression with a ‘bag of words’ representation of the texts. The system is ‘trained’, learning a coefficient (or weight) for each unique word in a manually labelled set of documents (typically in the 1000s). In use, the learned coefficients are used to predict a probability for an unknown documentCC BY · PubMed Central
Fig. 2 Bag of words modelling for classifying RCTs. Top left: Example of bag of words for three articles. Each column represents a unique word in the corpus (a real example would likely contain columns for 10,000s of words). Top right: Document labels, where 1 = relevant and 0 = irrelevant. Bottom: Coefficients (or weights) are estimated for each word (in this example using logistic regression). In this example, high +ve weights will increase the predicted probability that an unseen article is an RCT where it contains the words ‘random’ or ‘randomized’. The presence of the word ‘systematic’ (with a large negative weight) would reduce the predicted probability that an unseen document is an RCT
Fig. 2 Bag of words modelling for classifying RCTs. Top left: Example of bag of words for three articles. Each column represents a unique word in the corpus (a real example would likely contain columns for 10,000s of words). Top right: Document labels, where 1 = relevant and 0 = irrelevant. Bottom: Coefficients (or weights) are estimated for each word (in this example using logistic regression). In this example, high +ve weights will increase the predicted probability that an unseen article is an RCT where it contains the words ‘random’ or ‘randomized’. The presence of the word ‘systematic’ (with a large negative weight) would reduce the predicted probability that an unseen document is an RCTCC BY · PubMed Central
Fig. 3 Schematic of a typical data extraction process. The above illustration concerns the example task of extracting the study sample size. In general, these tasks involve labelling individual words. The word (or ‘token’) at position t is represented by a vector. This representation may encode which word is at this position and likely also communicates additional features, e.g. whether the word is capitalized or if the word is (inferred to be) a noun. Models for these kinds of tasks attempt to assign labels all T words in a document and for some tasks will attempt to maximize the joint likelihood of these labels to capitalize on correlations between adjacent labels
Fig. 3 Schematic of a typical data extraction process. The above illustration concerns the example task of extracting the study sample size. In general, these tasks involve labelling individual words. The word (or ‘token’) at position t is represented by a vector. This representation may encode which word is at this position and likely also communicates additional features, e.g. whether the word is capitalized or if the word is (inferred to be) a noun. Models for these kinds of tasks attempt to assign labels all T words in a document and for some tasks will attempt to maximize the joint likelihood of these labels to capitalize on correlations between adjacent labelsCC BY · PubMed Central
Fig. 4 Typical workflow for semi-automated abstract screening. The asterisk indicates that with uncertainty sampling, the articles which are predicted with least certainty are presented first. This aims to improve the model accuracy more efficiently
Fig. 4 Typical workflow for semi-automated abstract screening. The asterisk indicates that with uncertainty sampling, the articles which are predicted with least certainty are presented first. This aims to improve the model accuracy more efficientlyCC BY · PubMed Central

Verification

Claims the paper makes

13 Valid · 1 Not valid · 3 Not checked · 8 Own results and interpretations

9 more claims · 3 with their cited sources checked

Create a free account to see all 25 claims and every source verdict.

See all claims — free account
  • Probably not a substantial claim. 2
  • It may state two separate claims as one. 9

Create a free account to see these claims.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

14 more claims about this paper

Create a free account to see every claim other papers make about this one.

See all claims — free account

Citations

Cites

Show all 45 references

Research works printed but not linked: 1

  1. Bishop CM. Pattern recognition and machine learning. Springer New York; 2016.

Cited by

See all 638

Metadata

Authors
Marshall, I. J., & Wallace, B. C.
Affiliations
  • King's College London (GB)
  • Northeastern University (US)
Corresponding author
Marshall, Iain J.
Venue
Systematic Reviews
Published
2019-07-11
Volume
8
Issue
1
Pages
163-163
ISSN
2046-4053 (ISSN-L)
OpenAlex
W2961191798
Cited by
638
References
45
Citations by year
2026: 99 · 2025: 107 · 2024: 115 · 2023: 86 · 2022: 71 · 2021: 94 · 2020: 61 · 2019: 5
FWCI
11.72
Type
journal article
Language
en
Open access
Licence
Locations
  • Systematic Reviews (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
  • Research Portal (King's College London) (submittedVersion, other-oa)
Funders
  • Medical Research Council (MR/N015185/1)
  • U.S. National Library of Medicine (R01‐LM012086‐01A1)
Topics
  • Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
  • Social Sciences › General Social Sciences › Computational and Text Analysis Methods
  • Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
Keywords
systematic review automation, machine learning, evidence synthesis, literature search automation, data extraction
MeSH terms
Machine Learning, Systematic Reviews as Topic, Automation, Humans, Practice Guidelines as Topic, Evidence-Based Medicine

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-09-29, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-09-29 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (gpt-6-sol).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Toward systematic review automation: a practical guide to using machine learning tools in research synthesis. https://app.cochat.ai/papers/10.1186/s13643-019-1074-9
The paper
Marshall, I. J., & Wallace, B. C. (2019). Toward systematic review automation: a practical guide to using machine learning tools in research synthesis. Systematic Reviews. https://doi.org/10.1186/s13643-019-1074-9

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.