CoChat

Editorial: Systematic review automation thematic series

Open access
Lau, Joseph · 2019 · Systematic Reviews

The DOI matches its Crossref record; 4 of the 4 references it cites were found in a registry or an index.

Abstract

It has been over 35 years since health disciplines adopted systematic reviews and meta-analyses to inform healthcare decisions and policies. Advances in methodology and creation of standards have improved the reliability and consistency of these syntheses of research findings. Conducting a systematic review involves complex processes usually requiring many tedious steps and numerous hours of expert labor. A typical systematic review can take a team of researchers a year or more to produce [1]. Systematic reviews are costly. The cost varies according to the number of questions to answer and the amount of literature to evaluate. For instance, an Agency for Healthcare Research and Quality comparative effectiveness review with five key questions and the need to review about 10,000 citations may cost upward of US$300,000. Researchers have long used computer technologies to facilitate the production of systematic reviews. They use search engines to conduct searches for potentially eligible studies, use database programs to manage citations, create spreadsheets to collect and manage data, and use statistical software to perform data analyses. Dedicated software has been developed to assist specific aspects of the systematic review process. However, these tools have had minimal impact on the amount of time and costs to produce a systematic review. Had technological innovations of producing systematic reviews paralleled that of microprocessor development, Moore’s law of doubling of computer power every 2 years would imply that a team of researchers today could produce over 260,000 systematic reviews in 1 year using the same amount of resources spent 35 years ago. Looking at it in a different way, ignoring inflation and other cost considerations, hypothetical technological advancements would allow a team today to produce a systematic review for less than a dollar. While these highly exaggerated extrapolations are based on unrealistic assumptions, where we are today, in terms of incorporating technology into the review process, is off by many orders of magnitude. Importantly, systematic reviews today take longer to produce and cost more than 35 years ago. Given the comprehensive number of healthcare topics requiring formal systematic review and the need to update existing reviews or establish a living systematic review, one can readily see that the demands of research synthesis far outstrip the current supply and capacity of systematic review production, even with new generations of trained reviewers and thousands of Cochrane volunteers, and other groups worldwide. To keep up, the societal costs for review production will reach many tens of millions of dollars (perhaps even more). There is little debate that automation is needed to produce and maintain systematic reviews and to make their availability timely and affordable. Furthermore, the information provided in systematic reviews must be made available in a form that will improve their uptake by the users such as guideline developers and other decision makers. Having automated systematic reviews talk to systems that could directly incorporate their results is an opportunity to create a seamless system for evidence-based healthcare decisions and policies. Over the past 15 years, researchers have begun to develop automation tools for systematic reviews. Title and abstract screening is a laborious but essential task in systematic review. This task is one of several bottlenecks in the systematic review process that can take up to several weeks, sometimes months, to complete. It is a low hanging fruit that lends itself to automation using machine learning technology. Some of these tools have advanced far enough to allow them to be used routinely by systematic reviewers supported by the development team. However, the lack of user-friendly software and commercialized products has hindered their widespread adoption. This situation is very likely to change in the near future. Data extraction is another labor-intensive task that is a bottleneck in the review process. The potential for increasing efficiency in data extraction through automation is huge. One can envision that automation could generate evidence maps almost at a push of a button. While we are not there yet, we are at the brink of a revolution in automation of systematic reviews, a much-needed development that could spur evidence-based health decision-making and policy. Correspondence: joseph_lau@brown.edu Brown University School of Public Health, Providence, RI 02903, USA

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1186/s13643-019-0974-z

Cites

4 of 4 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 4 references

Cited by

See all 29

Metadata

Authors
Lau, J.
Affiliations
Brown University (US)
Corresponding author
Lau, Joseph
Venue
Systematic Reviews
Published
2019-03-11
Volume
8
Issue
1
Pages
70-70
ISSN
2046-4053 (ISSN-L)
OpenAlex
W2932069670
Cited by
27
References
4
Citations by year
2026: 2 · 2025: 2 · 2023: 4 · 2022: 2 · 2021: 7 · 2020: 9 · 2019: 1
Type
journal article
Language
en
Open access
Licence
  • cc-by
  • creativecommons.org/licenses/by/4.0/ (tdm)
Locations
  • Systematic Reviews (publishedVersion, cc-by)
  • Europe PMC (PubMed Central) (publishedVersion, other-oa)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
Topics
  • Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
  • Social Sciences › Library and Information Sciences › scientometrics and bibliometrics research
  • Social Sciences › Library and Information Sciences › Academic Publishing and Open Access
Keywords
systematic review automation, automated data extraction, systematic review methodology
MeSH terms
Machine Learning, Systematic Reviews as Topic, Automation, Efficiency, Humans, Software, Search Engine

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Editorial: Systematic review automation thematic series. https://app.cochat.ai/papers/10.1186/s13643-019-0974-z
The paper
Lau, J. (2019). Editorial: Systematic review automation thematic series. Systematic Reviews. https://doi.org/10.1186/s13643-019-0974-z

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.