Editorial: Systematic review automation thematic series
Open accessThe DOI matches its Crossref record; 4 of the 4 references it cites were found in a registry or an index.
Abstract
It has been over 35 years since health disciplines adopted systematic reviews and meta-analyses to inform healthcare decisions and policies. Advances in methodology and creation of standards have improved the reliability and consistency of these syntheses of research findings. Conducting a systematic review involves complex processes usually requiring many tedious steps and numerous hours of expert labor. A typical systematic review can take a team of researchers a year or more to produce [1]. Systematic reviews are costly. The cost varies according to the number of questions to answer and the amount of literature to evaluate. For instance, an Agency for Healthcare Research and Quality comparative effectiveness review with five key questions and the need to review about 10,000 citations may cost upward of US$300,000. Researchers have long used computer technologies to facilitate the production of systematic reviews. They use search engines to conduct searches for potentially eligible studies, use database programs to manage citations, create spreadsheets to collect and manage data, and use statistical software to perform data analyses. Dedicated software has been developed to assist specific aspects of the systematic review process. However, these tools have had minimal impact on the amount of time and costs to produce a systematic review. Had technological innovations of producing systematic reviews paralleled that of microprocessor development, Moore’s law of doubling of computer power every 2 years would imply that a team of researchers today could produce over 260,000 systematic reviews in 1 year using the same amount of resources spent 35 years ago. Looking at it in a different way, ignoring inflation and other cost considerations, hypothetical technological advancements would allow a team today to produce a systematic review for less than a dollar. While these highly exaggerated extrapolations are based on unrealistic assumptions, where we are today, in terms of incorporating technology into the review process, is off by many orders of magnitude. Importantly, systematic reviews today take longer to produce and cost more than 35 years ago. Given the comprehensive number of healthcare topics requiring formal systematic review and the need to update existing reviews or establish a living systematic review, one can readily see that the demands of research synthesis far outstrip the current supply and capacity of systematic review production, even with new generations of trained reviewers and thousands of Cochrane volunteers, and other groups worldwide. To keep up, the societal costs for review production will reach many tens of millions of dollars (perhaps even more). There is little debate that automation is needed to produce and maintain systematic reviews and to make their availability timely and affordable. Furthermore, the information provided in systematic reviews must be made available in a form that will improve their uptake by the users such as guideline developers and other decision makers. Having automated systematic reviews talk to systems that could directly incorporate their results is an opportunity to create a seamless system for evidence-based healthcare decisions and policies. Over the past 15 years, researchers have begun to develop automation tools for systematic reviews. Title and abstract screening is a laborious but essential task in systematic review. This task is one of several bottlenecks in the systematic review process that can take up to several weeks, sometimes months, to complete. It is a low hanging fruit that lends itself to automation using machine learning technology. Some of these tools have advanced far enough to allow them to be used routinely by systematic reviewers supported by the development team. However, the lack of user-friendly software and commercialized products has hindered their widespread adoption. This situation is very likely to change in the near future. Data extraction is another labor-intensive task that is a bottleneck in the review process. The potential for increasing efficiency in data extraction through automation is huge. One can envision that automation could generate evidence maps almost at a push of a button. While we are not there yet, we are at the brink of a revolution in automation of systematic reviews, a much-needed development that could spur evidence-based health decision-making and policy. Correspondence: joseph_lau@brown.edu Brown University School of Public Health, Providence, RI 02903, USA
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“The high demand for systematic reviews far exceeds the available resources [5], making it challenging to provide timely and comprehensive reviews, particularly when the research question is urgent.”
The cited source says what the claim attributes to it.
In the citing paper
“Despite numerous tools for automation, adoption of these tools has lagged [8, 22].”
From Machine learning for screening prioritization in systematic reviews: comparative performance of Abstrackr and EPPI-Reviewer (2020) · cites this paper as [8]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Borah, R., Brown, A. W., Capers, P. L., & Kaiser, K. A. (2017) · BMJ Open · doi:10.1136/bmjopen-2016-012545 · Cited by 737
Marshall, I. J., & Wallace, B. C. (2019) · Systematic Reviews · doi:10.1186/s13643-019-1074-9 · Cited by 638
O’Connor, A. M., Tsafnat, G., Thomas, J., Glasziou, P., Gilbert, S. B., & Hutton, B. (2019) · Systematic Reviews · doi:10.1186/s13643-019-1062-0 · Cited by 167
Time to consider sharing data extracted from trials included in systematic reviews
Wolfenden, L., Grimshaw, J., Williams, C. M., & Yoong, S. L. (2016) · Systematic Reviews · doi:10.1186/s13643-016-0361-y · Cited by 30
Cited by
Assay of renewable energy transition: A systematic literature review
Bhattarai, U., Maraseni, T., & Apan, A. (2022) · Science of The Total Environment · doi:10.1016/j.scitotenv.2022.155159 · Cited by 173
Chai, K. E. K., Lines, R. L. J., Gucciardi, D. F., & Ng, L. (2021) · Systematic Reviews · doi:10.1186/s13643-021-01635-3 · Cited by 190
Scoping Reviews, Systematic Reviews, and Meta-Analysis: Applications in Veterinary Medicine
Sargeant, J. M., & O'Connor, A. M. C. (2020) · Frontiers in Veterinary Science · doi:10.3389/fvets.2020.00011 · Cited by 198
Search strategy formulation for systematic reviews: Issues, challenges and opportunities
MacFarlane, A., Russell‐Rose, T., & Shokraneh, F. (2022) · Intelligent Systems with Applications · doi:10.1016/j.iswa.2022.200091 · Cited by 209
Tsou, A. Y., Treadwell, J. R., Erinoff, E., & Schoelles, K. (2020) · Systematic Reviews · doi:10.1186/s13643-020-01324-7 · Cited by 122
Metadata
- Authors
- Lau, J.
- Affiliations
- Brown University (US)
- Corresponding author
- Lau, Joseph
- ORCID
- Lau, Joseph
- Venue
- Systematic Reviews
- Published
- 2019-03-11
- Volume
- 8
- Issue
- 1
- Pages
- 70-70
- ISSN
- 2046-4053 (ISSN-L)
- PMC
- PMC6410513
- OpenAlex
- W2932069670
- Cited by
- 27
- References
- 4
- Citations by year
- 2026: 2 · 2025: 2 · 2023: 4 · 2022: 2 · 2021: 7 · 2020: 9 · 2019: 1
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- creativecommons.org/licenses/by/4.0/ (tdm)
- Locations
- Systematic Reviews (publishedVersion, cc-by)
- Europe PMC (PubMed Central) (publishedVersion, other-oa)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Library and Information Sciences › scientometrics and bibliometrics research
- Social Sciences › Library and Information Sciences › Academic Publishing and Open Access
- Keywords
- systematic review automation, automated data extraction, systematic review methodology
- MeSH terms
- Machine Learning, Systematic Reviews as Topic, Automation, Efficiency, Humans, Software, Search Engine
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Editorial: Systematic review automation thematic series. https://app.cochat.ai/papers/10.1186/s13643-019-0974-z
- The paper
- Lau, J. (2019). Editorial: Systematic review automation thematic series. Systematic Reviews. https://doi.org/10.1186/s13643-019-0974-z
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.