The DOI matches its Crossref record; 43 of the 45 references it cites were found in a registry or an index; 17 of the 23 sources its claims cite were checked for support.
Toward systematic review automation: a practical guide to using machine learning tools in research synthesis
Open accessAbstract
Technologies and methods to speed up the production of systematic reviews by reducing the manual labour involved have recently emerged. Automation has been proposed or used to expedite most steps of the systematic review process, including search, screening, and data extraction. However, how these technologies work in practice and when (and when not) to use them is often not clear to practitioners. In this practical guide, we provide an overview of current machine learning methods that have been proposed to expedite evidence synthesis. We also offer guidance on which of these are ready for use, their strengths and weaknesses, and how a systematic review team might go about using them in practice.
Overview
CoChat Summary
TL;DR
This practical guide surveys machine learning and natural language processing methods proposed to semi-automate systematic review steps, especially search, screening, and data extraction. It assesses which tools are ready for use, their strengths and weaknesses, and how review teams might adopt them in practice.
Key findings
- Machine learning for identifying RCTs is the most mature and ready for practice: RCT tagger achieved AUROC = 0.973, and RobotSearch achieved AUROC = 0.987 while reducing irrelevant articles retrieved by roughly half compared with the Cochrane Highly Sensitive Search Strategy without losing additional RCTs.
- Abstract screening systems are mature but semi-automatic and human-in-the-loop; available systems include Abstrackr, SWIFT-Review, EPPI reviewer, and RobotAnalyst, but the safe stopping point for manual screening is unclear and varies across reviews.
- Data extraction remains less mature and not readily accessible: ExaCT retrieves the five most likely sentences and had recall of 72% to 100% across variables; RobotReviewer’s bias classifications are reasonable but less accurate than published Cochrane reviews, though its identified sentences were similarly relevant to bias decisions.
- Synthesis automation is beyond current ML/NLP capabilities; full automation remains distant, and most tools are designed as human-in-the-loop systems.
Methods
- Design: practical guide and non-exhaustive survey aimed at non-computer scientists, providing an overview of current ML/NLP methods and guidance on practical use.
- Data/sample: ML systems available for use in practice were identified by manually screening records in SR Toolbox on January 3, 2019; the authors excluded ML methods from academic papers unless a system was available and excluded software tools unless they used ML.
- Analysis: narrative synthesis of methods and tools, with assessments of readiness, strengths, weaknesses, and suggested workflows for systematic review teams.
Limitations
- Stated: the review is not exhaustive; it covers tools available at the time of writing and excludes methods without an available system.
- Stated: many tools are academic prototypes that do not age well, with broken links, difficult or slow interfaces, and server errors; maintaining professional-grade software is difficult under fixed-term grant funding.
- Stated: automated abstract screening has an unclear and review-specific stopping point, and data extraction technologies remain in formative stages; full automation is a distant goal.
- Evident from the text: the paper does not report a new empirical evaluation or quantitative synthesis of tool performance; its readiness recommendations are expert assessments based on the described literature and available systems.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures




Verification
Claims the paper makes
13 Valid · 1 Not valid · 3 Not checked · 8 Own results and interpretations
Every source the paper cites for this claim was checked and supports it.
In the paper
“Thalia is a machine learning system (based on CRFs, reviewed above) that automatically indexes new PubMed articles daily for chemicals, diseases, drugs, genes, metabolites, proteins, species, and anatomical entities.”
§ Toward systematic review automation: a practical guide to using machine learning tools in research synthesis > Machine learning tools available for use in practice > Search
What the sources say
- Supportedcites [26] Soto, Axel J, Przybyła P, Ananiadou S. “Thalia: Semantic Search Engine for Biomedical Abstracts.” Bioinformatics. 2019;35(10):1799-1801. sourceExperimental — accuracy not yet measured
The Supplementary information describes Thalia as automatically downloading PubMed updates, mining concepts and adding them to its search index daily. It lists all eight entity types and says its recognition models use conditional random fields as well as dictionary matching; the claim does not say CRFs are the only method. (LLM, conf=0.90)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“Recently, Nye et al. released the EBM-NLP dataset [23], which comprises ~ 5000 abstracts of RCT reports manually annotated in detail.”
§ Toward systematic review automation: a practical guide to using machine learning tools in research synthesis > Machine learning tools available for use in practice > Data extraction
What the sources say
- Supportedcites [23] Nye B, Jessy Li J, Patel R, Yang Y, Marshall IJ, Nenkova A, et al. A corpus with multi-level annotations of patients, interventions and outcomes to support language processing for medical literature. Proc Conf Assoc Comput Linguist Meet. 2018;2018:197–207. source
The Introduction says Nye et al. introduced EBM-NLP as a publicly available corpus of approximately 5,000 medical abstracts describing clinical trials, annotated in detail for PICO elements. The Conclusions specify that the abstracts describe randomized controlled trials and that the corpus is publicly available. (LLM, conf=0.94)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The sources with a verdict support this claim; the rest weren’t checked yet, have no full text we could read, or their verdicts aren’t rated yet.
In the paper
“For systematic reviews of RCTs, there exist only a few prototype platforms that make such technologies available (ExaCT [33] and RobotReviewer [12, 34, 35] being among these).”
§ Toward systematic review automation: a practical guide to using machine learning tools in research synthesis > Machine learning tools available for use in practice > Data extraction
What the sources say
- Supportedcites [33] Kiritchenko S, de Bruijn B, Carini S, Martin J, Sim I. ExaCT: automatic extraction of clinical trial characteristics from journal publications. BMC Med Inform Decis Mak. 2010;10:56. source
The Background identifies ExaCT as a system for extracting characteristics from RCT publications and describes its browser-based interface. That supports the part of the citing sentence for which the ExaCT paper is cited: identifying ExaCT as an available extraction platform. (LLM, conf=0.92)
- Supportedcites [12] Marshall IJ, Kuiper J, Wallace BC. RobotReviewer: evaluation of a system for automatically assessing bias in clinical trials. J Am Med Inform Assoc. 2016;23:193–201. source
The citation identifies RobotReviewer as an example, rather than asking this paper to establish how many platforms exist or describe ExaCT. The Methods describe a prototype web-based tool for systematic reviewers that presents risk-of-bias assessments for uploaded articles; the Objectives state that it identifies supporting text from trial reports. The indexed copy provides no page number for these sections. (LLM, conf=0.91)
- Supportedcites [34] Marshall IJ, Kuiper J, Banner E, Wallace BC. Automating biomedical evidence synthesis: RobotReviewer. Proc Conf Assoc Comput Linguist Meet. 2017;2017:7–12. source
The Overview describes RobotReviewer as a web-based tool that extracts information from RCT articles and provides access through a prototype graphical interface and REST API. This supports the citation's identification of RobotReviewer as an example of a platform making data-extraction technology available; ExaCT is cited separately. (LLM, conf=0.92)
- Not checked — no full text of the sourcecites [35] Wallace BC, Kuiper J, Sharma A, Zhu MB, Marshall IJ. Extracting PICO sentences from clinical trial reports using supervised distant supervision. J Mach Learn Res. 2016;17:1–25.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“However, the sentences identified were found to be similarly relevant to bias decisions as those in Cochrane reviews [12].”
§ Toward systematic review automation: a practical guide to using machine learning tools in research synthesis > Machine learning tools available for use in practice > Data extraction
What the sources say
- Supportedcites [12] Marshall IJ, Kuiper J, Wallace BC. RobotReviewer: evaluation of a system for automatically assessing bias in clinical trials. J Am Med Inform Assoc. 2016;23:193–201. source
The Discussion states that RobotReviewer identified text supporting risk-of-bias judgments of similar quality to text in published systematic reviews. The Results (Table 2) show this for the best of its top three sentences: 60.4% were rated highly relevant, versus 56.5% of text quoted in Cochrane reviews; the difference was not statistically significant. (LLM, conf=0.94)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“The team validated the performance on a withheld portion of the same dataset, finding the system discriminated accurately between RCTs and non-RCTs (area under the receiver operating characteristics curve (AUROC) = 0.973).”
§ Toward systematic review automation: a practical guide to using machine learning tools in research synthesis > Machine learning tools available for use in practice > Search
What the sources say
- Supportedcites [25] Cohen AM, Smalheiser NR, McDonagh MS, Yu C, Adams CE, Davis JM, et al. Automated confidence ranked classification of randomized controlled trial articles: an aid to evidence-based medicine. J Am Med Inform Assoc. 2015;22:707–17. source
Table 1 (Materials and Methods; page not specified) reports an AUC of 0.973 for the CITATION_PLUS_MESH RCT classifier on the held-out 2011 test set. The Results section also describes validation on held-out 2011 and 2012 data; the 2012 AUC was 0.972. (LLM, conf=0.94)
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
9 more claims · 3 with their cited sources checked
Create a free account to see all 25 claims and every source verdict.
- Probably not a substantial claim. 2
- It may state two separate claims as one. 9
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Unique to systematic reviewing is that not only do all relevant records (that is, titles and abstracts) need to seen by a researcher, but an extremely diverse range of concepts also need to be learned, thereby requiring flexibility in the modelling approach as well as careful error evaluation 11 .”
From An open source machine learning framework for efficient and transparent systematic reviews (2021) · cites this paper as [11]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Therefore, research on this topic suggests that tools for automated or semi-automated data extraction based on NLP are not yet mature enough for practical use.[6]”
From Artificial Intelligence–Assisted Data Extraction With a Large Language Model: A Study Within Reviews (2025) · cites this paper as [6]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“One criticism of systematic review automation tools previously is that they are often found as discrete, scattered pieces of academic code which require substantial technical expertise to use in practice [11,26].”
From In a pilot study, automated real-time systematic review updates were feasible, accurate, and work-saving (2023) · cites this paper as [11]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“In fact, if the literature of a field is growing faster than the amount of time available for systematic reviews, adequate manual review of this field then becomes impossible 11 .”
From An open source machine learning framework for efficient and transparent systematic reviews (2021) · cites this paper as [11]
The cited source says what the claim attributes to it.
In the citing paper
“These tools use learning algorithms to acquire knowledge from training data, enabling the identification of specific keywords, phrases, or patterns in newly encountered abstracts and the subsequent prediction of their relevance (Marshall & Wallace, 2019).”
From Screening Smarter, Not Harder: A Comparative Analysis of Machine Learning Screening Algorithms and Heuristic Stopping Criteria for Systematic Reviews in Educational Research (2024) · cites this paper as Marshall & Wallace, 2019
14 more claims about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318
Book Review: Pattern Recognition and Machine Learning
Nasrabadi, N. M. (2007) · Journal of Electronic Imaging · doi:10.1117/1.2819119 · Cited by 22,066
Domain-Adversarial Training of Neural Networks
Ganin, Y., Ustinova, E., Ajakan, H., Germain, P., Larochelle, H., Laviolette, F., Marchand, M., & Lempitsky, V. (2017) · Advances in Computer Vision and Pattern Recognition · doi:10.1007/978-3-319-58347-1_10 · Cited by 8,318
Text categorization with Support Vector Machines: Learning with many relevant features
Joachims, T. (1998) · Lecture Notes in Computer Science · doi:10.1007/bfb0026683 · Cited by 7,999 · cited 1×
“Many state-of-the-art systems use more complex models than logistic regression (and in particular more sophisticated methods for representing documents [13], obtaining coefficients [14], or both [15]).”
BackgroundNeural Architectures for Named Entity Recognition
Lample, G., Ballesteros, M., Subramanian, S., Kawakami, K., & Dyer, C. (2016) · Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/n16-1030 · Cited by 4,401 · cited 1×
“…are based on neural networks, and specifically recurrent neural networks, or RNNs. Long short-term memory networks (LSTMs) [19] combined with CRFs (LSTM-CRFs) [19 – 21] have in particular shown compelling performance on such tasks generally, for extraction of data from RCTs specifically [22, 23].”
BackgroundEnd-to-end Sequence Labeling via Bi-directional LSTM-CNNs-CRF
Ma, X., & Hovy, E. (2016) · Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p16-1101 · Cited by 2,567
word2vec Explained: deriving Mikolov et al.'s negative-sampling word-embedding method
Goldberg, Y., & Levy, O. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1402.3722 · Cited by 1,245
Seventy-five trials and eleven systematic reviews a day: how will we ever keep up?
Bastian, H., Glasziou, P., & Chalmers, I. (2010) · PLoS Medicine · doi:10.1371/journal.pmed.1000326 · Cited by 1,188
Settles, B. (2012) · Cited by 1,183
Borah, R., Brown, A. W., Capers, P. L., & Kaiser, K. A. (2017) · BMJ Open · doi:10.1136/bmjopen-2016-012545 · Cited by 737
Research works printed but not linked: 1
- Bishop CM. Pattern recognition and machine learning. Springer New York; 2016.
Cited by
The PRISMA 2020 statement: An updated guideline for reporting systematic reviews
Page, M. J., McKenzie, J. E., Bossuyt, P. M., Boutron, I., Hoffmann, T. C., Mulrow, C. D., Shamseer, L., Tetzlaff, J. M., Akl, E. A., Brennan, S. E., et al. (2021) · BMJ · doi:10.1136/bmj.n71 · Cited by 108,283
The PRISMA 2020 statement: an updated guideline for reporting systematic reviews
Page, M. J., McKenzie, J. E., Bossuyt, P. M., Boutron, I., Hoffmann, T. C., Mulrow, C. D., Shamseer, L., Tetzlaff, J. M., Akl, E. A., Brennan, S. E., et al. (2021) · Systematic Reviews · doi:10.1186/s13643-021-01626-4 · Cited by 13,874
Page, M. J., Moher, D., Bossuyt, P. M., Boutron, I., Hoffmann, T. C., Mulrow, C. D., Shamseer, L., Tetzlaff, J. M., Akl, E. A., Brennan, S. E., et al. (2021) · BMJ · doi:10.1136/bmj.n160 · Cited by 13,262
The PRISMA 2020 Statement: An Updated Guideline for Reporting Systematic Reviews
Page, M. J., McKenzie, J. E., Bossuyt, P. M., Boutron, I., Hoffmann, T. C., Mulrow, C. D., Shamseer, L., Tetzlaff, J. M., Akl, E. A., Brennan, S. E., et al. (2021) · International Journal of Surgery · doi:10.1016/j.ijsu.2021.105906 · Cited by 11,748
The PRISMA 2020 statement: An updated guideline for reporting systematic reviews
Page, M. J., McKenzie, J. E., Bossuyt, P. M., Boutron, I., Hoffmann, T. C., Mulrow, C. D., Shamseer, L., Tetzlaff, J. M., Akl, E. A., Brennan, S. E., et al. (2021) · PLoS Medicine · doi:10.1371/journal.pmed.1003583 · Cited by 5,148
Metadata
- Authors
- Marshall, I. J., & Wallace, B. C.
- Affiliations
- King's College London (GB)
- Northeastern University (US)
- Corresponding author
- Marshall, Iain J.
- Venue
- Systematic Reviews
- Published
- 2019-07-11
- Volume
- 8
- Issue
- 1
- Pages
- 163-163
- ISSN
- 2046-4053 (ISSN-L)
- PMC
- PMC6621996
- OpenAlex
- W2961191798
- Cited by
- 638
- References
- 45
- Citations by year
- 2026: 99 · 2025: 107 · 2024: 115 · 2023: 86 · 2022: 71 · 2021: 94 · 2020: 61 · 2019: 5
- FWCI
- 11.72
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- creativecommons.org/licenses/by/4.0/ (tdm)
- creativecommons.org/licenses/by/4.0/ (vor)
- Locations
- Systematic Reviews (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Research Portal (King's College London) (submittedVersion, other-oa)
- Funders
- Medical Research Council (MR/N015185/1)
- U.S. National Library of Medicine (R01‐LM012086‐01A1)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › General Social Sciences › Computational and Text Analysis Methods
- Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
- Keywords
- systematic review automation, machine learning, evidence synthesis, literature search automation, data extraction
- MeSH terms
- Machine Learning, Systematic Reviews as Topic, Automation, Humans, Practice Guidelines as Topic, Evidence-Based Medicine
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-09-29, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-09-29 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Toward systematic review automation: a practical guide to using machine learning tools in research synthesis. https://app.cochat.ai/papers/10.1186/s13643-019-1074-9
- The paper
- Marshall, I. J., & Wallace, B. C. (2019). Toward systematic review automation: a practical guide to using machine learning tools in research synthesis. Systematic Reviews. https://doi.org/10.1186/s13643-019-1074-9
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.