CoChat

Machine learning computational tools to assist the performance of systematic reviews: A mapping review.

Open access
Cierco Jimenez, Ramon, Lee, Teresa, Rosillo, Nicolás, Cordova, Reynalda, Cree, Ian A, Gonzalez, Angel, Indave Ruiz, Blanca Iciar · 2022 · BMC Medical Research Methodology

The DOI matches its Crossref record; 65 of the 65 references it cites were found in a registry or an index; 18 of the 68 sources its claims cite were checked for support.

Abstract

Background: Within evidence-based practice (EBP), systematic reviews (SR) are considered the highest level of evidence in that they summarize the best available research and describe the progress in a determined field. Due its methodology, SR require significant time and resources to be performed; they also require repetitive steps that may introduce biases and human errors. Machine learning (ML) algorithms therefore present a promising alternative and a potential game changer to speed up and automate the SR process. This review aims to map the current availability of computational tools that use ML techniques to assist in the performance of SR, and to support authors in the selection of the right software for the performance of evidence synthesis. Methods: The mapping review was based on comprehensive searches in electronic databases and software repositories to obtain relevant literature and records, followed by screening for eligibility based on titles, abstracts, and full text by two reviewers. The data extraction consisted of listing and extracting the name and basic characteristics of the included tools, for example a tool’s applicability to the various SR stages, pricing options, open-source availability, and type of software. These tools were classified and graphically represented to facilitate the description of our findings. Results: A total of 9653 studies and 585 records were obtained from the structured searches performed on selected bibliometric databases and software repositories respectively. After screening, a total of 119 descriptions from publications and records allowed us to identify 63 tools that assist the SR process using ML techniques. Conclusions: This review provides a high-quality map of currently available ML software to assist the performance of SR. ML algorithms are arguably one of the best techniques at present for the automation of SR. The most promising tools were easily accessible and included a high number of user-friendly features permitting the automation of SR and other kinds of evidence synthesis reviews.

Overview

CoChat Summary

TL;DR

This mapping review searched bibliographic databases and software repositories to identify machine-learning tools that assist systematic reviews, finding 63 tools from 119 included descriptions. Most tools target early SR stages, especially screening, and many are free/open-source, but a substantial minority require payment or programming skills.

Key findings

  • From 9,653 database studies and 585 repository records, 119 descriptions yielded 63 ML tools; 52 (83%) were cited 1–5 times, while the top referenced tools were Abstrackr (23), EPPI-Reviewer (17), Rayyan (17), RobotReviewer (14), and RobotAnalyst (11).
  • All 63 tools addressed early SR stages: search 30 (47%), screening 35 (55%), and data extraction 11 (17%); late SR stages had 1 tool count. By software type, 39 (62%) were web applications, 11 (17%) algorithms, 5 (8%) libraries/packages, and 8 (13%) desktop applications.
  • NLP was used by 48 (76%) tools and supervised learning by 37 (58%); 22 (35%) required a programming background, while 41 (65%) could be used by less skilled users.
  • 31 (49%) were open-source, 44 (70%) freely available, and 14 (22%) required payment; 62 (98%) were updated or released in the last 5 years and 29 (46%) in the last year. Fee-based tools were updated in the last year more often (86%) than free tools (30%).

Methods

  • Mapping review with a protocol registered on OSF; searches were run in MEDLINE (PubMed), EMBASE, Web of Science, and repositories CPAN, CRAN, GitHub, NaCTeM, PECL, PyPI, SourceForge, and the SR toolbox from 1 April to 31 May 2021, with no language restrictions.
  • Eligibility included publications or repository records describing ML software to assist any SR stage, with sufficient technical description, in English, Spanish, German, or French, and developed or updated within the last 10 years; reference management software and inaccessible tools were excluded. Two reviewers screened titles/abstracts and full texts; repository records were screened by one reviewer and checked by a second.
  • Data extraction captured computational method, SR stage, release/update dates, licensing and pricing, open-source status, access source, and programming background; findings were compiled in a summary-of-findings table and a mapping infographic.

Limitations

  • Stated: the mapping review may have bias inherent to its methodology; repository search engines lacked advanced search functionalities, so some ML tool registries may not have been detected; new tools developed after the project, such as Elicit, were not considered; and the methodology does not synthesize findings or critically appraise the methodological quality of retrieved publications.
  • Stated: usability and acceptability could not be assessed because such features were poorly reported.
  • Evident: searches ended 31 May 2021, so tools released or updated after that date are absent; reliance on repository records and brief descriptions may underrepresent tools without sufficient technical description or accessible links.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1186/s12874-022-01805-4

Cites

65 of 65 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Figures

Fig. 1 Software tools assisting the Systematic Review process. The central figure represents from the early to the late stages of the systematic review process described by the Cochrane Foundation (Higgins JPT et al., Cochrane Handbook for Systematic Reviews of Interventions version 6.3, Cochrane, 2022). The obtained tools were grouped into blocks depending on the covered SR step (Planning: Process of planning and writing the protocol for a SR; Deduplication: Process of removing duplicated records and articles retrieved by the search in a SR; Screening: Process of screening records and articles retrieved by the search in a SR; Data Extraction: Process of data extraction in the SR included studies; Critical appraisal and/or Bias assessment: Evaluation of the methodological quality and/or risk of bias in a SR included studies; Meta-analysis: Process of pooling findings of included studies, using statistical methods; Summary of findings/Report: Process of summarizing and reporting of findings; Work network: Process of networking; Whole process: whole systematic review process including all steps; Search: process of elaboration of search strategy, running the search and/or obtaining records retrieved by the search; Reference Management: process of screening and selection of records, as well reporting and scientific writing allowing management of large numbers of reference records and in text citations; Figures: Visualisation of data; SR Databases: Registration, collection and dissemination of SRs; Publication: Process of publishing the results)
Fig. 1 Software tools assisting the Systematic Review process. The central figure represents from the early to the late stages of the systematic review process described by the Cochrane Foundation (Higgins JPT et al., Cochrane Handbook for Systematic Reviews of Interventions version 6.3, Cochrane, 2022). The obtained tools were grouped into blocks depending on the covered SR step (Planning: Process of planning and writing the protocol for a SR; Deduplication: Process of removing duplicated records and articles retrieved by the search in a SR; Screening: Process of screening records and articles retrieved by the search in a SR; Data Extraction: Process of data extraction in the SR included studies; Critical appraisal and/or Bias assessment: Evaluation of the methodological quality and/or risk of bias in a SR included studies; Meta-analysis: Process of pooling findings of included studies, using statistical methods; Summary of findings/Report: Process of summarizing and reporting of findings; Work network: Process of networking; Whole process: whole systematic review process including all steps; Search: process of elaboration of search strategy, running the search and/or obtaining records retrieved by the search; Reference Management: process of screening and selection of records, as well reporting and scientific writing allowing management of large numbers of reference records and in text citations; Figures: Visualisation of data; SR Databases: Registration, collection and dissemination of SRs; Publication: Process of publishing the results)CC BY · PubMed Central
Fig. 2 PRISMA flow diagram for the mapping review. Adapted From: Page MJ et al. 2021. SR: Systematic Review; ML: Machine Learning; CPAN: The Comprehensive Perl Archive Network; CRAN: The Comprehensive R Archive Network; NaCTeM: National Centre for Text Mining; PyPI: The Python Package Index
Fig. 2 PRISMA flow diagram for the mapping review. Adapted From: Page MJ et al. 2021. SR: Systematic Review; ML: Machine Learning; CPAN: The Comprehensive Perl Archive Network; CRAN: The Comprehensive R Archive Network; NaCTeM: National Centre for Text Mining; PyPI: The Python Package IndexCC BY · PubMed Central
Fig. 3 Machine learning tools that assist the Systematic Review process. The central figure represents from the early to the late stages of the systematic review process described by the Cochrane Foundation (Higgins JPT et al., Cochrane Handbook for Systematic Reviews of Interventions version 6.3, Cochrane, 2022). The obtained ML tools were grouped into blocks depending on the covered SR step. Next to the tool names, the yellow symbol means the requirement of a payment licencing to access to the full version of the tool and the green symbol open-source availability. SR = Systematic Review, ML = Machine learning
Fig. 3 Machine learning tools that assist the Systematic Review process. The central figure represents from the early to the late stages of the systematic review process described by the Cochrane Foundation (Higgins JPT et al., Cochrane Handbook for Systematic Reviews of Interventions version 6.3, Cochrane, 2022). The obtained ML tools were grouped into blocks depending on the covered SR step. Next to the tool names, the yellow symbol means the requirement of a payment licencing to access to the full version of the tool and the green symbol open-source availability. SR = Systematic Review, ML = Machine learningCC BY · PubMed Central

Verification

Claims the paper makes

8 Valid · 3 Not valid · 17 Not checked · 95 Own results and interpretations

50 more claims

Create a free account to see all 123 claims and to verify the 50 references still open.

See all claims — free account
  • It may state two separate claims as one. 36
  • Probably not a substantial claim. 42

Create a free account to see these claims.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

5 of 7 · the rest aren’t loaded

2 more findings

Create a free account to see every finding of this check.

See all findings — free account

Citations

Cites

Show all 65 references

Research works printed but not linked: 2

  1. Boudin F, Nie J-Y, Dawes M. Clinical information retrieval using document and PICO structure. Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics; Los Angeles, California: Association for Computational Linguistics; 2010. p. 822–30.
  2. The Systematic Review Toolbox (SR toolbox). http://systematicreviewtools.com/. Accessed 3 May 2021.

Cited by

See all 92

Metadata

Authors
Cierco Jimenez, R., Lee, T., Rosillo, N., Cordova, R., Cree, I. A., Gonzalez, A., & Indave Ruiz, B. I.
Affiliations
  • Centre international de recherche sur le cancer (FR)
  • Hospital Universitario 12 De Octubre (ES)
Venue
BMC Medical Research Methodology
Published
2022-12-16
Volume
22
Issue
1
Pages
322-322
ISSN
1471-2288 (ISSN-L)
OpenAlex
W4312018506
Cited by
85
References
65
Citations by year
2026: 13 · 2025: 35 · 2024: 26 · 2023: 11
FWCI
6.08
Type
journal article
Language
en
Open access
Licence
Locations
  • BMC Medical Research Methodology (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • Dipòsit Digital de Documents de la UAB (Universitat Autònoma de Barcelona) (submittedVersion, cc-by)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
  • OSF Preprints (OSF Preprints) (submittedVersion)
Funders
World Health Organization
Topics
  • Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
  • Social Sciences › General Social Sciences › Computational and Text Analysis Methods
  • Social Sciences › Library and Information Sciences › scientometrics and bibliometrics research
Keywords
machine learning, evidence synthesis, software tools, evidence-based practice, open source software
MeSH terms
Machine Learning, Systematic Reviews as Topic, Algorithms, Humans, Software, Bibliometrics

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-10-04, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-04 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (qwen3.8-omni-flash and gpt-6-sol).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Machine learning computational tools to assist the performance of systematic reviews: A mapping review. https://app.cochat.ai/papers/10.1186/s12874-022-01805-4
The paper
Cierco Jimenez, R., Lee, T., Rosillo, N., Cordova, R., Cree, I. A., Gonzalez, A., & Indave Ruiz, B. I. (2022). Machine learning computational tools to assist the performance of systematic reviews: A mapping review. BMC Medical Research Methodology. https://doi.org/10.1186/s12874-022-01805-4

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.