Machine learning computational tools to assist the performance of systematic reviews: A mapping review.
Open accessThe DOI matches its Crossref record; 65 of the 65 references it cites were found in a registry or an index; 18 of the 68 sources its claims cite were checked for support.
Abstract
Background: Within evidence-based practice (EBP), systematic reviews (SR) are considered the highest level of evidence in that they summarize the best available research and describe the progress in a determined field. Due its methodology, SR require significant time and resources to be performed; they also require repetitive steps that may introduce biases and human errors. Machine learning (ML) algorithms therefore present a promising alternative and a potential game changer to speed up and automate the SR process. This review aims to map the current availability of computational tools that use ML techniques to assist in the performance of SR, and to support authors in the selection of the right software for the performance of evidence synthesis. Methods: The mapping review was based on comprehensive searches in electronic databases and software repositories to obtain relevant literature and records, followed by screening for eligibility based on titles, abstracts, and full text by two reviewers. The data extraction consisted of listing and extracting the name and basic characteristics of the included tools, for example a tool’s applicability to the various SR stages, pricing options, open-source availability, and type of software. These tools were classified and graphically represented to facilitate the description of our findings. Results: A total of 9653 studies and 585 records were obtained from the structured searches performed on selected bibliometric databases and software repositories respectively. After screening, a total of 119 descriptions from publications and records allowed us to identify 63 tools that assist the SR process using ML techniques. Conclusions: This review provides a high-quality map of currently available ML software to assist the performance of SR. ML algorithms are arguably one of the best techniques at present for the automation of SR. The most promising tools were easily accessible and included a high number of user-friendly features permitting the automation of SR and other kinds of evidence synthesis reviews.
Overview
CoChat Summary
TL;DR
This mapping review searched bibliographic databases and software repositories to identify machine-learning tools that assist systematic reviews, finding 63 tools from 119 included descriptions. Most tools target early SR stages, especially screening, and many are free/open-source, but a substantial minority require payment or programming skills.
Key findings
- From 9,653 database studies and 585 repository records, 119 descriptions yielded 63 ML tools; 52 (83%) were cited 1–5 times, while the top referenced tools were Abstrackr (23), EPPI-Reviewer (17), Rayyan (17), RobotReviewer (14), and RobotAnalyst (11).
- All 63 tools addressed early SR stages: search 30 (47%), screening 35 (55%), and data extraction 11 (17%); late SR stages had 1 tool count. By software type, 39 (62%) were web applications, 11 (17%) algorithms, 5 (8%) libraries/packages, and 8 (13%) desktop applications.
- NLP was used by 48 (76%) tools and supervised learning by 37 (58%); 22 (35%) required a programming background, while 41 (65%) could be used by less skilled users.
- 31 (49%) were open-source, 44 (70%) freely available, and 14 (22%) required payment; 62 (98%) were updated or released in the last 5 years and 29 (46%) in the last year. Fee-based tools were updated in the last year more often (86%) than free tools (30%).
Methods
- Mapping review with a protocol registered on OSF; searches were run in MEDLINE (PubMed), EMBASE, Web of Science, and repositories CPAN, CRAN, GitHub, NaCTeM, PECL, PyPI, SourceForge, and the SR toolbox from 1 April to 31 May 2021, with no language restrictions.
- Eligibility included publications or repository records describing ML software to assist any SR stage, with sufficient technical description, in English, Spanish, German, or French, and developed or updated within the last 10 years; reference management software and inaccessible tools were excluded. Two reviewers screened titles/abstracts and full texts; repository records were screened by one reviewer and checked by a second.
- Data extraction captured computational method, SR stage, release/update dates, licensing and pricing, open-source status, access source, and programming background; findings were compiled in a summary-of-findings table and a mapping infographic.
Limitations
- Stated: the mapping review may have bias inherent to its methodology; repository search engines lacked advanced search functionalities, so some ML tool registries may not have been detected; new tools developed after the project, such as Elicit, were not considered; and the methodology does not synthesize findings or critically appraise the methodological quality of retrieved publications.
- Stated: usability and acceptability could not be assessed because such features were poorly reported.
- Evident: searches ended 31 May 2021, so tools released or updated after that date are absent; reliance on repository records and brief descriptions may underrepresent tools without sufficient technical description or accessible links.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures



Verification
Claims the paper makes
8 Valid · 3 Not valid · 17 Not checked · 95 Own results and interpretations
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Around 160 tools assist the reviewer in either one specific step (during record search [43], screening [39, 44, 45], data extraction [45], risk of bias assessment/ critical appraisal, etc.), or guide the user through several steps or the whole SR process [46–48].”
§ Machine learning computational tools to assist the performance of systematic reviews: A mapping review > Background
What the sources say
- Not checked yetcites [42] Marshall C, Brereton P. Systematic review toolbox: a catalogue of tools to support systematic reviews. Proceedings of the 19th International Conference on Evaluation and Assessment in Software Engineering. 2015. source
- Not checked yetcites [43] Russell-Rose T, Shokraneh F. 63 2Dsearch: facilitating reproducible and valid searching in evidence synthesis. BMJ Evidence-Based Medicine. 2019;24(Suppl 1):A36.
- Partly supportedcites [46] Hamel C, Kelly SE, Thavorn K, Rice DB, Wells GA, Hutton B. An evaluation of DistillerSR’s machine learning-based prioritization tool for title/abstract screening – impact on reviewer-relevant outcomes. BMC Med Res Methodol. 2020;20(1):256. sourceExperimental — accuracy not yet measured
Hamel et al. (2020) Background supports only the generic half of the claim: 'Several software tools exist that support title and abstract screening in SRs [18]' and it names six commonly evaluated packages (Abstrackr, DistillerSR, EPPI-Reviewer, RobotAnalyst, SWIFT-Active Screener, SWIFT-Review), i.e. tools assisting one specific step. Nothing in our copy reports a count of 'around 160' tools, nor tools spanning record search, data extraction, risk of bias/critical appraisal, or guiding the user through several steps or the whole SR process (searched: '160 software tools', 'tools assist the reviewer...whole systematic review process', 'data extraction risk of bias catalogue'). The paper's own tool discussion is confined to screening and credits the tool inventory to other works it cites ([18] Harrison et al.; [19] Marshall & Wallace), so the quantitative core of the claim is unsupported by this source. (LLM, conf=0.65)
- Not checked yetcites [47] Adams CE, Polzmacher S, Wolff A. Systematic reviews: work that needs to be done and not to be done. J Evid Based Med. 2013;6(4):232–5. source
- Partly supportedcites [48] Kohl C, McIntosh EJ, Unger S, Haddaway NR, Kecke S, Schiemann J, et al. Online tools supporting the conduct and reporting of systematic reviews and systematic maps: a case study on CADIMA and review of existing tools. Environmental Evidence. 2018;7(1):8. sourceExperimental — accuracy not yet measured
Our copy of Kohl et al. (2018) reports 22 software packages, not ~160: the 'Review of existing tools' section (p. 12) states 'Of the 22 software packages identified as being suitable to support the systematic review or systematic map process...', and the Discussion (p. 14) repeats 'We have identified 22 software packages'. The paper also explicitly excluded single-step tools ('software that only supported isolated aspects of, rather than the majority of, the systematic review process', p. 2), referring readers to the SR toolbox for other tools, so the 'one specific step' half of the claim is not its finding either. What the source does support is the qualitative part: packages vary in which stages they cover (screening, data extraction, critical appraisal; Table 3, pp. 12-14) and 'A handful of tools are particularly designed to lead the user in a stepwise manner through the review process' while 'No single software package guides the reviewer through all stages' (p. 14). The magnitude attributed (~160 tools) is therefore not in this source and overstates it. (LLM, conf=0.75)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“As mentioned above, these algorithms/software are more and more used for article screening in systematic reviews [53, 55, 56]”
§ Machine learning computational tools to assist the performance of systematic reviews: A mapping review > Background
What the sources say
- Partly supportedcites [53] O’Mara-Eves A, Thomas J, McNaught J, Miwa M, Ananiadou S. Using text mining for study identification in systematic reviews: a systematic review of current approaches. Syst Rev. 2015;4(1):5. sourceExperimental — accuracy not yet measured
The Results section 'Chronological developments' (p. n/a) documents an active, growing research base — 44 evaluations between 2006 and 2014, with 'almost every year saw the evaluation of a newly applied type of classifier', suggesting 'a rapidly evolving evidence base' — and the Abstract Background frames text mining as 'offered as a potential solution' for automating screening, so the source does support rising attention to and evaluation of ML algorithms/software for study screening. However, the part of the claim about actual increasing USE is not supported by our copy: 'Deployed systems' reports that 'Only six different systems (reported in 12 papers) are currently "deployed"', and the Discussion ('Summary of key findings', 'The research problem') states this 'limits the ability of reviewers to try out these technologies', that specialist advice is probably needed whenever they are used in a live review, and that broad uptake would require tools accessible without bespoke code. So the cited paper evidences a rapidly growing evaluation/technology base, not demonstrated uptake in routine screening practice. (LLM, conf=0.72)
- Not checked yetcites [55] Olorisade BK, Quincey Ed, Brereton P, Andras P. A critical analysis of studies that address the use of text mining for citation screening in systematic reviews. Proceedings of the 20th International Conference on Evaluation and Assessment in Software Engineering; Limerick, Ireland: Association for Computing Machinery; 2016. p. Article 14. source
- Supportedcites [56] van de Schoot R, de Bruin J, Schram R, Zahedi P, de Boer J, Weijdema F, et al. An open source machine learning framework for efficient and transparent systematic reviews. Nat Mach Intell. 2021;3(2):125–33. source
The Introduction ('W' section) states that 'the rapidly evolving field of machine learning has aided researchers by allowing the development of software tools that assist in developing systematic reviews' and that ML is applied to 'overcome the manual and time-consuming screening of large numbers of studies' via active learning — a direct fit for the claim's background point. The Abstract adds that 'the future of systematic reviewing will be an interaction with machine learning algorithms', and Table 1 catalogues existing RiTL-based active learning screening tools (abstrackr, Rayyan, RobotAnalyst, ASReview) while the Conclusion reports 'a wide range of real-world systematic reviewing applications', so our copy clearly treats ML screening software as an expanding, in-use practice rather than a proposal. The paper frames the growth mainly as tool development and anticipated uptake rather than measuring an increase in usage, but this is a broad background citation and no qualifier of the claim is left unsupported. (LLM, conf=0.84)
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
The sources with a verdict support this claim; the rest weren’t checked yet, have no full text we could read, or their verdicts aren’t rated yet.
In the paper
“Use of natural language processing (NLP) and machine learning (ML) algorithms to reduce time and workload in the SR process is becoming increasingly popular [29, 40, 41].”
§ Machine learning computational tools to assist the performance of systematic reviews: A mapping review > Background
What the sources say
- Supportedcites [29] Marshall IJ, Wallace BC. Toward systematic review automation: a practical guide to using machine learning tools in research synthesis. Syst Rev. 2019;8(1):163. source
The Background section states that 'Research on methods for semi-automating systematic reviews via machine learning and natural language processing now constitutes its own (small) subfield, with an accompanying body of work,' and calls it a 'burgeoning field' for which 'multiple systematic reviews of such efforts already exist' — i.e. growing activity in this area. The Abstract frames the same point as technologies 'to speed up the production of systematic reviews by reducing the manual labour involved' having 'recently emerged,' with automation 'proposed or used to expedite most steps of the systematic review process.' As a broad background statement the claim is consistent with how our copy treats the topic; the only qualification is that the Conclusions temper actual uptake (prototype tools, rough edges, and commercial disinterest absent 'substantial demand from users'), so 'popular' here reflects research/interest growth more than widespread deployment. (LLM, conf=0.82)
- Could not verifycites [40] Kreimeyer K, Foster M, Pandey A, Arya N, Halford G, Jones SF, et al. Natural language processing systems for capturing and standardizing unstructured clinical information: a systematic review. J Biomed Inform. 2017;73:14–29. sourceExperimental — accuracy not yet measured
Our copy of Kreimeyer et al. (2017) is a systematic review OF clinical NLP systems, and no passage addresses using NLP/ML to cut time or workload IN the systematic-review process. The nearest statements are about other things: the Introduction notes NLP has 'reduced the time required for screening candidates for clinical trial eligibility' (clinical-trial eligibility, not review screening), and the Discussion says 'Machine learning approaches have been rarer overall, but may grow in popularity' (about clinical NLP systems generally). The Methods in fact describe manual two-reviewer screening of 7149 records in EPPI-Reviewer 4, and the Discussion only observes that 'literature-based reviews like this one may have difficulty in keeping up with new developments'. Searches for automated literature screening / text mining to reduce review workload returned no supporting passage. (LLM, conf=0.72)
- Not checked yetcites [41] Jaspers S, De Troyer E, Aerts M. Machine learning techniques for the automation of literature reviews and systematic reviews in EFSA. EFSA Supporting Publications. 2018;15(6):1427E. source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“However, developing automatic screening tools based on combined ML techniques seems feasible; plenty of research and developments have been done in this field in recent years [26, 39, 41, 46, 52].”
§ Machine learning computational tools to assist the performance of systematic reviews: A mapping review > Background
What the sources say
- Partly supportedcites [26] Beller E, Clark J, Tsafnat G, Adams C, Diehl H, Lund H, et al. Making progress with the automation of systematic reviews: principles of the International Collaboration for the Automation of Systematic Reviews (ICASR). Syst Rev. 2018;7(1):77. sourceExperimental — accuracy not yet measured
The Background says title and abstract screening is amenable to automation and that machine-learning advances have produced algorithms for systematic-review tasks. The Vienna Principles advocate combining automation components, but neither passage establishes the feasibility of screening tools based on combined machine-learning techniques specifically. No page numbers are supplied in our copy. (LLM, conf=0.83)
- Partly supportedcites [39] Gates A, Johnson C, Hartling L. Technology-assisted title and abstract screening for systematic reviews: a retrospective evaluation of the Abstrackr machine learning tool. Syst Rev. 2018;7(1):45. sourceExperimental — accuracy not yet measured
The Background describes machine-learning tools that can semi-automate screening and says Abstrackr has shown potential. The Methods > Abstrackr section describes a tool that learns from human screening decisions, but does not establish the feasibility of combining multiple ML techniques or of fully automatic screening. No page numbers are available in our copy. (LLM, conf=0.84)
- Not checked yetcites [41] Jaspers S, De Troyer E, Aerts M. Machine learning techniques for the automation of literature reviews and systematic reviews in EFSA. EFSA Supporting Publications. 2018;15(6):1427E. source
- Partly supportedcites [46] Hamel C, Kelly SE, Thavorn K, Rice DB, Wells GA, Hutton B. An evaluation of DistillerSR’s machine learning-based prioritization tool for title/abstract screening – impact on reviewer-relevant outcomes. BMC Med Res Methodol. 2020;20(1):256. sourceExperimental — accuracy not yet measured
The Background describes ML-assisted title and abstract screening as a viable way to reduce screening burden, and the Conclusion reports that DistillerSR’s prioritization tool can do so. The paper evaluates one prioritization tool, however; it does not establish the feasibility of tools based on combined ML techniques or of fully automatic screening. Page numbers are not available in our copy. (LLM, conf=0.84)
- Partly supportedcites [52] Bannach-Brown A, Przybyła P, Thomas J, Rice ASC, Ananiadou S, Liao J, et al. Machine learning algorithms for systematic review: reducing workload in a preclinical review of animal studies and reducing human screening error. Syst Rev. 2019;8(1):23. sourceExperimental — accuracy not yet measured
The Methods section reports testing feature sets with several classifiers, and the Conclusions describe ML-assisted screening of preclinical animal studies as promising. However, the Methods explicitly say the two approaches were developed independently; the paper does not establish the feasibility of a screening tool that combines those approaches. (LLM, conf=0.84)
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Other projects as SR-Accelerator [28], integrate several tools in a suite to assist in more than one step of the SR procedure”
§ Machine learning computational tools to assist the performance of systematic reviews: A mapping review > Discussion
What the sources say
- Partly supportedcites [28] Clark J, Glasziou P, Del Mar C, Bannach-Brown A, Stehlik P, Scott AM. A full systematic review was completed in 2 weeks using automation tools: a case study. J Clin Epidemiol. 2020;121:81–90. sourceExperimental — accuracy not yet measured
Our copy does show the SRA/CREBP-SRA suite as a set of several tools covering more than one SR step: Table 2 maps SR tasks 1-19 to tools and Table 4 (Time to complete individual SR tasks) lists SARA (obtain full texts), SRA-RevMan Replicant (write up SR), Polyglot Search Translator (run search), SRA Helper (screen abstracts/obtain full texts) and the De-duplicator, several flagged 'available within CREBP-SRA' at crebp-sra.com. What the paper does not state is that these tools are integrated with one another: in Facilitators and Barriers it records 'lack of integration between various software' as a current barrier and proposes an 'SRA tool integrator' plus 'enhanced integration between tools that already exist' only as future development (Tables 5/6). So the multi-step suite is supported, the integration claim is not. (LLM, conf=0.68)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
50 more claims
Create a free account to see all 123 claims and to verify the 50 references still open.
- It may state two separate claims as one. 36
- Probably not a substantial claim. 42
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Cierco Jimenez et al. 8 included 63 references in a mapping review of machine-learning to assist SRs during different workflow steps, of which 41 were available end-user tools for use by researchers without informatics background.”
From Data extraction methods for systematic review (semi)automation: Update of a living systematic review (2025) · cites this paper as [8]
Experimental — accuracy not yet measuredIn the citing paper
“In accordance with other reviews they describe screening as the most frequently automated step, while automated data extraction tools are lacking due to the complexity of the task.”
From Data extraction methods for systematic review (semi)automation: Update of a living systematic review (2025) · cites this paper as [8]
In the citing paper
“These are, to the best of our knowledge, the only end-user tools with automation for this task [10].”
From A narrative review of recent tools and innovations toward automating living systematic reviews and evidence syntheses (2023) · cites this paper as [10]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
“from the 44 freely available tools, 1 (15%) remained out to date”
§ Machine learning computational tools to assist the performance of systematic reviews: A mapping review > Results
Reported 15 · recomputed 2.272727
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
“within the 14 fee-based tools, only 1 (13%) were outdated”
§ Machine learning computational tools to assist the performance of systematic reviews: A mapping review > Results
Reported 13 · recomputed 7.142857
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
“from the 14 fee-based tools, 10 (72%) were not open-source, one (7%) was open-source and three (7%) were not described.”
§ Machine learning computational tools to assist the performance of systematic reviews: A mapping review > Results
Reported 7 · recomputed 21.428571
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Machine learning computational tools to assist the performance of systematic reviews: A mapping review > Results
“Algorithms and packages/libraries had the highest proportion of freely available tools with 5 (100%) and 11 (100%) respectively”
§ Machine learning computational tools to assist the performance of systematic reviews: A mapping review > Results
“Thirty-nine (62%) of the included tools were web applications, 11 (17%) were algorithms, 5 (8%) were libraries or packages and 8 (13%) were desktop applications.”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Machine learning computational tools to assist the performance of systematic reviews: A mapping review > Results
“only 1 (2%) case was considered nearly out of date due to older updates”
§ Machine learning computational tools to assist the performance of systematic reviews: A mapping review > Results
“within the 14 fee-based tools, only 1 (13%) were outdated, while 13 (92%) have been updated over the last five years and 13 (86%) in the last year. In contrast, from the 44 freely available tools, 1 (15%) remained out to date”
2 more findings
Create a free account to see every finding of this check.
Citations
Cites
Evidence based medicine: what it is and what it isn't
Sackett, D. L., Rosenberg, W. M. C., Gray, J. A. M., Haynes, R. B., & Richardson, W. S. (1996) · BMJ · doi:10.1136/bmj.312.7023.71 · Cited by 15,769 · cited 1×
“Evidence-based medicine (EBM) developed these principles to identify and evaluate medical information and provide structured summaries of the available evidence to inform decision in health care and improve the diagnosis and treatment of patients [1, 3, 8].”
BackgroundA typology of reviews: an analysis of 14 review types and associated methodologies
Grant, M. J., & Booth, A. (2009) · Health Information & Libraries Journal · doi:10.1111/j.1471-1842.2009.00848.x · Cited by 10,817 · cited 1×
“Considering these advances, the rapid evolution of the area, and the difficulties in identifying the best suited tool for each task and field, we aimed with this project to systematically map available ML tools that assist the SR process [61].”
MethodThe 2019 WHO classification of tumours of the digestive system
Nagtegaal, I. D., Odze, R. D., Klimstra, D., Paradis, V., Rugge, M., Schirmacher, P., Washington, K. M., Carneiro, F., Cree, I. A., & the WHO Classification of Tumours Editorial Board (2020) · Histopathology · doi:10.1111/his.13975 · Cited by 4,192 · cited 2×
“As the WHO Classification on Tumours Programme (WCT) [34, 35] we wish to promote evidence-based practice in pathology.”
BackgroundSelf-citationDeo, R. C. (2015) · Circulation · doi:10.1161/circulationaha.115.001593 · Cited by 3,466 · cited 2×
“Especially promising are ML techniques [49–51] for the automation of systematic reviews steps [41, 46, 52].”
MethodAssessing Risk of Bias in Included Studies
Higgins, J. P., & Altman, D. G. (2008) · Cochrane Handbook for Systematic Reviews of Interventions · doi:10.1002/9780470712184.ch8 · Cited by 3,396 · cited 1×
“However being a mapping exercise, risk of bias as that of selective reporting [77], could be minimized by applying few exclusion criteria and reporting on all identified tools for which we could retrieve sufficient information.”
BackgroundSystematic Reviews: Synthesis of Best Evidence for Clinical Decisions
Cook, D. J., Mulrow, C. D., & Haynes, R. B. (1997) · Annals of Internal Medicine · doi:10.7326/0003-4819-126-5-199703010-00006 · Cited by 2,184 · cited 1×
“SR and meta-analyses are recognized as the highest level of evidence [3, 9, 11, 13, 17], and the growing number of available tools to assist during the performance of SR probably reflects an increasing recognition of the utility of this type of studies (see Fig.”
BackgroundSystematic Reviews: Rationale for systematic reviews
Mulrow, C. D. (1994) · BMJ · doi:10.1136/bmj.309.6954.597 · Cited by 1,677
The Mass Production of Redundant, Misleading, and Conflicted Systematic Reviews and Meta‐analyses
IOANNIDIS, J. P. A. (2016) · The Milbank Quarterly · doi:10.1111/1468-0009.12210 · Cited by 1,483 · cited 1×
“The number of published systematic reviews has increased exponentially in recent years [18, 19].”
Background'Scoping the scope' of a cochrane review
Armstrong, R., Hall, B. J., Doyle, J., & Waters, E. (2011) · Journal of Public Health · doi:10.1093/pubmed/fdr015 · Cited by 1,260 · cited 1×
“A SR aims to select, identify, critically appraise, and synthesise the best available evidence within pre-specified eligibility criteria to answer a clearly defined research question [9, 12-14].”
MethodSeventy-five trials and eleven systematic reviews a day: how will we ever keep up?
Bastian, H., Glasziou, P., & Chalmers, I. (2010) · PLoS Medicine · doi:10.1371/journal.pmed.1000326 · Cited by 1,188 · cited 1×
“The number of published systematic reviews has increased exponentially in recent years [18, 19].”
Background
Research works printed but not linked: 2
- Boudin F, Nie J-Y, Dawes M. Clinical information retrieval using document and PICO structure. Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics; Los Angeles, California: Association for Computational Linguistics; 2010. p. 822–30.
- The Systematic Review Toolbox (SR toolbox). http://systematicreviewtools.com/. Accessed 3 May 2021.
Cited by
Artificial intelligence for literature reviews: opportunities and challenges
Bolaños, F., Salatino, A., Osborne, F., & Motta, E. (2024) · Artificial Intelligence Review · doi:10.1007/s10462-024-10902-3 · Cited by 246
Lieberum, J.-L., Toews, M., Metzendorf, M.-I., Heilmeyer, F., Siemens, W., Haverkamp, C., Böhringer, D., Meerpohl, J. J., & Eisele-Metzger, A. (2025) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2025.111746 · Cited by 148
Boetje, J., & van de Schoot, R. (2024) · Systematic Reviews · doi:10.1186/s13643-024-02502-7 · Cited by 146
Data extraction methods for systematic review (semi)automation: a living systematic review
Schmidt, L., Olorisade, B. K., McGuinness, L. A., Thomas, J., & Higgins, J. P. T. (2021) · F1000Research · doi:10.12688/f1000research.51117.1 · Cited by 106
Hanegraaf, P., Wondimu, A., Mosselman, J. J., de Jong, R., Abògúnr̀in, Ṣ., Queiros, L., Lane, M., Postma, M. J., Boersma, C., & van der Schans, J. (2024) · BMJ Open · doi:10.1136/bmjopen-2023-076912 · Cited by 69
Metadata
- Authors
- Cierco Jimenez, R., Lee, T., Rosillo, N., Cordova, R., Cree, I. A., Gonzalez, A., & Indave Ruiz, B. I.
- Affiliations
- Centre international de recherche sur le cancer (FR)
- Hospital Universitario 12 De Octubre (ES)
- Venue
- BMC Medical Research Methodology
- Published
- 2022-12-16
- Volume
- 22
- Issue
- 1
- Pages
- 322-322
- ISSN
- 1471-2288 (ISSN-L)
- PMC
- PMC9756658
- OpenAlex
- W4312018506
- Cited by
- 85
- References
- 65
- Citations by year
- 2026: 13 · 2025: 35 · 2024: 26 · 2023: 11
- FWCI
- 6.08
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- creativecommons.org/licenses/by/4.0 (tdm)
- creativecommons.org/licenses/by/4.0 (vor)
- Locations
- BMC Medical Research Methodology (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Dipòsit Digital de Documents de la UAB (Universitat Autònoma de Barcelona) (submittedVersion, cc-by)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- OSF Preprints (OSF Preprints) (submittedVersion)
- Funders
- World Health Organization
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › General Social Sciences › Computational and Text Analysis Methods
- Social Sciences › Library and Information Sciences › scientometrics and bibliometrics research
- Keywords
- machine learning, evidence synthesis, software tools, evidence-based practice, open source software
- MeSH terms
- Machine Learning, Systematic Reviews as Topic, Algorithms, Humans, Software, Bibliometrics
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-04, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-04 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (qwen3.8-omni-flash and gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Machine learning computational tools to assist the performance of systematic reviews: A mapping review. https://app.cochat.ai/papers/10.1186/s12874-022-01805-4
- The paper
- Cierco Jimenez, R., Lee, T., Rosillo, N., Cordova, R., Cree, I. A., Gonzalez, A., & Indave Ruiz, B. I. (2022). Machine learning computational tools to assist the performance of systematic reviews: A mapping review. BMC Medical Research Methodology. https://doi.org/10.1186/s12874-022-01805-4
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.