Using artificial intelligence methods for systematic review in health sciences: A systematic review
Open accessAbstract
The exponential increase in published articles makes a thorough and expedient review of literature increasingly challenging. This review delineated automated tools and platforms that employ artificial intelligence (AI) approaches and evaluated the reported benefits and challenges in using such methods. A search was conducted in 4 databases (Medline, Embase, CDSR, and Epistemonikos) up to April 2021 for systematic reviews and other related reviews implementing AI methods. To be included, the review must use any form of AI method, including machine learning, deep learning, neural network, or any other applications used to enable the full or semi‐autonomous performance of one or more stages in the development of evidence synthesis. Twelve reviews were included, using nine different tools to implement 15 different AI methods. Eleven methods were used in the screening stages of the review (73%). The rest were divided: two in data extraction (13%) and two in risk of bias assessment (13%). The ambiguous benefits of the data extractions, combined with the reported advantages from 10 reviews, indicating that AI platforms have taken hold with varying success in evidence synthesis. However, the results are qualified by the reliance on the self‐reporting of the review authors. Extensive human validation still appears required at this stage in implementing AI methods, though further evaluation is required to define the overall contribution of such platforms in enhancing efficiency and quality in evidence synthesis.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Large language models, by leveraging their capabilities to capture contextual information and advanced semantic understanding, can more accurately identify relevant literature and enhance both the efficiency and accuracy of the screening process.
“LLMs, by leveraging their capabilities to capture contextual information and advanced semantic understanding [17], can more accurately identify relevant literature and enhance both the efficiency and accuracy of the screening process.”
From Artificial intelligence for the science of evidence synthesis: how good are AI-powered tools for automatic literature screening? (2025) · cites this paper as [17]
Split into 2 atoms — 0 confirmed, 2 partially supported, 0 disputed, 0 misattributed, 0 unverifiable. Expand the claim to see per-atom verdicts.
Experimental — accuracy not yet measuredUntil recently, most work in this field used natural language processing and similar text mining approaches.
“Until recently, most work in this field used natural language processing (NLP) and similar text mining approaches (Blaizot et al., 2022; Bolaños et al., 2024; Cowie et al., 2022; de la Torre-López et al., 2023; van Dinter et al., 2021).”
From Analysis of article screening and data extraction performance by an AI systematic literature review platform (2025) · cites this paper as Blaizot et al., 2022
The Introduction describes NLP and machine learning as approaches used in AI-assisted reviews, and the Study Findings identify NLP and text-mining tools among the methods examined. But the paper does not establish that most earlier work used NLP or similar text mining, or document a shift away from those approaches; it reports a range of methods across 12 reviews. (LLM, conf=0.82)
Experimental — accuracy not yet measuredExisting AI tools for systematic reviews have a major shortcoming: their performance significantly deteriorates without a human in the decision-making process.
“The major shortcoming of these tools is that their performance significantly deteriorates without looping in a human in the decision-making process, as shown by Blaizot and colleagues' (2022) meta-analysis.”
From Can large language models replace humans in systematic reviews? Evaluating <scp>GPT</scp> ‐4's efficacy in screening and extracting data from peer‐reviewed and grey literature in multiple languages (2024) · cites this paper as Blaizot et al., 2022
The Conclusion (included in the Study Limitations passage; page unavailable) says existing methods still need human judgment and that no platform appeared sufficiently accurate and reliable. The Discussion also reports successful screening without unplanned human intervention. The paper does not show a significant performance deterioration when humans are removed, and it describes itself as a systematic review, not a meta-analysis. (LLM, conf=0.88)
Experimental — accuracy not yet measuredArtificial intelligence has emerged as a potential solution to systematic review challenges, with recent studies suggesting its capability to enhance the quality and efficiency of systematic reviews.
“Artificial intelligence (AI) has emerged as a potential solution to these challenges, with recent studies and evaluations suggesting its capability to enhance the quality and efficiency of systematic reviews (Blaizot et al., 2022; Dijk et al., 2023; Kebede et al., 2023; Mahuli et al., 2023; Moreno-Garcia et al., 2023; Nugroho et al., 2023; Santos et al., 2023).”
From Can large language models replace humans in systematic reviews? Evaluating <scp>GPT</scp> ‐4's efficacy in screening and extracting data from peer‐reviewed and grey literature in multiple languages (2024) · cites this paper as Blaizot et al., 2022
The Introduction says AI methods show promise for reducing the labor required for reviews, supporting their potential to improve efficiency. The Study Limitations section says the review did not account for study quality and calls for further evaluation of AI's relationship with study quality; it does not establish an improvement in review quality. No page numbers are available in our copy. (LLM, conf=0.86)
Experimental — accuracy not yet measuredTime constraints and language barriers can further heighten the risk of data extraction errors.
“Time constraints and language barriers can further heighten the risk of data extraction errors,[5-7]”
From Artificial Intelligence–Assisted Data Extraction With a Large Language Model: A Study Within Reviews (2025) · cites this paper as [7]
The Study Limitations passages discuss failures of AI-assisted data extraction and the need for human intervention, but do not link extraction errors to time constraints or language barriers. A search for those factors found no passage making that connection; page numbers were not available in our copy. (LLM, conf=0.75)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Rayyan-a web and mobile app for systematic reviews.
Ouzzani, M., Hammady, H., Fedorowicz, Z., & Elmagarmid, A. (2016) · Systematic Reviews · doi:10.1186/s13643-016-0384-4 · Cited by 26,092 · see citing works
Shea, B. J., Reeves, B. C., Wells, G., Thuku, M., Hamel, C., Moran, J., Moher, D., Tugwell, P., Welch, V., Kristjansson, E., et al. (2017) · BMJ · doi:10.1136/bmj.j4008 · Cited by 10,636 · see citing works
Borah, R., Brown, A. W., Capers, P. L., & Kaiser, K. A. (2017) · BMJ Open · doi:10.1136/bmjopen-2016-012545 · Cited by 737 · cited 3× · see citing works
“of increasingly long systematic review publication time.(1) It must also be noted that Deng et al.”
BackgroundMethodInfluentialMarshall, I. J., Noel‐Storr, A., Kuiper, J., Thomas, J., & Wallace, B. C. (2018) · Research Synthesis Methods · doi:10.1002/jrsm.1287 · Cited by 435 · see citing works
Rathbone, J., Hoffmann, T., & Glasziou, P. (2015) · Systematic Reviews · doi:10.1186/s13643-015-0067-6 · Cited by 184 · cited 1× · see citing works
“found Abstrackr to reliably reduce workloads with very little risk of omitting records.(5,6) The 12 identified studies have implemented AI methods in the review and evidence synthesis process to varying degrees of success.”
BackgroundHoward, B. E., Phillips, J., Tandon, A., Maharana, A., Elmore, R., Mav, D., Sedykh, A., Thayer, K., Merrick, B. A., Walker, V., et al. (2020) · Environment International · doi:10.1016/j.envint.2020.105623 · Cited by 157 · cited 1× · see citing works
“SWIFT-review and SWIFT-Active Screener are a freely available interactive workbench that provides numerous tools to assist with problem formulation and literature prioritization.(23) Abstrackr is a freely available application that allows the reviewer to tag the records depending on the ML's relevance.”
MethodGates, A., Johnson, C., & Hartling, L. (2018) · Systematic Reviews · doi:10.1186/s13643-018-0707-8 · Cited by 148 · see citing works
Decoding semi-automated title-abstract screening: findings from a convenience sample of reviews
Gates, A., Gates, M., DaRosa, D., Elliott, S. A., Pillay, J., Rahman, S., Vandermeer, B., & Hartling, L. (2020) · Systematic Reviews · doi:10.1186/s13643-020-01528-x · Cited by 31 · see citing works
Flagged first, then by citations
Research works printed but not linked: 16
- Yaffe J, Montgomery P, Hopewell S, Shepard LD. Empty reviews: a description and consideration of Cochrane systematic reviews with no included studies. PloS One. 2012;7(5):e36626.
- Giummarra MJ, Lau G, Gabbe BJ. Evaluation of text mining to reduce screening workload for injury-focused systematic reviews. Inj Prev J Int Soc Child Adolesc Inj Prev. 2020 Feb;26(1):55-60.
- Viner R, Russell S, Saulle R, Croker H, Stansfeld C, Packer J, et al. Impacts of school closures on physical and mental health of children and young people: a systematic review. medRxiv. 2021 Feb 12;2021.02.10.21251526.
- Giummarra MJ, Lau G, Grant G, Gabbe BJ. A systematic review of the association between fault or blame-related attributions and procedures after transport injury and health and work-related outcomes. Accid Anal Prev. 2020 Feb 1;135:105333.
- Goldkuhle M, Dimaki M, Gartlehner G, Monsef I, Dahm P, Glossmann J, et al. Nivolumab for adults with Hodgkin's lymphoma (a rapid review using the software RobotReviewer). Cochrane Database Syst Rev. 2018 Jul 12;2018(7):CD012556.
- Pinna F, Manchia M, Paribello P, Carpiniello B. The Impact of Alexithymia on Treatment Response in Psychiatric Disorders: A Systematic Review. Front Psychiatry [Internet]. 2020 [cited 2021 Jul 18];0. Available from: https://www.frontiersin.org/articles/10.3389/fpsyt.2020.00311/full · DOI
- Gaskins NJ, Bray E, Hill JE, Doherty PJ, Harrison A, Connell LA. Factors influencing implementation of aerobic exercise after stroke: a systematic review. Disabil Rehabil. 2019 Dec 25;0(0):1-15.
- Rogers CR, Matthews P, Xu L, Boucher K, Riley C, Huntington M, et al. Interventions for increasing colorectal cancer screening uptake among African- American men: A systematic review and meta-analysis. PLOS ONE. 2020 Sep 16;15(9):e0238354.
- Siqueira TV, Nascimento J da SG, Oliveira JLG de, Regino D da SG, Dalri MCB. The use of serious games as an innovative educational strategy for learning cardiopulmonary resuscitation: an integrative review. Rev Gaucha Enferm. 2020;41:e20190293.
- Nascimento J da SG, Siqueira TV, Oliveira JLG de, Alves MG, Regino D da SG, Dalri MCB. Development of clinical competence in nursing in simulation: the perspective of Bloom's taxonomy. Rev Bras Enferm [Internet]. 2021 Mar 24 [cited 2021 Jul 18];74. Available from: http://www.scielo.br/j/reben/a/zgmY8gmZF3Q98JrxzLdCLrC/?lang=en
- Xiong Z, Liu T, Tse G, Gong M, Gladding PA, Smaill BH, et al. A Machine Learning Aided Systematic Review and Meta-Analysis of the Relative Risk of Atrial Fibrillation in Patients With Diabetes Mellitus. Front Physiol [Internet]. 2018 [cited 2021 Jul 18];0. Available from: https://www.frontiersin.org/articles/10.3389/fphys.2018.00835/full · DOI
- Lam J, Howard BE, Thayer K, Shah RR. Low-calorie sweeteners and health outcomes: A demonstration of rapid evidence mapping (rEM). Environ Int. 2019 Feb 1;123:451-8.
- Deng Z, Yin K, Bao Y, Armengol VD, Wang C, Tiwari A, et al. Validation of a Semiautomated Natural Language Processing-Based Procedure for Meta-Analysis of Cancer Susceptibility Gene Penetrance. JCO Clin Cancer Inform. 2019 Aug;3:1- 9.
- Aali G, Drummond A, das Nair R, Shokraneh F. Post-stroke fatigue: a scoping review. F1000Research. 2020 Aug 25;9:242.
- Park SE, Thomas J. Evidence synthesis software. BMJ Evid-Based Med. 2018 Aug;23(4):140-1.
- Crossingham I, Turner S, Ramakrishnan S, Fries A, Gowell M, Yasmin F, et al. Combination fixed-dose beta agonist and steroid inhaler as required for adults or children with mild asthma. Cochrane Database Syst Rev [Internet]. 2021 [cited 2021 Nov 24];(5). Available from: https://www.cochranelibrary.com/cdsr/doi/10.1002/14651858.CD013518.pub2/full · DOI
Cited by
Sorted by citations
Khraisha, Q., Put, S., Kappenberg, J., Warraitch, A., & Hadfield, K. (2023) · arXiv · doi:10.48550/arxiv.2310.17526 · Cited by 268 · see citing works
Khraisha, Q., Put, S., Kappenberg, J., Warraitch, A., & Hadfield, K. (2024) · Research Synthesis Methods · doi:10.1002/jrsm.1715 · Cited by 269 · see citing works
Automated paper screening for clinical reviews using large language models: Data analysis study
Guo, E., Gupta, M., Deng, J., Park, Y.-J., Paget, M., & Naugler, C. (2024) · Journal of Medical Internet Research · doi:10.2196/48996 · Cited by 205 · see citing works
Data extraction for evidence synthesis using a large language model: A proof-of-concept study
Gartlehner, G., Kahwati, L., Hilscher, R., Thomas, I., Kugley, S., Crotty, K., Viswanathan, M., Nussbaumer‐Streit, B., Booth, G., Erskine, N., et al. (2024) · Research Synthesis Methods · doi:10.1002/jrsm.1710 · Cited by 164 · see citing works
“8 The use of artificial intelligence (AI) can potentially increase efficiency of the data extraction process.”
How to optimize the systematic review process using AI tools
Fabiano, N., Gupta, A., Bhambra, N., Luu, B., Wong, S., Maaz, M., Fiedorowicz, J. G., Smith, A. L., & Solmi, M. (2024) · JCPP Advances · doi:10.1002/jcv2.12234 · Cited by 130 · see citing works
ChatGPT-4o can serve as the second rater for data extraction in systematic reviews
Motzfeldt Jensen, M., Brix Danielsen, M., Riis, J., Assifuah Kristjansen, K., Andersen, S., Okubo, Y., & Jørgensen, M. G. (2025) · PLOS ONE · doi:10.1371/journal.pone.0313401 · Cited by 60 · see citing works
Kebede, M. M., Le Cornet, C., & Fortner, R. T. (2023) · Research Synthesis Methods · doi:10.1002/jrsm.1589 · Cited by 50 · see citing works
Towards evidence-based practice 2.0: leveraging artificial intelligence in healthcare
Nilsen, P., Sundemo, D., Heintz, F., Neher, M., Nygren, J., Svedberg, P., & Petersson, L. (2024) · Frontiers in Health Services · doi:10.3389/frhs.2024.1368030 · Cited by 39 · see citing works
Mahmoudi, H., Chang, D., Lee, H., Ghaffarzadegan, N., & Jalali, M. S. (2025) · JMIR AI · doi:10.2196/68097 · Cited by 35 · see citing works
Artificial Intelligence–Assisted Data Extraction With a Large Language Model: A Study Within Reviews
Gartlehner, G., Kugley, S., Crotty, K., Viswanathan, M., Dobrescu, A., Nussbaumer-Streit, B., Booth, G., Treadwell, J. R., Han, J. M., Wagner, J., et al. (2025) · Annals of Internal Medicine · doi:10.7326/annals-25-00739 · Cited by 36 · see citing works
Bianchi, J., Hirt, J., Vogt, M., & Vetsch, J. (2025) · Cochrane Evidence Synthesis and Methods · doi:10.1002/cesm.70033 · Cited by 19 · see citing works
Do it faster with PICOS: Generative AI-Assisted systematic review screening
Vallamchetla, S. K., Abdelkader, O., Elnaggar, A., Ramadan, D., Islam Shourav, M. M., Riaz, I. B., & Lin, M. P. (2025) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2025.104860 · Cited by 13 · see citing works
Accelerating the pace and accuracy of systematic reviews using AI: a validation study
Zhan, J., Suvada, K., Xu, M., Tian, W., Cara, K. C., Wallace, T. C., & Ali, M. K. (2025) · Systematic Reviews · doi:10.1186/s13643-025-02997-8 · Cited by 12 · see citing works
Ruan, M., Fan, J., Liu, M., Meng, Z., Zhang, X., & Zhang, C. (2025) · BMC Medical Research Methodology · doi:10.1186/s12874-025-02644-9 · Cited by 12 · see citing works
Evaluating the Accuracy of scite, a Smart Citation Index
Bakker, C., Theis-Mahon, N., & Brown, S. J. (2023) · Hypothesis: Research Journal for Health Information Professionals · doi:10.18060/26528 · Cited by 10 · see citing works
Collaborative Large Language Models for Automated Data Extraction in Living Systematic Reviews
Khan, M. A., Ayub, U., Naqvi, S. A. A., Khakwani, K. Z. R., Sipra, Z. B. R., Raina, A., Zou, S., He, H., Hossein, S. A., Hasan, B., et al. (2024) · doi:10.1101/2024.09.20.24314108 · Cited by 54 · see citing works
Cassell, K., Ologunowa, A., Rastegar-Mojarad, M., Chun, B., Huang, Y.-L., Wang, D., & Cossrow, N. (2025) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2025.1662202 · Cited by 6 · see citing works
AI Voice Methodology: Using Generative AI in Qualitative Social Research
Hughes, J., Homan, L., O’Reilly, M., & Hughes, K. (2025) · Qualitative Inquiry · doi:10.1177/10778004251401842 · Cited by 2 · see citing works
- Authors
- Blaizot, A., Veettil, S. K., Saidoung, P., Moreno‐Garcia, C. F., Wiratunga, N., Aceves‐Martins, M., Lai, N. M., & Chaiyakunapruk, N.
- Affiliations
- University of Utah (US)
- Chiang Mai University (TH)
- School of Computing Robert Gordon University Aberdeen Scotland, UK
- University of Aberdeen (GB)
- Monash University Malaysia (MY)
- Corresponding author
- Lai, Nai Ming
- Chaiyakunapruk, Nathorn
- ORCID
- Venue
- Research Synthesis Methods
- Publisher
- Wiley
- Published
- 2022-02-28
- Volume
- 13
- Issue
- 3
- Pages
- 353-362
- ISSN
- 1759-2879 (ISSN-L)
- 1759-2887
- OpenAlex
- W4213127247
- Cited by
- 223
- References
- 8
- Citations by year
- 2026: 50 · 2025: 82 · 2024: 53 · 2023: 30 · 2022: 8
- FWCI
- 14.38
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- Research Synthesis Methods (publishedVersion)
- PubMed (publishedVersion)
- Aberdeen University Research Archive (Aberdeen University) (submittedVersion)
- Open Access Institutional Repository at Robert Gordon University (Robert Gordon University) (submittedVersion, cc-by)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Social Sciences › General Social Sciences › Computational and Text Analysis Methods
- Keywords
- artificial intelligence, evidence synthesis, literature screening, data extraction, risk of bias assessment, machine learning, deep learning, human validation, review efficiency
- MeSH terms
- Machine Learning, Systematic Reviews as Topic, Artificial Intelligence, Humans, Medicine
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
