Towards the automation of systematic reviews using natural language processing, machine learning, and deep learning: a comprehensive review
Open accessThe DOI matches its Crossref record; 51 of the 51 references it cites were found in a registry or an index.
Abstract
Systematic reviews (SRs) constitute a critical foundation for evidence-based decision-making and policy formulation across various disciplines, particularly in healthcare and beyond. However, the inherently rigorous and structured nature of the SR process renders it laborious for human reviewers. Moreover, the exponential growth in daily published literature exacerbates the challenge, as SRs risk missing out on incorporating recent studies that could potentially influence research outcomes. This pressing need to streamline and enhance the efficiency of SRs has prompted significant interest in leveraging Artificial Intelligence (AI) techniques to automate various stages of the SR process. This review paper provides a comprehensive overview of the current AI methods employed for SR automation, a subject area that has not been exhaustively covered in previous literature. Through an extensive analysis of 52 related works and an original online survey, the primary AI techniques and their applications in automating key SR stages, such as search, screening, data extraction, and risk of bias assessment, are identified. The survey results offer practical insights into the current practices, experiences, opinions, and expectations of SR practitioners and researchers regarding future SR automation. Synthesis of the literature review and survey findings highlights gaps and challenges in the current landscape of SR automation using AI techniques. Based on these insights, potential future directions are discussed. This review aims to equip researchers and practitioners with a foundational understanding of the basic concepts, primary methodologies, and recent advancements in AI-driven SR automation while guiding computer scientists in exploring novel techniques to invigorate further and advance this field.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318
Cortes, C., & Vapnik, V. (1995) · Machine Learning · doi:10.1007/bf00994018 · Cited by 59,421
Gradient-based learning applied to document recognition
Lecun, Y., Bottou, L., Bengio, Y., & Haffner, P. (1998) · Proceedings of the IEEE · doi:10.1109/5.726791 · Cited by 58,805
Rayyan-a web and mobile app for systematic reviews.
Ouzzani, M., Hammady, H., Fedorowicz, Z., & Elmagarmid, A. (2016) · Systematic Reviews · doi:10.1186/s13643-016-0384-4 · Cited by 26,092
Literature review as a research methodology: An overview and guidelines
Snyder, H. (2019) · Journal of Business Research · doi:10.1016/j.jbusres.2019.07.039 · Cited by 11,524
Reinforcement Learning: A Survey
Kaelbling, L. P., Littman, M. L., & Moore, A. W. (1996) · Journal of Artificial Intelligence Research · doi:10.1613/jair.301 · Cited by 8,972
Machine learning: Algorithms, real-world applications and research directions
Sarker, I. H. (2021) · SN Computer Science · doi:10.1007/s42979-021-00592-x · Cited by 5,583
Systematic Literature Reviews in Software Engineering—A Systematic Literature Review
Kitchenham, B., Pearl Brereton, O., Budgen, D., Turner, M., Bailey, J., & Linkman, S. (2009) · Information and Software Technology · doi:10.1016/j.infsof.2008.09.009 · Cited by 4,537
Moher, D. (2001) · JAMA · doi:10.1001/jama.285.15.1987 · Cited by 2,545
The k-means Algorithm: A Comprehensive Survey and Performance Evaluation
Ahmed, M., Seraj, R., & Islam, S. M. S. (2020) · Electronics · doi:10.3390/electronics9081295 · Cited by 1,717
Research works printed but not linked: 68
- Abramovich F, Grinshtein V, Levy T (2021) Multiclass classification by sparse multinomial logistic regression. IEEE Trans Inf Theory 67(7):4637–4646. https://doi.org/10.1109/tit.2021.3075137 · DOI
- Aceves-Martins M, López-Cruz L, García-Botello M et al (2021) Interventions to prevent obesity in Mexican children and adolescents: systematic review. Prev Sci 23(4):563–586. https://doi.org/10.1007/s11121-021-01316-6 · DOI
- AHO AV (1990) Algorithms for finding patterns in strings. Elsevier, Amsterdam, pp 255–300. https://doi.org/10.1016/b978-0-444-88071-0.50010-2 · DOI · Paper page
- Aklouche B, Bounhas I, Slimani Y (2018) Query expansion based on NLP and word embeddings. In: Text retrieval conference. https://api.semanticscholar.org/CorpusID:155085448 · doi:10.6028/nist.sp.500-331.core-jarir
- Aklouche B, Bounhas I, Slimani Y (2019) Automatic query reweighting using co-occurrence graphs. In: Proceedings of the 16th international conference on applied computing 2019. IADIS Press, AC 2019. https://doi.org/10.33965/ac2019_201912l005 · doi:10.33965/ac2019_201912l005
- Alaofi M, Gallagher L, Sanderson M et al (2023) Can generative LLMS create query variants for test collections? An exploratory study. In: Proceedings of the 46th international ACM SIGIR conference on research and development in information retrieval. ACM, SIGIR ’23. https://doi.org/10.1145/3539618.3591960 · DOI
- Albawi S, Mohammed TA, Al-Zawi S (2017) Understanding of a convolutional neural network. In: 2017 international conference on engineering and technology (ICET). pp 1–6. https://doi.org/10.1109/ICEngTechnol.2017.8308186 · DOI · Paper page
- Almeida H, Meurs MJ, Kosseim L et al (2016) Data sampling and supervised learning for HIV literature screening. IEEE Trans Nanobiosci 15(4):354–361. https://doi.org/10.1109/bibm.2015.7359733 · DOI
- Angluin D (1988) Queries and concept learning. Mach Learn 2:319–342 (https://api.semanticscholar.org/CorpusID:11357867) · DOI · Paper page
- Aromataris E, Pearson A (2014) The systematic review: an overview. Am J Nurs 114(3):53–58. https://doi.org/10.1097/01.NAJ.0000444496.24228.2c · DOI · Paper page
- August ST (2001) Active learning: theory and applications. Stanford University 13(4):182
- Baranwal A, Bagwe BR, Vanitha M (2022) Machine learning in Python: diabetes prediction using machine learning. IGI Global, pp 882–908. https://doi.org/10.4018/978-1-6684-6291-1.ch046 · DOI
- Blaizot A, Veettil SK, Saidoung P et al (2022) Using artificial intelligence methods for systematic review in health sciences: a systematic review. Res Synth Methods 13(3):353–362. https://doi.org/10.1002/jrsm.1553 · DOI · Paper page
- Blei DM, Ng AY, Jordan MI (2003) Latent Dirichlet allocation. J Mach Learn Res 3(null):993–1022
- Booth A, Sutton A, Papaioannou D (2016) Systematic approaches to a successful literature review, 2nd edn. Sage, Thousand Oaks
- Bui DDA, Jonnalagadda S, Del Fiol G (2015) Automatically finding relevant citations for clinical guideline development. J Biomed Inform 57:436–445. https://doi.org/10.1016/j.jbi.2015.09.003 · DOI
- Cawley M, Beardslee R, Beverly B et al (2020) Novel text analytics approach to identify relevant literature for human health risk assessments: a pilot study with health effects of in utero exposures. Environ Int 134:105228. https://doi.org/10.1016/j.envint.2019.105228 · DOI
- Cessie SL, Houwelingen JCV (1992) Ridge estimators in logistic regression. Appl Stat 41(1):191. https://doi.org/10.2307/2347628 · DOI · Paper page
- Chiu B, Crichton G, Korhonen A et al (2016) How to train good word embeddings for biomedical NLP. In: Proceedings of the 15th workshop on biomedical natural language processing. Association for Computational Linguistics. https://doi.org/10.18653/v1/w16-2922 · DOI · Paper page
- Cho K, van Merrienboer B, Gulcehre C et al (2014) Learning phrase representations using RNN encoder-decoder for statistical machine translation. In: Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP). Association for Computational Linguistics. https://doi.org/10.3115/v1/d14-1179 · DOI · Paper page
- Cohn D, Atlas L, Ladner R (1994) Improving generalization with active learning. Mach Learn 15(2):201–221. https://doi.org/10.1007/bf00993277 · DOI · Paper page
- Devlin J, Chang MW, Lee K et al (2019) BERT: pre-training of deep bidirectional transformers for language understanding. http://arxiv.org/abs/1810.04805 · arXiv
- Frunza O, Inkpen D, Matwin S et al (2011) Exploiting the systematic review protocol for classification of medical abstracts. Artif Intell Med 51(1):17–25. https://doi.org/10.1016/j.artmed.2010.10.005 · DOI
- Gonzalez-Toral S, Freire R, Gualan R et al (2019) A ranking-based approach for supporting the initial selection of primary studies in a systematic literature review. In: 2019 XLV Latin American computing conference (CLEI). IEEE. https://doi.org/10.1109/clei47609.2019.235079 · DOI
- Gosavi A (2009) Reinforcement learning: a tutorial survey and recent advances. INFORMS J Comput 21(2):178–192. https://doi.org/10.1287/ijoc.1080.0305 · DOI · Paper page
- Gulo CA, Rúbio TR, Tabassum S et al (2015) Mining scientific articles powered by machine learning techniques. In: 2015 Imperial College computing student workshop (ICCSW 2015). Schloss Dagstuhl-Leibniz-Zentrum fuer Informatik. https://doi.org/10.4230/OASIcs.ICCSW.2015.21 · DOI
- Guo G, Wang H, Bell D et al (2003) KNN model-based approach in classification. Springer, Berlin/Heidelberg, pp 986–996. https://doi.org/10.1007/978-3-540-39964-3_62 · DOI · Paper page
- Hans C (2011) Elastic net regression modeling with the orthant normal prior. J Am Stat Assoc 106(496):1383–1393. https://doi.org/10.1198/jasa.2011.tm09241 · DOI · Paper page
- Hashimoto K, Kontonatsios G, Miwa M et al (2016) Topic detection using paragraph vectors to support active learning in systematic reviews. J Biomed Inform 62:59–65. https://doi.org/10.1016/j.jbi.2016.06.001 · DOI · Paper page
- Hasny M, Vasile AP, Gianni M et al (2023) BERT for complex systematic review screening to support the future of medical research. Springer Nature Switzerland, Cham, pp 173–182. https://doi.org/10.1007/978-3-031-34344-5_21 · DOI
- Higgins JPT, Altman DG, Gotzsche PC et al (2011) The Cochrane collaboration’s tool for assessing risk of bias in randomised trials. BMJ 343(oct18 2):d5928–d5928. https://doi.org/10.1136/bmj.d5928 · DOI · Paper page
- Hoi SCH, Jin R, Lyu MR (2006) Large-scale text categorization by batch mode active learning. In: Proceedings of the 15th international conference on World Wide Web. ACM, WWW06. https://doi.org/10.1145/1135777.1135870 · DOI · Paper page
- Iparragirre A, Barrio I, Aramendi J et al (2023) Estimation of logistic regression parameters for complex survey data: a real data based simulation study. http://arxiv.org/abs/2303.01754 · arXiv
- Jelodar H, Wang Y, Yuan C et al (2018) Latent Dirichlet allocation (LDA) and topic modeling: models, applications, a survey. http://arxiv.org/abs/1711.04305 · arXiv
- Jha KK, Jha R, Jha AK et al (2021) A brief comparison on machine learning algorithms based on various applications: a comprehensive survey. In: 2021 IEEE international conference on computation system and information technology for sustainable solutions (CSITSS). IEEE. https://doi.org/10.1109/csitss54238.2021.9683524 · DOI
- Joachims T (2006) Training linear SVMs in linear time. In: Proceedings of the 12th ACM SIGKDD international conference on Knowledge discovery and data mining. ACM, KDD06. https://doi.org/10.1145/1150402.1150429 · DOI · Paper page
- Jolliffe I (2014) Principal component analysis. https://doi.org/10.1002/9781118445112.stat06472 · DOI · Paper page
- Klein D, Manning CD (2003) Accurate unlexicalized parsing. In: Proceedings of the 41st annual meeting on association for computational linguistics—ACL ’03. Association for Computational Linguistics, ACL ’03. https://doi.org/10.3115/1075096.1075150 · DOI · Paper page
- Kontonatsios G, Spencer S, Matthew P et al (2020) Using a neural network-based feature extraction method to facilitate citation screening for systematic reviews. Expert Syst Appl X 6:100030. https://doi.org/10.1016/j.eswax.2020.100030 · DOI
- Kotsiantis SB (2011) Decision trees: a recent overview. Artif Intell Rev 39(4):261–283. https://doi.org/10.1007/s10462-011-9272-4 · DOI · Paper page
- Lewis DD (1998) Naive (Bayes) at forty: the independence assumption in information retrieval. Springer, Berlin/Heidelberg, pp 4–15. https://doi.org/10.1007/bfb0026666 · DOI · Paper page
- Mahendra MFR, Azizah NL (2023) Implementation of machine learning to predict the weather using a support vector machine: Implementasi machine learning untuk memprediksi cuaca menggunakan support vector machine. Preprint. https://doi.org/10.21070/ups.2889 · doi:10.21070/ups.2889
- McGreevy KM, Church FC (2020). Active learning survey. https://doi.org/10.1037/t81767-000 · doi:10.1037/t81767-000
- Mergel GD, Silveira MS, da Silva TS (2015) A method to support search string building in systematic literature reviews through visual text mining. In: Proceedings of the 30th annual ACM symposium on applied computing. ACM, SAC 2015. https://doi.org/10.1145/2695664.2695902 · DOI · Paper page
- Mitchell TM (1997) Machine learning. McGraw-Hill, New York
- Moreno-Garcia CF, Jayne C, Elyan E et al (2023) A novel application of machine learning and zero-shot classification methods for automated abstract screening in systematic reviews. Decis Anal J 6:100162. https://doi.org/10.1016/j.dajour.2023.100162 · DOI · Paper page
- Nadkarni PM (2002) An introduction to information retrieval: applications in genomics. Pharmacogenomics J 2(2):96–102. https://doi.org/10.1038/sj.tpj.6500084 · doi:10.1038/sj.tpj.6500084
- Natukunda A, Muchene LK (2023) Unsupervised title and abstract screening for systematic review: a retrospective case-study using topic modelling methodology. Syst Rev. https://doi.org/10.1186/s13643-022-02163-4 · DOI
- Nye B, Li JJ, Patel R et al (2018) A corpus with multi-level annotations of patients, interventions and outcomes to support language processing for medical literature. In: Proceedings of the 56th annual meeting of the association for computational linguistics (volume 1: long papers). Association for Computational Linguistics. https://doi.org/10.18653/v1/p18-1019 · DOI · Paper page
- Ofori-Boateng R, Aceves-Martins M, Jayne C et al (2023) Evaluation of attention-based LSTM and Bi-LSTM networks for abstract text classification in systematic literature review automation. Procedia Comput Sci 222:114–126. https://doi.org/10.1016/j.procs.2023.08.149 · DOI
- Orel E, Ciglenecki I, Thiabaud A et al (2023) An automated literature review tool (literev) for streamlining and accelerating research using natural language processing and machine learning: descriptive performance evaluation study. J Med Internet Res 25:e39736. https://doi.org/10.2196/39736 · DOI
- Paul L, Suman A, Sultan N (2013) Methodological analysis of principal component analysis (PCA) method. Int J Comput Eng Manag 16:32–38
- Popuri SK (2022) An approximation method for fitted random forests. http://arxiv.org/2207.02184. https://api.semanticscholar.org/CorpusID:250279991
- Radford A, Wu J, Child R et al (2019) Language models are unsupervised multitask learners. OpenAI. https://api.semanticscholar.org/CorpusID:160025533
- Rogers A, Gardner M, Augenstein I (2023) QA dataset explosion: a taxonomy of NLP resources for question answering and reading comprehension. ACM Comput Surv 55(10):1–45. https://doi.org/10.1145/3560260 · DOI · Paper page
- Rúbio TR, Gulo CA (2016) Enhancing academic literature review through relevance recommendation: using bibliometric and text-based features for classification. In: 2016 11th Iberian conference on information systems and technologies (CISTI). IEEE, pp 1–6. https://doi.org/10.1109/cisti.2016.7521620 · DOI
- Russell-Rose T, Chamberlain J, Shokraneh F (2019) A visual approach to query formulation for systematic search. In: Proceedings of the 2019 conference on human information interaction and retrieval. ACM, CHIIR ’19. https://doi.org/10.1145/3295750.3298919 · DOI
- Scells H, Zuccon G, Koopman B et al (2020) Automatic Boolean query formulation for systematic review literature search. In: Proceedings of the web conference 2020. ACM, WWW ’20. https://doi.org/10.1145/3366423.3380185 · DOI · Paper page
- Scheffer T, Decomain C, Wrobel S (2001) Active hidden Markov models for information extraction. In: International symposium on intelligent data analysis. Springer, pp 309–318 · DOI · Paper page
- Schmidt L, Weeds J, Higgins J (2020) Data mining in clinical trial text: transformers for classification and question answering tasks. In: Proceedings of the 13th international joint conference on biomedical engineering systems and technologies. SCITEPRESS—Science and Technology Publications. https://doi.org/10.5220/0008945700830094 · DOI
- Shannon CE (1948) A mathematical theory of communication. Bell Syst Tech J 27(3):379–423. https://doi.org/10.1002/j.1538-7305.1948.tb01338.x · DOI · Paper page
- Simon C, Davidsen K, Hansen C et al (2019) Bioreader: a text mining tool for performing classification of biomedical literature. BMC Bioinform. https://doi.org/10.1186/s12859-019-2607-x · DOI · Paper page
- Song J, Lee JK, Choi J et al (2020) Deep learning-based extraction of predicate-argument structure (PAS) in building design rule sentences★. J Comput Des Eng 7(5):563–576. https://doi.org/10.1093/jcde/qwaa046 · DOI
- Tomassetti F, Rizzo G, Vetro A et al (2011) Linked data approach for selection process automation in systematic reviews. In: 15th annual conference on evaluation and assessment in software engineering (EASE 2011). IET. https://doi.org/10.1049/ic.2011.0004 · DOI · Paper page
- Vaswani A, Shazeer N, Parmar N et al (2023) Attention is all you need. http://arxiv.org/abs/1706.03762 · arXiv
- Walkowiak T, Datko S, Maciejewski H (2018) Bag-of-Words, Bag-of-Topics and Word-to-Vec based subject classification of text documents in polish–a comparative study. Springer International Publishing, Cham, pp 526–535. https://doi.org/10.1007/978-3-319-91446-6_49 · doi:10.1007/978-3-319-91446-6_49
- Xie Q, Bishop JA, Tiwari P et al (2022) Pre-trained language models with domain knowledge for biomedical extractive summarization. Knowl-Based Syst 252:109460. https://doi.org/10.1016/j.knosys.2022.109460 · DOI · Paper page
- Zhang D, Baclawski KP, Tsotras VJ (2009) B+-Tree. Springer US, pp 197–200. https://doi.org/10.1007/978-0-387-39940-9_739 · doi:10.1007/978-0-387-39940-9_739
Cited by
Data extraction methods for systematic review (semi)automation: a living systematic review
Schmidt, L., Olorisade, B. K., McGuinness, L. A., Thomas, J., & Higgins, J. P. T. (2021) · F1000Research · doi:10.12688/f1000research.51117.1 · Cited by 106
Data extraction methods for systematic review (semi)automation: Update of a living systematic review
Schmidt, L., Finnerty Mutlu, A. N., Elmore, R., Olorisade, B. K., Thomas, J., & Higgins, J. P. T. (2025) · F1000Research · doi:10.12688/f1000research.51117.3 · Cited by 27
ChatGPT as a Tool for Biostatisticians: A Tutorial on Applications, Opportunities, and Limitations
Dobler, D., Binder, H., Boulesteix, A., Igelmann, J., Köhler, D., Mansmann, U., Pauly, M., Scherag, A., Schmid, M., Al Tawil, A., et al. (2025) · Statistics in Medicine · doi:10.1002/sim.70263 · Cited by 20
Nykvist, B., Macura, B., Xylia, M., & Olsson, E. (2025) · Environmental Evidence · doi:10.1186/s13750-025-00360-x · Cited by 17
Robleto, E., & Shehadeh, L. A. (2025) · American Journal of Physiology. Heart and Circulatory Physiology · doi:10.1152/ajpheart.00374.2025 · Cited by 2
Metadata
- Authors
- Ofori-Boateng, R., Aceves-Martins, M., Wiratunga, N., & Moreno-Garcia, C. F.
- Venue
- Artificial Intelligence Review
- Published
- 2024-07-09
- Volume
- 57
- Issue
- 8
- ISSN
- 0269-2821 (ISSN-L)
- 1573-7462
- OpenAlex
- W4400454376
- Cited by
- 86
- References
- 51
- FWCI
- 11.80
- Type
- journal article
- Language
- en
- Open access
- hybrid
- Licence
- creativecommons.org/licenses/by/4.0 (tdm)
- creativecommons.org/licenses/by/4.0 (vor)
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Towards the automation of systematic reviews using natural language processing, machine learning, and deep learning: a comprehensive review. https://app.cochat.ai/papers/10.1007/s10462-024-10844-w
- The paper
- Ofori-Boateng, R., Aceves-Martins, M., Wiratunga, N., & Moreno-Garcia, C. F. (2024). Towards the automation of systematic reviews using natural language processing, machine learning, and deep learning: a comprehensive review. Artificial Intelligence Review. https://doi.org/10.1007/s10462-024-10844-w
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.