CoChat

Towards the automation of systematic reviews using natural language processing, machine learning, and deep learning: a comprehensive review

Open access
Ofori-Boateng, Regina, Aceves-Martins, Magaly, Wiratunga, Nirmalie, Moreno-Garcia, Carlos Francisco · 2024 · Artificial Intelligence Review
Cited by 86

The DOI matches its Crossref record; 51 of the 51 references it cites were found in a registry or an index.

Abstract

Systematic reviews (SRs) constitute a critical foundation for evidence-based decision-making and policy formulation across various disciplines, particularly in healthcare and beyond. However, the inherently rigorous and structured nature of the SR process renders it laborious for human reviewers. Moreover, the exponential growth in daily published literature exacerbates the challenge, as SRs risk missing out on incorporating recent studies that could potentially influence research outcomes. This pressing need to streamline and enhance the efficiency of SRs has prompted significant interest in leveraging Artificial Intelligence (AI) techniques to automate various stages of the SR process. This review paper provides a comprehensive overview of the current AI methods employed for SR automation, a subject area that has not been exhaustively covered in previous literature. Through an extensive analysis of 52 related works and an original online survey, the primary AI techniques and their applications in automating key SR stages, such as search, screening, data extraction, and risk of bias assessment, are identified. The survey results offer practical insights into the current practices, experiences, opinions, and expectations of SR practitioners and researchers regarding future SR automation. Synthesis of the literature review and survey findings highlights gaps and challenges in the current landscape of SR automation using AI techniques. Based on these insights, potential future directions are discussed. This review aims to equip researchers and practitioners with a foundational understanding of the basic concepts, primary methodologies, and recent advancements in AI-driven SR automation while guiding computer scientists in exploring novel techniques to invigorate further and advance this field.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1007/s10462-024-10844-w

Cites

51 of 51 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 51 references

Research works printed but not linked: 68

  1. Abramovich F, Grinshtein V, Levy T (2021) Multiclass classification by sparse multinomial logistic regression. IEEE Trans Inf Theory 67(7):4637–4646. https://doi.org/10.1109/tit.2021.3075137 · DOI
  2. Aceves-Martins M, López-Cruz L, García-Botello M et al (2021) Interventions to prevent obesity in Mexican children and adolescents: systematic review. Prev Sci 23(4):563–586. https://doi.org/10.1007/s11121-021-01316-6 · DOI
  3. AHO AV (1990) Algorithms for finding patterns in strings. Elsevier, Amsterdam, pp 255–300. https://doi.org/10.1016/b978-0-444-88071-0.50010-2 · DOI · Paper page
  4. Aklouche B, Bounhas I, Slimani Y (2018) Query expansion based on NLP and word embeddings. In: Text retrieval conference. https://api.semanticscholar.org/CorpusID:155085448 · doi:10.6028/nist.sp.500-331.core-jarir
  5. Aklouche B, Bounhas I, Slimani Y (2019) Automatic query reweighting using co-occurrence graphs. In: Proceedings of the 16th international conference on applied computing 2019. IADIS Press, AC 2019. https://doi.org/10.33965/ac2019_201912l005 · doi:10.33965/ac2019_201912l005
  6. Alaofi M, Gallagher L, Sanderson M et al (2023) Can generative LLMS create query variants for test collections? An exploratory study. In: Proceedings of the 46th international ACM SIGIR conference on research and development in information retrieval. ACM, SIGIR ’23. https://doi.org/10.1145/3539618.3591960 · DOI
  7. Albawi S, Mohammed TA, Al-Zawi S (2017) Understanding of a convolutional neural network. In: 2017 international conference on engineering and technology (ICET). pp 1–6. https://doi.org/10.1109/ICEngTechnol.2017.8308186 · DOI · Paper page
  8. Almeida H, Meurs MJ, Kosseim L et al (2016) Data sampling and supervised learning for HIV literature screening. IEEE Trans Nanobiosci 15(4):354–361. https://doi.org/10.1109/bibm.2015.7359733 · DOI
  9. Angluin D (1988) Queries and concept learning. Mach Learn 2:319–342 (https://api.semanticscholar.org/CorpusID:11357867) · DOI · Paper page
  10. Aromataris E, Pearson A (2014) The systematic review: an overview. Am J Nurs 114(3):53–58. https://doi.org/10.1097/01.NAJ.0000444496.24228.2c · DOI · Paper page
  11. August ST (2001) Active learning: theory and applications. Stanford University 13(4):182
  12. Baranwal A, Bagwe BR, Vanitha M (2022) Machine learning in Python: diabetes prediction using machine learning. IGI Global, pp 882–908. https://doi.org/10.4018/978-1-6684-6291-1.ch046 · DOI
  13. Blaizot A, Veettil SK, Saidoung P et al (2022) Using artificial intelligence methods for systematic review in health sciences: a systematic review. Res Synth Methods 13(3):353–362. https://doi.org/10.1002/jrsm.1553 · DOI · Paper page
  14. Blei DM, Ng AY, Jordan MI (2003) Latent Dirichlet allocation. J Mach Learn Res 3(null):993–1022
  15. Booth A, Sutton A, Papaioannou D (2016) Systematic approaches to a successful literature review, 2nd edn. Sage, Thousand Oaks
  16. Bui DDA, Jonnalagadda S, Del Fiol G (2015) Automatically finding relevant citations for clinical guideline development. J Biomed Inform 57:436–445. https://doi.org/10.1016/j.jbi.2015.09.003 · DOI
  17. Cawley M, Beardslee R, Beverly B et al (2020) Novel text analytics approach to identify relevant literature for human health risk assessments: a pilot study with health effects of in utero exposures. Environ Int 134:105228. https://doi.org/10.1016/j.envint.2019.105228 · DOI
  18. Cessie SL, Houwelingen JCV (1992) Ridge estimators in logistic regression. Appl Stat 41(1):191. https://doi.org/10.2307/2347628 · DOI · Paper page
  19. Chiu B, Crichton G, Korhonen A et al (2016) How to train good word embeddings for biomedical NLP. In: Proceedings of the 15th workshop on biomedical natural language processing. Association for Computational Linguistics. https://doi.org/10.18653/v1/w16-2922 · DOI · Paper page
  20. Cho K, van Merrienboer B, Gulcehre C et al (2014) Learning phrase representations using RNN encoder-decoder for statistical machine translation. In: Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP). Association for Computational Linguistics. https://doi.org/10.3115/v1/d14-1179 · DOI · Paper page
  21. Cohn D, Atlas L, Ladner R (1994) Improving generalization with active learning. Mach Learn 15(2):201–221. https://doi.org/10.1007/bf00993277 · DOI · Paper page
  22. Devlin J, Chang MW, Lee K et al (2019) BERT: pre-training of deep bidirectional transformers for language understanding. http://arxiv.org/abs/1810.04805 · arXiv
  23. Frunza O, Inkpen D, Matwin S et al (2011) Exploiting the systematic review protocol for classification of medical abstracts. Artif Intell Med 51(1):17–25. https://doi.org/10.1016/j.artmed.2010.10.005 · DOI
  24. Gonzalez-Toral S, Freire R, Gualan R et al (2019) A ranking-based approach for supporting the initial selection of primary studies in a systematic literature review. In: 2019 XLV Latin American computing conference (CLEI). IEEE. https://doi.org/10.1109/clei47609.2019.235079 · DOI
  25. Gosavi A (2009) Reinforcement learning: a tutorial survey and recent advances. INFORMS J Comput 21(2):178–192. https://doi.org/10.1287/ijoc.1080.0305 · DOI · Paper page
  26. Gulo CA, Rúbio TR, Tabassum S et al (2015) Mining scientific articles powered by machine learning techniques. In: 2015 Imperial College computing student workshop (ICCSW 2015). Schloss Dagstuhl-Leibniz-Zentrum fuer Informatik. https://doi.org/10.4230/OASIcs.ICCSW.2015.21 · DOI
  27. Guo G, Wang H, Bell D et al (2003) KNN model-based approach in classification. Springer, Berlin/Heidelberg, pp 986–996. https://doi.org/10.1007/978-3-540-39964-3_62 · DOI · Paper page
  28. Hans C (2011) Elastic net regression modeling with the orthant normal prior. J Am Stat Assoc 106(496):1383–1393. https://doi.org/10.1198/jasa.2011.tm09241 · DOI · Paper page
  29. Hashimoto K, Kontonatsios G, Miwa M et al (2016) Topic detection using paragraph vectors to support active learning in systematic reviews. J Biomed Inform 62:59–65. https://doi.org/10.1016/j.jbi.2016.06.001 · DOI · Paper page
  30. Hasny M, Vasile AP, Gianni M et al (2023) BERT for complex systematic review screening to support the future of medical research. Springer Nature Switzerland, Cham, pp 173–182. https://doi.org/10.1007/978-3-031-34344-5_21 · DOI
  31. Higgins JPT, Altman DG, Gotzsche PC et al (2011) The Cochrane collaboration’s tool for assessing risk of bias in randomised trials. BMJ 343(oct18 2):d5928–d5928. https://doi.org/10.1136/bmj.d5928 · DOI · Paper page
  32. Hoi SCH, Jin R, Lyu MR (2006) Large-scale text categorization by batch mode active learning. In: Proceedings of the 15th international conference on World Wide Web. ACM, WWW06. https://doi.org/10.1145/1135777.1135870 · DOI · Paper page
  33. Iparragirre A, Barrio I, Aramendi J et al (2023) Estimation of logistic regression parameters for complex survey data: a real data based simulation study. http://arxiv.org/abs/2303.01754 · arXiv
  34. Jelodar H, Wang Y, Yuan C et al (2018) Latent Dirichlet allocation (LDA) and topic modeling: models, applications, a survey. http://arxiv.org/abs/1711.04305 · arXiv
  35. Jha KK, Jha R, Jha AK et al (2021) A brief comparison on machine learning algorithms based on various applications: a comprehensive survey. In: 2021 IEEE international conference on computation system and information technology for sustainable solutions (CSITSS). IEEE. https://doi.org/10.1109/csitss54238.2021.9683524 · DOI
  36. Joachims T (2006) Training linear SVMs in linear time. In: Proceedings of the 12th ACM SIGKDD international conference on Knowledge discovery and data mining. ACM, KDD06. https://doi.org/10.1145/1150402.1150429 · DOI · Paper page
  37. Jolliffe I (2014) Principal component analysis. https://doi.org/10.1002/9781118445112.stat06472 · DOI · Paper page
  38. Klein D, Manning CD (2003) Accurate unlexicalized parsing. In: Proceedings of the 41st annual meeting on association for computational linguistics—ACL ’03. Association for Computational Linguistics, ACL ’03. https://doi.org/10.3115/1075096.1075150 · DOI · Paper page
  39. Kontonatsios G, Spencer S, Matthew P et al (2020) Using a neural network-based feature extraction method to facilitate citation screening for systematic reviews. Expert Syst Appl X 6:100030. https://doi.org/10.1016/j.eswax.2020.100030 · DOI
  40. Kotsiantis SB (2011) Decision trees: a recent overview. Artif Intell Rev 39(4):261–283. https://doi.org/10.1007/s10462-011-9272-4 · DOI · Paper page
  41. Lewis DD (1998) Naive (Bayes) at forty: the independence assumption in information retrieval. Springer, Berlin/Heidelberg, pp 4–15. https://doi.org/10.1007/bfb0026666 · DOI · Paper page
  42. Mahendra MFR, Azizah NL (2023) Implementation of machine learning to predict the weather using a support vector machine: Implementasi machine learning untuk memprediksi cuaca menggunakan support vector machine. Preprint. https://doi.org/10.21070/ups.2889 · doi:10.21070/ups.2889
  43. McGreevy KM, Church FC (2020). Active learning survey. https://doi.org/10.1037/t81767-000 · doi:10.1037/t81767-000
  44. Mergel GD, Silveira MS, da Silva TS (2015) A method to support search string building in systematic literature reviews through visual text mining. In: Proceedings of the 30th annual ACM symposium on applied computing. ACM, SAC 2015. https://doi.org/10.1145/2695664.2695902 · DOI · Paper page
  45. Mitchell TM (1997) Machine learning. McGraw-Hill, New York
  46. Moreno-Garcia CF, Jayne C, Elyan E et al (2023) A novel application of machine learning and zero-shot classification methods for automated abstract screening in systematic reviews. Decis Anal J 6:100162. https://doi.org/10.1016/j.dajour.2023.100162 · DOI · Paper page
  47. Nadkarni PM (2002) An introduction to information retrieval: applications in genomics. Pharmacogenomics J 2(2):96–102. https://doi.org/10.1038/sj.tpj.6500084 · doi:10.1038/sj.tpj.6500084
  48. Natukunda A, Muchene LK (2023) Unsupervised title and abstract screening for systematic review: a retrospective case-study using topic modelling methodology. Syst Rev. https://doi.org/10.1186/s13643-022-02163-4 · DOI
  49. Nye B, Li JJ, Patel R et al (2018) A corpus with multi-level annotations of patients, interventions and outcomes to support language processing for medical literature. In: Proceedings of the 56th annual meeting of the association for computational linguistics (volume 1: long papers). Association for Computational Linguistics. https://doi.org/10.18653/v1/p18-1019 · DOI · Paper page
  50. Ofori-Boateng R, Aceves-Martins M, Jayne C et al (2023) Evaluation of attention-based LSTM and Bi-LSTM networks for abstract text classification in systematic literature review automation. Procedia Comput Sci 222:114–126. https://doi.org/10.1016/j.procs.2023.08.149 · DOI
  51. Orel E, Ciglenecki I, Thiabaud A et al (2023) An automated literature review tool (literev) for streamlining and accelerating research using natural language processing and machine learning: descriptive performance evaluation study. J Med Internet Res 25:e39736. https://doi.org/10.2196/39736 · DOI
  52. Paul L, Suman A, Sultan N (2013) Methodological analysis of principal component analysis (PCA) method. Int J Comput Eng Manag 16:32–38
  53. Popuri SK (2022) An approximation method for fitted random forests. http://arxiv.org/2207.02184. https://api.semanticscholar.org/CorpusID:250279991
  54. Radford A, Wu J, Child R et al (2019) Language models are unsupervised multitask learners. OpenAI. https://api.semanticscholar.org/CorpusID:160025533
  55. Rogers A, Gardner M, Augenstein I (2023) QA dataset explosion: a taxonomy of NLP resources for question answering and reading comprehension. ACM Comput Surv 55(10):1–45. https://doi.org/10.1145/3560260 · DOI · Paper page
  56. Rúbio TR, Gulo CA (2016) Enhancing academic literature review through relevance recommendation: using bibliometric and text-based features for classification. In: 2016 11th Iberian conference on information systems and technologies (CISTI). IEEE, pp 1–6. https://doi.org/10.1109/cisti.2016.7521620 · DOI
  57. Russell-Rose T, Chamberlain J, Shokraneh F (2019) A visual approach to query formulation for systematic search. In: Proceedings of the 2019 conference on human information interaction and retrieval. ACM, CHIIR ’19. https://doi.org/10.1145/3295750.3298919 · DOI
  58. Scells H, Zuccon G, Koopman B et al (2020) Automatic Boolean query formulation for systematic review literature search. In: Proceedings of the web conference 2020. ACM, WWW ’20. https://doi.org/10.1145/3366423.3380185 · DOI · Paper page
  59. Scheffer T, Decomain C, Wrobel S (2001) Active hidden Markov models for information extraction. In: International symposium on intelligent data analysis. Springer, pp 309–318 · DOI · Paper page
  60. Schmidt L, Weeds J, Higgins J (2020) Data mining in clinical trial text: transformers for classification and question answering tasks. In: Proceedings of the 13th international joint conference on biomedical engineering systems and technologies. SCITEPRESS—Science and Technology Publications. https://doi.org/10.5220/0008945700830094 · DOI
  61. Shannon CE (1948) A mathematical theory of communication. Bell Syst Tech J 27(3):379–423. https://doi.org/10.1002/j.1538-7305.1948.tb01338.x · DOI · Paper page
  62. Simon C, Davidsen K, Hansen C et al (2019) Bioreader: a text mining tool for performing classification of biomedical literature. BMC Bioinform. https://doi.org/10.1186/s12859-019-2607-x · DOI · Paper page
  63. Song J, Lee JK, Choi J et al (2020) Deep learning-based extraction of predicate-argument structure (PAS) in building design rule sentences★. J Comput Des Eng 7(5):563–576. https://doi.org/10.1093/jcde/qwaa046 · DOI
  64. Tomassetti F, Rizzo G, Vetro A et al (2011) Linked data approach for selection process automation in systematic reviews. In: 15th annual conference on evaluation and assessment in software engineering (EASE 2011). IET. https://doi.org/10.1049/ic.2011.0004 · DOI · Paper page
  65. Vaswani A, Shazeer N, Parmar N et al (2023) Attention is all you need. http://arxiv.org/abs/1706.03762 · arXiv
  66. Walkowiak T, Datko S, Maciejewski H (2018) Bag-of-Words, Bag-of-Topics and Word-to-Vec based subject classification of text documents in polish–a comparative study. Springer International Publishing, Cham, pp 526–535. https://doi.org/10.1007/978-3-319-91446-6_49 · doi:10.1007/978-3-319-91446-6_49
  67. Xie Q, Bishop JA, Tiwari P et al (2022) Pre-trained language models with domain knowledge for biomedical extractive summarization. Knowl-Based Syst 252:109460. https://doi.org/10.1016/j.knosys.2022.109460 · DOI · Paper page
  68. Zhang D, Baclawski KP, Tsotras VJ (2009) B+-Tree. Springer US, pp 197–200. https://doi.org/10.1007/978-0-387-39940-9_739 · doi:10.1007/978-0-387-39940-9_739

Cited by

See all 86

Metadata

Authors
Ofori-Boateng, R., Aceves-Martins, M., Wiratunga, N., & Moreno-Garcia, C. F.
Venue
Artificial Intelligence Review
Published
2024-07-09
Volume
57
Issue
8
ISSN
  • 0269-2821 (ISSN-L)
  • 1573-7462
OpenAlex
W4400454376
Cited by
86
References
51
FWCI
11.80
Type
journal article
Language
en
Open access
hybrid
Licence

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Towards the automation of systematic reviews using natural language processing, machine learning, and deep learning: a comprehensive review. https://app.cochat.ai/papers/10.1007/s10462-024-10844-w
The paper
Ofori-Boateng, R., Aceves-Martins, M., Wiratunga, N., & Moreno-Garcia, C. F. (2024). Towards the automation of systematic reviews using natural language processing, machine learning, and deep learning: a comprehensive review. Artificial Intelligence Review. https://doi.org/10.1007/s10462-024-10844-w

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.