CoChat
Sign Up
PreprintCited by 156Meta-analysis and systematic reviewsArtificial Intelligence in Healthcare and EducationComputational and Text Analysis Methods

The emergence of Large Language Models (LLM) as a tool in literature reviews: an LLM automated systematic review

Open access
Scherbakov, Dmitry, Hubig, Nina, Jansari, Vinita, Bakumenko, Alexander, Lenert, Leslie A.·2024·arXiv
DOI 10.48550/arxiv.2409.04600arXiv 2409.04600PMC PMC12089777
Open this paper in CoChat — free accountOpen PDF

Abstract

Objective: This study aims to summarize the usage of Large Language Models (LLMs) in the process of creating a scientific review. We look at the range of stages in a review that can be automated and assess the current state-of-the-art research projects in the field. Materials and Methods: The search was conducted in June 2024 in PubMed, Scopus, Dimensions, and Google Scholar databases by human reviewers. Screening and extraction process took place in Covidence with the help of LLM add-on which uses OpenAI gpt-4o model. ChatGPT was used to clean extracted data and generate code for figures in this manuscript, ChatGPT and Scite.ai were used in drafting all components of the manuscript, except the methods and discussion sections. Results: 3,788 articles were retrieved, and 172 studies were deemed eligible for the final review. ChatGPT and GPT-based LLM emerged as the most dominant architecture for review automation (n=126, 73.2%). A significant number of review automation projects were found, but only a limited number of papers (n=26, 15.1%) were actual reviews that used LLM during their creation. Most citations focused on automation of a particular stage of review, such as Searching for publications (n=60, 34.9%), and Data extraction (n=54, 31.4%). When comparing pooled performance of GPT-based and BERT-based models, the former were better in data extraction with mean precision 83.0% (SD=10.4), and recall 86.0% (SD=9.8), while being slightly less accurate in title and abstract screening stage (Maccuracy=77.3%, SD=13.0). Discussion/Conclusion: Our LLM-assisted systematic review revealed a significant number of research projects related to review automation using LLMs. The results looked promising, and we anticipate that LLMs will change in the near future the way the scientific reviews are conducted.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2409.04600

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Compared to machine learning-assisted review, GPT-based models were slightly less accurate in title and abstract screening.

    “Compared to machine learning-assisted review, GPT-based models were slightly less accurate in title and abstract screening (Scherbakov et al., 2024).”

    From Machine Learning-Assisted Systematic Review: A Case Study in Learning Analytics (2025) · cites this paper as Scherbakov et al., 2024

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

No references could be resolved yet.

Research works printed but not linked: 56

  1. Toh TS, Lee JH. Statistical note: using scoping and systematic reviews. Pediatr Crit Care Med. 2021;22:572-575. 002. Abushouk AI, Yunusa I, Elmehrath AO, et al. Quality assessment of published systematic reviews in high impact cardiology jour- nals: revisiting the evidence pyramid. Front Cardiovasc Med 2021;8:671569. 003. Acar IH, Avcılar G, Yazıcı G, Bostancı S. The roles of adolescents' emotional problems and social media addiction on their self-esteem. Curr Psychol. 2020;41:6838-6847. https://doi. org/10.1007/s12144-020-01174-5 004. · DOI
  2. Borah R, Brown AW, Capers PL, et al. Analysis of the time and workers needed to conduct systematic reviews of medical inter- ventions using data from the PROSPERO registry. BMJ Open. 2017;7:e012545. 005. Munn Z, Peters MDJ, Stern C, et al. Systematic review or scoping review? Guidance for authors when choosing between a system- atic or scoping review approach. BMC Med Res Methodol. 2018;18:143-147. 006. Kellermeyer L, Harnke B, Knight S. Covidence and Rayyan. JMLA. 2018;106:580. 007. Chan JL, Murphy KA, Sarna JR. Myoclonus and cerebellar ataxia associated with COVID-19: a case report and systematic review. J Neurol. 2021;268:3517-3548. 008. Machine learning-the game changer for trustworthy evidence. Accessed August, 14, 2024. https://www.covidence.org/blog/ machine-learning-the-game-changer-for-trustworthy-evidence/ 009.
  3. Gartlehner G, Affengruber L, Titscher V, et al. Single-reviewer abstract screening missed 13 percent of relevant studies: a crowd- based, randomized controlled trial. J Clin Epidemiol. 2020;121:20-28. 010. Marshall IJ, Wallace BC. Toward systematic review automation: a practical guide to using machine learning tools in research syn- thesis. Syst Rev. 2019;8:163-110. 011. Rasheed Z, Waseem M, Syst€ a K, Abrahamsson P. 2024. Large language model evaluation via multi ai agents: preliminary results. arXiv, preprint arXiv: 2404.01023, preprint: not peer reviewed. 012. Wang S, Scells H, Zhuang S, Potthast M, Koopman B, Zuccon G, eds. Zero-shot generative large language models for systematic review screening automation. In: European Conference on Infor- mation Retrieval. Springer; 2024:403-420. 013. Zaki M, Namireddy SR, Pittie T, et al. Natural language processing-guided meta-analysis and structure factor database extraction from glass literature. J Non-Crystalline Solids: X. 2022;15:100103. 014. National Institute for Health and Care Excellence. Use of AI in evidence generation: NICE position statement. Accessed April 24, 2025. https://www.nice.org.uk/about/what-we-do/our- research-work/use-of-ai-in-evidence-generation-nice-position- statement?utm_medium=social&utm_source=linkedin&utm_ campaign=aiposition 015. · arXiv
  4. Liu Z. ChatGPT-a new milestone in the field of education. ACE. 2024;35:129-133. 016. Mu Y, He D. The potential applications and challenges of ChatGPT in the medical field. Int J Gen Med. 2024;17:817-826. 017. Tlili A, Shehata B, Adarkwah MA, et al. What if the devil is my guardian angel: ChatGPT as a case study of using Chatbots in education. Smart Learn Environ. 2023;10:15. 018. Wang Y, Wang L, Rastegar-Mojarad M, et al. Clinical informa- tion extraction applications: a literature review. J Biomed Inform. 2018;77:34-49. 019. Jonnalagadda SR, Goyal P, Huffman MD. Automating data extraction in systematic reviews: a systematic review. Syst Rev. 2015;4:78-16. 020. Tsou AY, Treadwell JR, Erinoff E, et al. Machine learning for screening prioritization in systematic reviews: comparative per- formance of Abstrackr and EPPI-Reviewer. Syst Rev. 2020;9:73-14. 021. O'Mara-Eves A, Thomas J, McNaught J, et al. Using text mining for study identification in systematic reviews: a systematic review of current approaches. Syst Rev. 2015;4:1-22. 022. Thomas J, Noel-Storr A, Marshall I, Living Systematic Review Network, et al. Living systematic reviews: 2. Combining human and machine effort. J Clin Epidemiol. 2017;91:31-37. 023. Marshall IJ, Kuiper J, Wallace BC. RobotReviewer: evaluation of a system for automatically assessing bias in clinical trials. J Am Med Inform Assoc. 2016;23:193-201. 024. Goddard K, Roudsari A, Wyatt JC. Automation bias: a system- atic review of frequency, effect mediators, and mitigators. J Am Med Inform Assoc. 2012;19:121-127. 025. Scherbakov D. Large language models in scoping and systematic reviews automation: an automated systematic review [protocol registration]. Accessed April 24, 2025. https://doi.org/10.17605/ OSF.IO/EJKSY 026. Page MJ, McKenzie JE, Bossuyt PM, et al. The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. BMJ. 2021;372:n71. 027. Dimensions (database). 2025. Accessed April 24, 2025. https:// app.dimensions.ai/ 028. Microsoft. Azure OpenAI Service [cited 2025 Mar 20]. Accessed April 24, 2025. https://azure.microsoft.com/en-us/products/ai- services/openai-service 029. Google. NotebookLM [cited 2025 Mar 20]. Accessed April 24, 2025. https://notebooklm.google.com 030. OpenAI. Hello GPT-4o. 2025 [cited 2025 Mar 20]. Accessed April 24, 2025. https://openai.com/index/hello-gpt-4o/ 031. scite. AI for Research. [cited 2025 Mar 20]. Accessed April 24, 2025. https://scite.ai/ 032. · DOI
  5. Haddaway NR, Westgate MJ. Predicting the time needed for environmental systematic reviews and systematic maps. Conserv Biol. 2019;33:434-443. 033. Agarwal S, Laradji IH, Charlin L, Pal C. 2024. LitLLM: a toolkit for scientific literature review. arXiv, preprint arXiv, not peer reviewed. 034.
  6. Anghelescu A, Ciobanu I, Munteanu C, et al. ChatGPT: "To be or not to be" . . . in academic research. The human mind's analyti- cal rigor and capacity to discriminate between AI bots' truths and hallucinations. Balneo PRM Res J 2023;14:614. 035. Bersenev D, Yachie-Kinoshita A, Palaniappan SK. Replicating a high-impact scientific publication using systems of large language models. bioRxiv 2024.04.08.588614, 2024. 036. Dossantos J, An J, Javan R. Eyes on AI: ChatGPT's transformative potential impact on ophthalmology. Cureus. 2023;15:e40765. 037.
  7. Jenko N, Ariyaratne S, Jeys L, et al. An evaluation of AI gener- ated literature reviews in musculoskeletal radiology. Surgeon. 2024;22:194-197. 038. Khlaif ZN, Mousa A, Hattab MK, et al. The potential and con- cerns of using AI in scientific research: ChatGPT performance evaluation. JMIR Med Educ. 2023;9:e47049. 039. Li X, Ouyang J. 2024. Explaining relationships among research papers. arXiv, preprint arXiv, not peer reviewed. 040. Livberber T. Toward non-human-centered design: designing an academic article with ChatGPT. EPI. 2023;32 041. Lozano A, Fleming SL, Chiang C-C, et al. Clinfo.ai: an open- source retrieval-augmented large language model system for answering medical questions using scientific literature. Pac Symp Biocomput. 2024;29:8-23. 042. Najafali D, Camacho JM, Reiche E, et al. Truth or lies? The pit- falls and limitations of ChatGPT in systematic review creation. Aesthet Surg J. 2023;43:NP654-NP655. 043. Semrl N, Feigl S, Taumberger N, et al. AI language models in human reproduction research: exploring ChatGPT's potential to assist academic writing. Hum Reprod. 2023;38:2281-2288. 044. Teperikidis E, Boulmpou A, Papadopoulos C. Prompting ChatGPT to perform an umbrella review. Acta Cardiol. 2024;79:403-404. 045. Wang J, Huang C, Yan S, Xie W, He D. When young scholars cooperate with LLMs in academic tasks: the influence of individ- ual differences and task complexities. Int J Hum-Comput Inter- act 2024:41:4624-4639. 046. Wu CL, Cho B, Gabriel R, et al. Addition of dexamethasone to prolong peripheral nerve blocks: a ChatGPT-created narrative review. Reg Anesth Pain Med. 2023;49:777-781 047.
  8. Yun HS, Marshall IJ, Trikalinos TA, Wallace BC. 2023. Apprais- ing the potential uses and harms of LLMs for medical systematic reviews. arXiv, preprint arXiv, not peer reviewed. 048.
  9. Zhao S, Chen S, Zhou J, et al. Potential to transform words to watts with large language models in battery research. Cell Rep Phys Sci. 2024;5:101844. 049. Huang J, Tan M. The role of ChatGPT in scientific communica- tion: writing better scientific review articles. Am J Cancer Res. 2023;13:1148-1154. 050. Lamov� sek N. Analysis of research on artificial intelligence in pub- lic administration. CEPAR. 2023;21:77-96. 051. Liu H, Peng Y, Weng C. How good is ChatGPT for medication evidence synthesis? Stud Health Technol Inform. 2023;302:1062-1066. 052. Pedroso-Roussado C. Investigating the limitations of fashion research methods in applying a sustainable design practice: a sys- tematic review. Preprints.org 202310.0250.v2, 2023. 053. Schopow N, Osterhoff G, Baur D. Applications of the natural language processing tool ChatGPT in clinical practice: compara- tive study and augmented systematic review. JMIR Med Inform. 2023;11:e48933. 054. Teperikidis E, Boulmpou A, Potoupni V, et al. Does the long- term administration of proton pump inhibitors increase the risk of adverse cardiovascular outcomes? A ChatGPT powered umbrella review. Acta Cardiol. 2023;78:980-988. 055. Ahmed U. Reimagining open data ecosystems: a practical approach using AI, CI, and Knowledge Graphs. In: BIR Work- shops. 2023:235-249. 056. Marshalova A, Bruches E, Batura T. 2023. Automatic aspect extraction from scientific texts. arXiv, preprint arXiv, not peer reviewed. 057.
  10. Mutinda FW, Liew K, Yada S, et al. Automatic data extraction to support meta-analysis statistical analysis: a case study on breast cancer. BMC Med Inform Decis Mak. 2022;22:158. 058. Panayi A, Ward K, Benhadji-Schaff A, et al. Evaluation of a pro- totype machine learning tool to semi-automate data extraction for systematic literature reviews. Syst Rev. 2023;12:187. 059. Scells H, Schlatt F, Potthast M. Smooth operators for effective systematic review queries. 2023. Accessed April 24, 2025. https:// doi.org/10.1145/3539618.3591768 060. · DOI
  11. Shinde K, Roy T, Ghosal. An T. Extractive-abstractive approach for multi-document summarization of scientific articles for literature review. 2022. Accessed April 24, 2025. https://www. scopus.com/inward/record.uri?eid=2-s2.0-85174492822&part- nerID=40&md5=657f586349a915d084d9edd9031ecc23 061.
  12. Wang Q, Liao J, Lapata M, et al. PICO entity extraction for pre- clinical animal literature. Syst Rev. 2022;11:209. 062. Whitton J, Hunter A. Automated tabulation of clinical trial results: a joint entity and relation extraction approach with transformer-based language representations. Artif Intell Med. 2023;144:102661. 063. Yazi FS, Vong WT, Raman V, Then PHH, Lunia MJ. Towards automated detection of contradictory research claims in medical literature using deep learning approach. 2021. Accessed April 24, 2025. https://doi.org/10.1109/CAMP51653.2021.9498061 064. · DOI
  13. Oami T, Okada Y, Nakada T-A Accuracy and reliability of data extraction for systematic reviews using large language models: a protocol for a prospective study. medRxiv 2024.05.22.24307740, 2024. 065. Prasad D, Pimpude M, Alankar A. 2024. Towards development of automated knowledge maps and databases for materials engi- neering using large language models. arXiv, preprint arXiv, not peer reviewed. 066. Ye A, Maiti A, Schmidt M, et al. A hybrid semi-automated work- flow for systematic and literature review processes with large lan- guage model analysis. Future Internet. 2024;16:167. 067. Gartlehner G, Kahwati L, Hilscher R, et al. Data extraction for evidence synthesis using a large language model: a proof-of- concept study. Res Synth Methods. 2024;15:576-589. 068. Aronson SJ, Machini K, Shin J, et al. 2023. Preparing to integrate generative pretrained transformer series 4 models into genetic variant assessment workflows: assessing performance, drift, and nondeterminism characteristics relative to classifying functional evidence in literature. arXiv, preprint arXiv, not peer reviewed. 069. Flaherty HB, Yurch J. Beyond plagiarism: ChatGPT as the Van- guard of technological revolution in research and citation. Res Social Work Pract. 2024;34:483-486. 070. Gue CCY, Rahim NDA, Rojas-Carabali W, et al. Evaluating the OpenAI's GPT-3.5 Turbo's performance in extracting informa- tion from scientific articles on diabetic retinopathy. Syst Rev. 2024;13:135. 071.
  14. Khraisha Q, Put S, Kappenberg J, et al. Can large language mod- els replace humans in systematic reviews? Evaluating GPT-4's efficacy in screening and extracting data from peer-reviewed and grey literature in multiple languages. Res Synth Methods. 2024;15:616-626. 073. Kılıc¸ DK, Vasegaard AE, Desoeuvres A, Nielsen P. 2023. A semi- automated solution approach recommender for a given use case: a case study for AI/ML in oncology via Scopus and OpenAI. arXiv, preprint arXiv, not peer reviewed. 074.
  15. Mahmoudi H, Chang D, Lee H, Ghaffarzadegan N, Jalali MS. A critical assessment of large language models for systematic reviews: utilizing ChatGPT for complex data extraction. SSRN J 2024. 075. Mahuli SA, Rai A, Mahuli AV, et al. Application ChatGPT in conducting systematic reviews and meta-analyses. Br Dent J. 2023;235:90-92. 076. Miao H, Yu X, Wu H. Mining topic structure of AI algorithmic literature. 2023. Accessed April 24, 2025. https://doi.org/10. 1109/IEIR59294.2023.10391253 077. Reason T, Benbow E, Langham J, et al. Artificial intelligence to automate network meta-analyses: four case studies to evaluate the potential application of large language models. Pharmacoe- con Open. 2024;8:205-220. 078. Schmidt L, Hair K, Graziozi S, et al. 2024. Exploring the use of a large language model for data extraction in systematic reviews: a rapid feasibility study. arXiv, preprint arXiv, not peer reviewed. 079. Serajeh NT, Mohammadi I, Fuccella V, De Rosa M. 2024. LLMs in HCI data work: bridging the gap between information retrieval and responsible research practices. arXiv, preprint arXiv, not peer reviewed. 080. · DOI
  16. Shah-Mohammadi F, Finkelstein J. Large language model-based architecture for automatic outcome data extraction to support meta-analysis. 2024. Accessed April 24, 2025. https://doi.org/10. 1109/CCWC60891.2024.10427829 081. Susnjak T. 2023. PRISMA-DFLLM: an extension of PRISMA for systematic literature reviews using domain-specific finetuned large language models. arXiv, preprint arXiv: 2306.14905, not peer reviewed. 082. · DOI
  17. Susnjak T, Hwang P, Reyes NH, et al. 2024. Automating research synthesis with domain-specific large language model fine-tuning. arXiv, preprint arXiv: 2404.08680, not peer reviewed. 083. Tang Y, Xiao Z, Li X, et al. Large language model in medical information extraction from titles and abstracts with prompt engineering strategies: a comparative study of GPT-3.5 and GPT- 4. medRxiv 2024.03.20.24304572, 2024. 084. Tao K, Osman ZA, Tzou PL, et al. GPT-4 performance on query- ing scientific publications: reproducibility, accuracy, and impact of an instruction sheet. BMC Med Res Methodol. 2024;24:139. 085. Tovar DA. 2023. AI literature review suite. arXiv, preprint arXiv, not peer reviewed. 086. · arXiv
  18. Uittenhove K, Martinelli P, Roquet A. 2024. Large language models in psychology: application in the context of a systematic literature review. PsyArXiv, preprint PsyArXiv, not peer reviewed. 087.
  19. Urrutia F, Buc C, Barriere V. 2023. Deep natural language feature learning for interpretable prediction. arXiv, preprint arXiv, not peer reviewed. 088.
  20. Wang X, Luo G. 2024. MetaMate: large language model to the rescue of automated data extraction for educational systematic reviews and meta-analyses. EdArXiv, preprint EdArXiv, not peer reviewed. 089. Yun HS, Pogrebitskiy D, Marshall IJ, Wallace BC. 2024. Auto- matically extracting numerical results from randomized con- trolled trials with large language models. arXiv, preprint arXiv, not peer reviewed. 090.
  21. Zamani S, Sinha R. Generative AI-the end of systematic reviews in PhD projects? ACM Inroads. 2024;15:48-50. 091. Ghosh M, Mukherjee S, Ganguly A, et al. AlpaPICO: extraction of PICO frames from clinical trial documents using LLMs. Meth- ods. 2024;226:78-88. 092. Tsai H-C, Huang Y-F, Kuo C-W. Comparative analysis of auto- matic literature review using Mistral large language model and human reviewers. Research Square, 2024. Accessed April 24, 2025. https://doi.org/10.21203/rs.3.rs-4022248/v1 093. Hossain MM. Using ChatGPT and other forms of generative AI in systematic reviews: challenges and opportunities. J Med Imag- ing Radiat Sci. 2024;55:11-12. 094. Jain S, Kumar A, Roy T, Shinde K, Vignesh G, Tondulkar R. Sci- Space literature review: harnessing AI for effortless scientific dis- covery. 2024. 095. Sami AM, Rasheed Z, Kemell K-K, et al. 2024. System for sys- tematic literature review using multiple AI agents: concept and an empirical evaluation. arXiv, preprint arXiv, not peer reviewed. 096. Grokhowsky N. Reducing knowledge synthesis workload time using a text-mining algorithm for research location and subtopic extraction from geographically dependent research publications. Research Square, 2023. Accessed April 24, 2025. https://doi.org/ 10.21203/rs.3.rs-3129370/v1 097. · DOI
  22. Sun Z, Zhang R, Doi SA, et al. How good are large language models for automated data extraction from randomized trials? medRxiv 2024.02. 20.24303083, 2024. 098. White M. Sample size in quantitative instrument-based studies published in Scopus up to 2022: an artificial intelligence aided systematic review. Acta Psychol (Amst). 2023;241:104095. 099. Janes A, Li X, Lenarduzzi V. Open tracing tools: overview and critical comparison. J Syst Softw. 2023;204:111793. 100. Noe-Steinm€ uller N, Scherbakov D, Zhuravlyova A, et al. Defin- ing suffering in pain: a systematic review on pain-related suffer- ing using natural language processing. Pain. 2024;165:1434-1449. 101. Pattyn F. Preliminary structured literature review results using ChatGPT: towards a pragmatic framework for product managers at software startups. In: 2023 IEEE 31st International Require- ments Engineering Conference Workshops (REW). IEEE; 2023:367-370. 102. Beheshti M, Amoozad Mahdiraji H, Rocha-Lona L. Transition- ing drivers from linear to circular economic models: evidence of entrepreneurship in emerging nations. Manage Decis. 2024;62:2714-2736. 103. Ambalavanan AK, Devarakonda M. 2020. Cascade neural ensemble for identifying scientifically sound articles. arXiv, pre- print arXiv, not peer reviewed. 104.
  23. Martenot V, Masdeu V, Cupe J, et al. LiSA: an assisted literature search pipeline for detecting serious adverse drug events with deep learning. BMC Med Inform Decis Mak. 2022;22:338. 105. Guo F, Luo Y, Yang L, Zhang Y. SciMine: an efficient systematic prioritization model based on richer semantic information. In: Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval. 2023:205-215. 106. Atkinson CF. ChatGPT and computational-based research: bene- fits, drawbacks, and machine learning applications. Discov Artif Intell. 2023;3:article 42. 107. Demir GB, S€ uk€ ut Y, Duran GS, et al. Enhancing systematic reviews in orthodontics: a comparative examination of GPT-3.5 and GPT-4 for generating PICO-based queries with tailored prompts and configurations. Eur J Orthod. 2024;46:cjae011. 108. Giunti G, Doherty CP. Cocreating an automated mHealth apps systematic review process with generative AI: design science research approach. JMIR Med Educ. 2024;10:e48949. 109.
  24. Qureshi R, Shaughnessy D, Gill KAR, et al. Are ChatGPT and large language models "the answer" to bringing us closer to sys- tematic review automation? Syst Rev. 2023;12:72. 110. Whang Y. ChatGPT for editors: enhancing efficiency and effec- tiveness. Sci Editing. 2024;11:84-90. 111. Abd-Alrazaq A, Nashwan AJ, Shah Z, et al. Machine learning- based approach for identifying research gaps: COVID-19 as a case study. JMIR Form Res. 2024;8:e49411. 112. Khadhraoui M, Bellaaj H, Ammar MB, et al. Survey of BERT- base models for scientific text classification: COVID-19 case study. Appl Sci. 2022;12:2891. 113. Liang F, Hou F, Farshidi S, Jansen S. Sentiment analysis for soft- ware quality assessment. In: CEUR Workshop Proceedings, Vol. 3567. CEUR WS; 2023:17-24. 114.
  25. Likhareva D, Sankaran H, Thiyagarajan S. 2024. Empowering interdisciplinary research with BERT-based models: an approach through SciBERT-CNN with topic modeling. arXiv, preprint arXiv, not peer reviewed. 115.
  26. Alshami A, Elsayed M, Ali E, et al. Harnessing the power of ChatGPT for automating systematic review process: methodol- ogy, case study, limitations, and future directions. Systems. 2023;11:351. 116. Guler N, Kirshner S, Vidgen R. Artificial intelligence research in business and management: a literature review leveraging machine learning and large language models. SSRN J. 2023; 117.
  27. Lam MS, Teoh J, Landay JA, Heer J, Bernstein MS. Concept induction: analyzing unstructured text with high-level concepts using LLooM. In: Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems. 2024:1-28. 118. Platt M, Platt D. Effectiveness of generative artificial intelligence for scientific content analysis. 2023. Accessed April 24, 2025. https://doi.org/10.1109/AICT59525.2023.10313167 119. Raja H, Munawar A, Mylonas N, et al. Automated category and trend analysis of scientific articles on ophthalmology using large language models: development and usability study. JMIR Form Res. 2024;8:e52462. 120. Twinomurinzi H, Gumbo S. ChatGPT in scholarly discourse: sentiments and an inflection point. 2023. Accessed April 24, 2025. https://doi.org/10.1007/978-3-031-39652-6_17 121. · DOI
  28. Wang Q, Liao J, Lapata M, et al. Risk of bias assessment in pre- clinical literature using natural language processing. Res Synth Methods. 2022;13:368-380. 122. Lai H, Ge L, Sun M, et al. Assessing the risk of bias in random- ized clinical trials with large language models. JAMA Netw Open. 2024;7:e2412687. 123.
  29. Woelfle T, Hirt J, Janiaud P, Kappos L, Ioannidis JPA, Hemkens LG. Benchmarking human-AI collaboration for common evi- dence appraisal tools. medRxiv 2024.04.21.24306137, 2024. 124. Barsby J, Hume S, Lemmey HA, Cutteridge J, Lee R, Bera KD. Pilot study on large language models for risk-of-bias assessments in systematic reviews: A(I) new type of bias? BMJ Evid Based Med. 2025;30:71-74. 125. Chern IC, Chern S, Chen S, et al. 2023. FacTool: factuality detec- tion in generative AI-a tool augmented framework for multi- task and multi-domain scenarios. arXiv, preprint arXiv, not peer reviewed. 126.
  30. Hasan B, Saadi S, Rajjoub NS, et al. Integrating large language models in systematic reviews: a framework and case study using ROBINS-I for risk of bias assessment. BMJ Evid Based Med. 2024;29:394-398. 127. Pitre T, Jassal T, Talukdar JR, Shahab M, Ling M, Zeraatkar D. ChatGPT for assessing risk of bias of randomized trials using the RoB 2.0 tool: a methods study. medRxiv 2023.11.19.23298727, 2024. 128. Roberts RH, Ali SR, Hutchings HA, et al. Comparative study of ChatGPT and human evaluators on the assessment of medical lit- erature according to recognised reporting standards. BMJ Health Care Inform. 2023;30:e100830. 129. Srivastava M. A day in the life of ChatGPT as an academic reviewer: investigating the potential of large language model for scientific literature review. OSF Preprints, 2023. 130. Trevino-Juarez AS. Assessing risk of bias using ChatGPT-4 and Cochrane ROB2 Tool. Med Sci Educ. 2024;34:691-694. 131. Alchokr R, Borkar M, M, Thotadarya S, Saake G, Leich T. Sup- porting systematic literature reviews using deep-learning-based language models. 2022. Accessed April 24, 2025. https://doi.org/ 10.1145/3528588.3528658 132. Lu ZH, Wang JX, Li X. Revealing opinions for COVID-19 ques- tions using a context retriever, opinion aggregator, and question- answering model: model development study. J Med Internet Res. 2021;23:e22860. 133. Tang F-SK-B, Bukowski M, Schmitz-Rode T, et al. Guidance for clinical evaluation under the medical device regulation through automated scoping searches. Appl Sci. 2023;13:7639. 134. Aiumtrakul N, Thongprayoon C, Suppadungsuk S, et al. Navi- gating the landscape of personalized medicine: the relevance of ChatGPT, BingChat, and Bard AI in Nephrology literature searches. J Pers Med. 2023;13:1457. 135. Chelli M, Descamps J, Lavou� e V, et al. Hallucination rates and reference accuracy of ChatGPT and Bard for systematic reviews: comparative analysis. J Med Internet Res. 2024;26:e53164. 136. · DOI
  31. Antu SA, Chen H, Richards CK. Using LLM (large language model) to improve efficiency in literature review for undergradu- ate research. In: LLM@ AIED, Tokyo, Japan. CEUR-WS.org; 2023:8-16. 137. Choueka D, Tabakin AL, Shalom DF. ChatGPT in urogynecol- ogy research: novel or not? Urogynecology (Phila). 2022:10-97. 138. D� ıaz O, Garmendia X, Contell JP, Pereira J. Inquiry frameworks for research question scoping in DSR: a realization for ChatGPT. 2023. Accessed April 24, 2025. https://doi.org/10.1007/978-3- 031-32808-4_19 139. · DOI
  32. Goldfarb N, Tal N, Cohen I-C, et al. Barriers and suggested solu- tions to nursing participation in research: a systematic review with NLP Tools (Preprint). JMIR Preprints, 2024. 140. Gupta R, Herzog I, Weisberger J, et al. Utilization of ChatGPT for plastic surgery research: friend or foe? J Plast Reconstr Aes- thet Surg. 2023;80:145-147. 141. Gupta R, Park JB, Bisht C, et al. Expanding cosmetic plastic sur- gery research with ChatGPT. Aesthet Surg J. 2023;43:930-937. 142. Gwon YN, Kim JH, Chung HS, et al. The use of generative AI for scientific literature searches for systematic reviews: ChatGPT and Microsoft Bing AI performance evaluation. JMIR Med Inform. 2024;12:e51187. 143. Herbst P, Baars H. Accelerating literature screening for system- atic literature reviews with large language models-development, application, and first evaluation of a solution. 2023. Accessed April 24, 2025. https://ceur-ws.org/Vol-3630/LWDA2023- paper4.pdf 144. Jafari SMA. 2024. Streamlining the selection phase of systematic literature reviews (SLRs) using AI-enabled GPT-4 assistant API. arXiv, preprint arXiv, not peer reviewed. 145.
  33. Kim J, Lee J-S, Kim H, Lee T. 2024. Systematic review on healthcare systems engineering utilizing ChatGPT. arXiv, preprint arXiv, not peer reviewed. 146. Li Y, Zhao J, Li M, et al. RefAI: a GPT-powered retrieval-aug- mented generative tool for biomedical literature recommendation and summarization. J Am Med Inform Assoc. 2024;31:2030-2039. 147. Liu Y, Chen S, Cheng H, et al. 2023. CoQuest: exploring research question co-creation with an LLM-based agent. arXiv, preprint arXiv, not peer reviewed. 148.
  34. Maniaci A, Saibene AM, Calvo-Henriquez C, et al. Is generative pre-trained transformer artificial intelligence (Chat-GPT) a reli- able tool for guidelines synthesis? A preliminary evaluation for biologic CRSwNP therapy. Eur Arch Otorhinolaryngol. 2024;281:2167-2173. 149. Roy K, Khandelwal V, Vera V, Surana H, Heckman H, Sheth A. GEAR-Up: generative AI and external knowledge-based retrieval upgrading scholarly article searches for systematic reviews. 2024. Accessed April 24, 2025. https://doi.org/10.1609/aaai.v38i21. 30577 150. · DOI
  35. Ruksakulpiwat S, Phianhasin L, Benjasirisan C, et al. Assessing the efficacy of ChatGPT versus human researchers in identifying relevant studies on mHealth interventions for improving medica- tion adherence in patients with ischemic stroke when conducting systematic reviews: comparative analysis. JMIR Mhealth Uhealth. 2024;12:e51526. 151. Sanii RY, Kasto JK, Wines WB, et al. Utility of artificial intelli- gence in orthopedic surgery literature review: a comparative pilot study. Orthopedics. 2024;47:e125-e130. 152. Singh S, Watson S. ChatGPT as a tool for conducting literature review for dry eye disease. Clin Exp Ophthalmol. 2023;51:731-732. 153. Spillias S, et al. Human-AI collaboration to identify literature for evidence synthesis. Research Square, 2023. Accessed April 24, 2025. https://doi.org/10.21203/rs.3.rs-3099291/v1 154. · DOI
  36. Suppadungsuk S, Thongprayoon C, Krisanapan P, et al. Examin- ing the validity of ChatGPT in identifying relevant nephrology lit- erature: findings and implications. J Clin Med. 2023;12:5550. 155. Wang S, Scells H, Koopman B, Zuccon G. 2023. Can ChatGPT write a good Boolean query for systematic review literature search? arXiv, preprint arXiv, not peer reviewed. 156.
  37. Yan C, Grabowska ME, Dickson AL, et al. Leveraging generative AI to prioritize drug repurposing candidates for Alzheimer's dis- ease with real-world clinical validation. NPJ Digit Med. 2024;7:46. 157. Zhu K, Feng X, Feng X, Wu Y, Qin B. 2023. Hierarchical cata- logue generation for literature review: a benchmark. arXiv, pre- print arXiv, not peer reviewed. 1084 Journal of the American Medical Informatics Association, 2025, Vol. 32, No. 6 158. · DOI
  38. Zimmermann R, Staab M, Nasseri M, Brandtner P. Leveraging large language models for literature review tasks-a case study using ChatGPT. 2024. Accessed April 24, 2025. https://doi.org/ 10.1007/978-3-031-48858-0_25 159. · DOI
  39. Anghelescu A, Firan FC, Onose G, et al. PRISMA systematic liter- ature review, including with Meta-Analysis vs Chatbot/GPT (AI) regarding current scientific data on the main effects of the calf blood deproteinized hemoderivative medicine (Actovegin) in ischemic stroke. Biomedicines. 2023;11:1623 160.
  40. Cambaz D, Zhang X. Use of AI-driven code generation models in teaching and learning programming: a systematic literature review. 2024. Accessed April 24, 2025. https://doi.org/10.1145/ 3626252.3630958 161. Haltaufderheide J, Ranisch R. The ethics of ChatGPT in medicine and healthcare: a systematic review on Large Language Models (LLMs). NPJ Digit Med. 2024;7:183. 162. Ruksakulpiwat S, Kumar A, Ajibade A. Using ChatGPT in medi- cal research: current status and future directions. J Multidiscip Healthc. 2023;16:1513-1520. 163. Sallam M. ChatGPT utility in healthcare education, research, and practice: systematic review on the promising perspectives and valid concerns. Healthcare (Basel). 2023;11:887. 164. Srivastava A. 2023. A rapid scoping review and conceptual anal- ysis of the educational metaverse in the Global South: socio- technical perspectives. arXiv, preprint arXiv, not peer reviewed. 165. · DOI
  41. Zhao P, Zhang X, Cao J, Cheng MM, Yang J, Li X. 2024. A liter- ature review of literature reviews in pattern analysis and machine intelligence. arXiv, preprint arXiv, not peer reviewed. 166.
  42. Lan M, Cheng M, Hoang L, et al. Automatic categorization of self-acknowledged limitations in randomized controlled trial publications. J Biomed Inform. 2024;152:104628. 167. Teslyuk A. The concept of system for automated scientific litera- ture reviews generation. 2020. Accessed April 24, 2025. https:// doi.org/10.1007/978-3-030-50420-5_32 168. · DOI
  43. Aydın € O, Karaarslan E. OpenAI ChatGPT generated literature review: digital twin in healthcare. In: Aydın € O, ed. Emerging Computer Technologies. Vol. 2. Elsevier; 2022:22-31. 169. Blasingame MN, Koonce TY, Williams AM, et al. Evaluating a large language model's ability to answer clinicians' requests for evidence summaries. medRxiv 2024.05.01.24306691, 2024. 170. Chaker SC, Hung Y-C, Saad M, et al. Easing the burden on care- givers -applications of artificial intelligence for physicians and caregivers of children with cleft lip and palate. Cleft Palate Cra- niofac J. 2024. doi: 10.1177/10556656231223596. 171. Lam Hoai XL, Simonart T. Comparing meta-analyses with ChatGPT in the evaluation of the effectiveness and tolerance of systemic therapies in moderate-to-severe plaque psoriasis. J Clin Med. 2023;12:5410. 172. Tang L, Sun Z, Idnay B, et al. Evaluating large language models on medical evidence summarization. NPJ Digit Med. 2023;6:158. 173. Yan C, Grabowska ME, Dickson AL, et al. Leveraging generative AI to prioritize drug repurposing candidates: validating identified candidates for Alzheimer's disease in real-world clinical datasets. medRxiv 2023.07.07.23292388, 2023. 174. Yu B. Evaluating pre-trained language models on multi- document summarization for literature reviews. In: Proceedings of the Third Workshop on Scholarly Document Processing. 2022:188-192. 175. Li Y, Chen L, Liu A, Yu K, Wen L. 2024. ChatCite: LLM agent with human workflow guidance for comparative literature sum- mary. arXiv, preprint arXiv, not peer reviewed. 176. · DOI
  44. Rajjoub R, Arroyave JS, Zaidat B, et al. ChatGPT and its role in the decision-making for the diagnosis and treatment of lumbar spinal stenosis: a comparative analysis and narrative review. Global Spine J. 2024;14:998-1017. 177. Temsah O, Khan SA, Chaiah Y, et al. Overview of early ChatGPT's presence in medical literature: insights from a hybrid literature review by ChatGPT and human experts. Cureus. 2023;15:e37281. 178. Ambalavanan AK, Devarakonda MV. Using the contextual lan- guage model BERT for multi-criteria classification of scientific articles. J Biomed Inform. 2020;112:103578. 179. Aum S, Choe S. srBERT: automatic article classification model for systematic review using BERT. Syst Rev. 2021;10:285. 180. Edwards KM, Song B, Porciello J, et al. ADVISE: accelerating the creation of evidence syntheses for global development using natu- ral language processing-supported human-AI collaboration. J Mech Desig. 2024;146:1-15. 181. Hasny M, Vasile AP, Gianni M, et al. BERT for complex system- atic review screening to support the future of medical research. 2023. Accessed April 24, 2025. https://doi.org/10.1007/978-3- 031-34344-5_21 182. · DOI
  45. Kats T, van der Putten P, Scholtes J. 2023. Relevance feedback strategies for recall-oriented neural information retrieval. arXiv, preprint arXiv, not peer reviewed. 183.
  46. Mao X, Koopman B, Zuccon G. 2024. A reproducibility study of goldilocks: just-right tuning of BERT for TAR. arXiv, preprint arXiv, not peer reviewed. 184.
  47. Ng SH-X, Teow KL, Ang GY, et al. Semi-automating abstract screening with a natural language model pretrained on biomedi- cal literature. Syst Rev. 2023;12:172. 185. Qin X, Liu J, Wang Y, et al. Natural language processing was effective in assisting rapid title and abstract screening when updating systematic reviews. J Clin Epidemiol. 2021;133:121-129. 186. Wang S, Scells H, Koopman B, et al. 2022. Neural rankers for effective screening prioritisation in medical systematic review lit- erature search. arXiv, preprint arXiv, not peer reviewed. 187.
  48. Castillo-Segura P, Alario-Hoyos C, Kloos CD, Fernandez Pana- dero C. Leveraging the potential of generative AI to accelerate systematic literature reviews: an example in the area of educa- tional technology. 2023. Accessed April 24, 2025. https://doi. org/10.1109/WEEF-GEDC59520.2023.10344098 188. Akinseloyin O, Jiang X, Palade V. A novel question-answering framework for automated abstract screening using large language models. medRxiv 2023.12.17.23300102, 2024. 189. Ali F. 2024. Can machine learning help accelerate article screen- ing for systematic reviews? Yes, when article separability in embedding space is high. EdArXiv, preprint EdArXiv, not peer reviewed. 190. · DOI
  49. Cai X, Geng Y, Du Y, et al. Utilizing ChatGPT to select literature for meta-analysis shows workload reduction while maintaining a similar recall level as manual curation. medRxiv 2023.09.06.23295072, 2023. 191. Guo E, Gupta M, Deng J, et al. Automated paper screening for clinical reviews using large language models: data analysis study. J Med Internet Res. 2024;26:e48996. 192.
  50. Huotala A, Kuutila M, Ralph P, M€ antyl€ a M. 2024. The promise and challenges of using LLMs to accelerate the screening process of systematic reviews. arXiv, preprint arXiv, not peer reviewed. 193.
  51. Issaiy M, Ghanaati H, Kolahi S, et al. Methodological insights into ChatGPT's screening performance in systematic reviews. BMC Med Res Methodol. 2024;24:78. 194. Kataoka Y, So R, Banno M, et al. Development of meta-prompts for large language models to screen titles and abstracts for diag- nostic test accuracy reviews. medRxiv 2023.10.31.23297818, 2023. 195. Li M, Sun J, Tan X. Evaluating the effectiveness of large language models in abstract screening: a comparative analysis. Syst Rev. 2024;13:219. 196. Robinson A, Thorne W, Wu BP, et al. 2023. Bio-SIEVE: explor- ing instruction tuning large language models for systematic review automation. arXiv, preprint arXiv, not peer reviewed. Accessed April 24, 2025. https://doi.org/10.48550/arxiv.2308. 06610 197. · DOI
  52. Syriani E, David I, Kumar G. 2023. Assessing the ability of ChatGPT to screen articles for systematic reviews. arXiv, pre- print arXiv, not peer reviewed. Accessed April 24, 2025. https:// doi.org/10.48550/arxiv.2307.06464 198. · DOI · Paper page
  53. Tran V-T, Gartlehner G, Yaacoub S, et al. Sensitivity and specific- ity of using GPT-3.5 Turbo models for title and abstract screen- ing in systematic reviews and meta-analyses. Ann Intern Med. 2024;177:791-799. 199. Wang S, Scells H, Koopman B, Potthast M, Zuccon G. Gen- erating natural language queries for more effective systematic review screening prioritisation. In: Proceedings of the Annual International ACM SIGIR Conference on Research and Development in Information Retrieval in the Asia Pacific Region. 2023:73-83. 200. Wang S, Scells H, Zhuang S, Potthast M, Koopman B, Zuccon G. Zero-shot generative large language models for systematic review screening automation. In: European Conference on Infor- mation Retrieval. Springer; 2024:403-420. 201.
  54. Wilkins D. 2023. Automated title and abstract screening for scoping reviews using the GPT-4 large language model. arXiv, preprint arXiv, not peer reviewed. 202. Yang J, Walker KC, Bekar-Cesaretli AA, et al. Automating bio- medical literature review for rapid drug discovery: leveraging GPT-4 to expedite pandemic response. Int J Med Inform. 2024;189:105500. 203. Buchlak QD, Esmaili N, Bennett C, et al. Natural language proc- essing applications in the clinical neurosciences: a machine learning augmented systematic review. Acta Neurochir Suppl. 2022;134:277-289. 204. Buchlak QD, Clair J, Esmaili N, et al. Clinical outcomes associ- ated with robotic and computer-navigated total knee arthro- plasty: a machine learning-augmented systematic review. Eur J Orthop Surg Traumatol. 2022;32:915-931. 205. Cao C, Sang J, Arora R, et al. Development of prompt templates for large language model-driven screening in systematic reviews. Ann Intern Med. 2025;178:389-401. 206. Agapiou A, Lysandrou V. Interacting with the artificial intelli- gence (AI) language model ChatGPT: a synopsis of Earth obser- vation and remote sensing in archaeology. Heritage. 2023;6:4072-4085. 207. Waffenschmidt S, Knelangen M, Sieben W, et al. Single screening versus conventional double screening for study selection in sys- tematic reviews: a methodological systematic review. BMC Med Res Methodol. 2019;19:132. 208. Orenstrakh MS, Karnalim O, Suarez CA, Liut M. Detecting LLM- generated text in computing education: comparative study for ChatGPT cases. In: 2024 IEEE 48th Annual Computers, Software, and Applications Conference (COMPSAC). IEEE; 2024:121-126. 209. Liang W, Zhang Y, Wu Z, et al. 2024. Mapping the increasing use of LLMS in scientific papers. arXiv, preprint arXiv: 2404.01268, not peer reviewed. 210. · arXiv
  55. Hosseini M, Resnik DB, Holmes KL. The ethics of disclosing the use of artificial intelligence tools in writing scholarly manuscripts. Res Ethics. 2023;19:449-465.
  56. This is an Open Access article distributed under the terms of the Creative Commons Attribution-NonCommercial License (https://creativecommons.org/licenses/by-nc/ 4.0/), which permits non-commercial re-use, distribution, and reproduction in any medium, provided the original work is properly cited. For commercial re-use, please contact reprints@oup.com for reprints and translation rights for can on the article page on our site-for further information @oup.com. Journal of the American Medical Informatics Association, 2025, 32, 1071-1086 https://doi.org/10.1093/jamia/ocaf063 Review 1086 Journal of the American Medical Informatics Association, 2025, Vol. 32, No. 6 · DOI

Cited by

Sorted by citations

Authors
Scherbakov, D., Hubig, N., Jansari, V., Bakumenko, A., & Lenert, L. A.
Affiliations
  • Medical University of South Carolina (US)
  • Clemson University (US)
Corresponding author
Lenert, Leslie A.
ORCID
Venue
arXiv
Publisher
arXiv
Published
2024
Volume
32
Issue
6
Pages
1071-1086
ISSN
1067-5027 (ISSN-L)
DOI
10.48550/arxiv.2409.04600
arXiv
2409.04600
PMC
PMC12089777
OpenAlex
W4410157139
Cited by
156
References
0
Citations by year
2026: 116 · 2025: 39 · 2024: 1
FWCI
29.37
Type
journal article
Language
en
Open access
Licence
Locations
  • Journal of the American Medical Informatics Association (publishedVersion, cc-by-nc)
  • PubMed Central (publishedVersion, cc-by-nc)
  • PubMed (publishedVersion)
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • arXiv (Cornell University) (cc-by)
Funders
  • National Institutes of Health (UL1TR001450)
  • U.S. National Library of Medicine (T15LM013977)
  • National Center for Advancing Translational Sciences (TR001450)
Topics
  • Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
  • Social Sciences › General Social Sciences › Computational and Text Analysis Methods
Keywords
large language models, systematic review automation, ChatGPT, GPT-4o, BERT, data extraction, literature search automation, review methodology
MeSH terms
Systematic Reviews as Topic, Large Language Models, Humans, Programming Languages, Review Literature as Topic, Information Storage and Retrieval

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review