Automated Paper Screening for Clinical Reviews Using Large Language Models: Data Analysis Study
Open accessPublished version of a preprint
The DOI matches its Crossref record; 36 of the 37 references it cites were found in a registry or an index.
Abstract
Background: The systematic review of clinical research papers is a labor-intensive and time-consuming process that often involves the screening of thousands of titles and abstracts. The accuracy and efficiency of this process are critical for the quality of the review and subsequent health care decisions. Traditional methods rely heavily on human reviewers, often requiring a significant investment of time and resources. Objective: This study aims to assess the performance of the OpenAI generative pretrained transformer (GPT) and GPT-4 application programming interfaces (APIs) in accurately and efficiently identifying relevant titles and abstracts from real-world clinical review data sets and comparing their performance against ground truth labeling by 2 independent human reviewers. Methods: We introduce a novel workflow using the Chat GPT and GPT-4 APIs for screening titles and abstracts in clinical reviews. A Python script was created to make calls to the API with the screening criteria in natural language and a corpus of title and abstract data sets filtered by a minimum of 2 human reviewers. We compared the performance of our model against human-reviewed papers across 6 review papers, screening over 24,000 titles and abstracts. Results: Our results show an accuracy of 0.91, a macro F1-score of 0.60, a sensitivity of excluded papers of 0.91, and a sensitivity of included papers of 0.76. The interrater variability between 2 independent human screeners was κ=0.46, and the prevalence and bias-adjusted κ between our proposed methods and the consensus-based human decisions was κ=0.96. On a randomly selected subset of papers, the GPT models demonstrated the ability to provide reasoning for their decisions and corrected their initial decisions upon being asked to explain their reasoning for incorrect classifications. Conclusions: Large language models have the potential to streamline the clinical review process, save valuable time and effort for researchers, and contribute to the overall quality of clinical reviews. By prioritizing the workflow and acting as an aid rather than a replacement for researchers and reviewers, models such as GPT-4 can enhance efficiency and lead to more accurate and reliable conclusions in medical research.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures


Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source doesn’t support the claim as written.
In the citing paper
“This finding is consistent with several other studies (Alshami et al., 2023; Guo et al., 2024; Khraisha et al., 2024; Kohandel Gargari et al., 2024; Matsui et al., 2024; Tran et al., 2024), although a smaller number of studies have reported both high sensitivity and high specificity (Landschaft et al., 2024; Li et al., 2024).”
From Analysis of article screening and data extraction performance by an AI systematic literature review platform (2025) · cites this paper as Guo et al., 2024
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“ChatGPT has high sensitivity and accuracy (Guo et al., 2024).”
From Machine Learning-Assisted Systematic Review: A Case Study in Learning Analytics (2025) · cites this paper as Guo et al., 2024
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“LLMs can automate critical tasks for SLRs, including screening abstracts and full texts, assessing risk of bias and extracting data for use in network meta-analyses with high accuracy, all while significantly reducing the time and costs required by using human review [2, 4–7].”
From Automated Mass Extraction of Over 680,000 PICOs from Clinical Study Abstracts Using Generative AI: A Proof-of-Concept Study. (2024) · cites this paper as [6]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“these LLMs have demonstrated promising performance in general text classification tasks [10, 16, 25]”
From Artificial intelligence for the science of evidence synthesis: how good are AI-powered tools for automatic literature screening? (2025) · cites this paper as [25]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“This may due to the limitations of traditional tools, which often segment text into fixed portions, potentially hindering their ability to capture contextual nuances, particularly in longer text [25].”
From Artificial intelligence for the science of evidence synthesis: how good are AI-powered tools for automatic literature screening? (2025) · cites this paper as [25]
Experimental — accuracy not yet measured
2 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Rayyan-a web and mobile app for systematic reviews.
Ouzzani, M., Hammady, H., Fedorowicz, Z., & Elmagarmid, A. (2016) · Systematic Reviews · doi:10.1186/s13643-016-0384-4 · Cited by 26,092
Cochrane Handbook for Systematic Reviews of Interventions
Higgins, J. P. T., Green, S., & Ben Van Den, A. (2020) · International Coaching Psychology Review · doi:10.53841/bpsicpr.2020.15.2.123 · Cited by 5,670
An open source machine learning framework for efficient and transparent systematic reviews
van de Schoot, R., de Bruin, J., Schram, R., Zahedi, P., de Boer, J., Weijdema, F., Kramer, B., Huijts, M., Hoogerwerf, M., Ferdinands, G., et al. (2021) · Nature Machine Intelligence · doi:10.1038/s42256-020-00287-7 · Cited by 1,157 · cited 1×
“This can be particularly beneficial in situations where a large number of papers need to be screened, and human resources are scarce [25].”
BackgroundO’Mara-Eves, A., Thomas, J., McNaught, J., Miwa, M., & Ananiadou, S. (2015) · Systematic Reviews · doi:10.1186/2046-4053-4-5 · Cited by 681
Marshall, I. J., & Wallace, B. C. (2019) · Systematic Reviews · doi:10.1186/s13643-019-1074-9 · Cited by 638 · cited 1×
“Recent developments in artificial intelligence and machine learning have made it possible to semiautomate or fully automate repetitive steps within the systematic review workflow [12-14].”
What is heterogeneity and is it important?
Fletcher, J. (2007) · BMJ · doi:10.1136/bmj.39057.406644.68 · Cited by 546
Best practice guidelines for abstract screening large‐evidence systematic reviews and meta‐analyses
Polanin, J. R., Pigott, T. D., Espelage, D. L., & Grotpeter, J. K. (2019) · Research Synthesis Methods · doi:10.1002/jrsm.1354 · Cited by 401
O’Mara-Eves, A., Thomas, J., McNaught, J., Miwa, M., & Ananiadou, S. (2015) · Systematic Reviews · doi:10.1186/s13643-015-0031-5 · Cited by 353
Semi-automated screening of biomedical citations for systematic reviews
Wallace, B. C., Trikalinos, T. A., Lau, J., Brodley, C., & Schmid, C. H. (2010) · BMC Bioinformatics · doi:10.1186/1471-2105-11-55 · Cited by 351 · cited 2×
“While automation using these tools was found to save time, there was still minimal to substantive risk that there would be missing studies if the tool were fully independent or automated [22,23].”
RobotReviewer: evaluation of a system for automatically assessing bias in clinical trials
Marshall, I. J., Kuiper, J., & Wallace, B. C. (2016) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocv044 · Cited by 317
Research works printed but not linked: 1
- Gates, A, 2019, AHRQ Methods for Effective Health Care
Cited by
Khraisha, Q., Put, S., Kappenberg, J., Warraitch, A., & Hadfield, K. (2024) · Research Synthesis Methods · doi:10.1002/jrsm.1715 · Cited by 269
“%) was lower than in previous work, 16,22 possibly because our study was the only one that artificially balanced its dataset.”
InfluentialDennstädt, F., Zink, J., Putora, P. M., Hastings, J., & Cihoric, N. (2024) · Systematic Reviews · doi:10.1186/s13643-024-02575-4 · Cited by 145
Scherbakov, D., Hubig, N., Jansari, V., Bakumenko, A., & Lenert, L. A. (2024) · arXiv · doi:10.48550/arxiv.2409.04600 · Cited by 156
Scherbakov, D., Hubig, N., Jansari, V., Bakumenko, A., & Lenert, L. A. (2025) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocaf063 · Cited by 154
Potential Roles of Large Language Models in the Production of Systematic Reviews and Meta-Analyses
Luo, X., Chen, F., Zhu, D., Wang, L., Wang, Z., Liu, H., Lyu, M., Wang, Y., Wang, Q., & Chen, Y. (2024) · Journal of Medical Internet Research · doi:10.2196/56780 · Cited by 109
Development of Prompt Templates for Large Language Model–Driven Screening in Systematic Reviews
Cao, C., Sang, J., Arora, R., Chen, D., Kloosterman, R., Cecere, M., Gorla, J., Saleh, R., Drennan, I., Teja, B., et al. (2025) · Annals of Internal Medicine · doi:10.7326/annals-24-02189 · Cited by 91
“However, previous LLM assessments for SR screening relied on basic zero-shot prompting — where LLMs are only given task instructions without examples — and demonstrated low sensitivity and recall or speci fi city that precludes their use (12 – 16).”
Generative artificial intelligence use in evidence synthesis: A systematic review
Clark, J., Barton, B., Albarqouni, L., Byambasuren, O., Jowsey, T., Keogh, J., Liang, T., Moro, C., O’Neill, H., & Jones, M. (2025) · Research Synthesis Methods · doi:10.1017/rsm.2025.16 · Cited by 90
Generative artificial intelligence in graduate medical education
Janumpally, R., Nanua, S., Ngo, A., & Youens, K. (2025) · Frontiers in Medicine · doi:10.3389/fmed.2024.1525604 · Cited by 54
Li, Y., Datta, S., Rastegar-Mojarad, M., Lee, K., Paek, H., Glasgow, J., Liston, C., He, L., Wang, X., & Xu, Y. (2025) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocaf030 · Cited by 69
Tran, V.-T., Gartlehner, G., Yaacoub, S., Boutron, I., Schwingshackl, L., Stadelmaier, J., Sommer, I., Alebouyeh, F., Afach, S., Meerpohl, J., et al. (2024) · Annals of Internal Medicine · doi:10.7326/m23-3389 · Cited by 66
“Of note, another article describing the performance of GPT-4 for title and abstract screening showed performances similar to ours (21).”
ResultLarge Language Models for Health Care Text Classification: Systematic Review
Sakai, H., & Lam, S. S. (2025) · JMIR AI · doi:10.2196/79202 · Cited by 72
Generative AI as a tool to accelerate the field of ecology
Rafiq, K., Beery, S., Palmer, M. S., Harchaoui, Z., & Abrahms, B. L. (2025) · Nature Ecology & Evolution · doi:10.1038/s41559-024-02623-1 · Cited by 66
Trad, F. S., Yammine, R., Charafeddine, J., Chakhtoura, M., Rahme, M., Fuleihan, G. E., & Chehab, A. (2025) · BMC Medical Research Methodology · doi:10.1186/s12874-025-02583-5 · Cited by 62
Fleurence, R. L., Bian, J., Wang, X., Xu, H., Dawoud, D., Higashi, M., & Chhatwal, J. (2025) · Value in Health · doi:10.1016/j.jval.2024.10.3846 · Cited by 54
Analyzing evaluation methods for large language models in the medical field: a scoping review
Lee, J., Park, S., Shin, J., & Cho, B. (2024) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-024-02709-7 · Cited by 52
Matsui, K., Utsumi, T., Aoki, Y., Maruki, T., Takeshima, M., & Takaesu, Y. (2024) · Journal of Medical Internet Research · doi:10.2196/52758 · Cited by 50
Generative Artificial Intelligence in Anatomic Pathology
Brodsky, V., Ullah, E., Bychkov, A., Song, A. H., Walk, E. E., Louis, P., Rasool, G., Singh, R. S., Mahmood, F., Bui, M. M., et al. (2025) · Archives of Pathology & Laboratory Medicine · doi:10.5858/arpa.2024-0215-ra · Cited by 48
Enhancing title and abstract screening for systematic reviews with GPT-3.5 turbo
Gargari, O. K., Mahmoudi, M. H., Hajisafarali, M., & Samiee, R. (2023) · BMJ Evidence-Based Medicine · doi:10.1136/bmjebm-2023-112678 · Cited by 46
The Promise and Challenges of Using LLMs to Accelerate the Screening Process of Systematic Reviews
Huotala, A., Kuutila, M., Ralph, P., & Mäntylä, M. (2024) · International Conference on Evaluation and Assessment in Software Engineering (EASE) · doi:10.1145/3661167.3661172 · Cited by 64
Landschaft, A., Antweiler, D., Mackay, S., Kugler, S., Rüping, S., Wrobel, S., Höres, T., & Allende-Cid, H. (2024) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2024.105531 · Cited by 41
A question-answering framework for automated abstract screening using large language models
Akinseloyin, O., Jiang, X., & Palade, V. (2024) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocae166 · Cited by 28
Generative Artificial Intelligence in Medicine
Shanmugam, D., Agrawal, M., Movva, R., Chen, I. Y., Ghassemi, M., Jacobs, M. L., & Pierson, E. (2025) · Annual Review of Biomedical Data Science · doi:10.1146/annurev-biodatasci-103123-095332 · Cited by 40
Ye, A., Maiti, A., Schmidt, M. W., & Pedersen, S. (2024) · Future Internet · doi:10.3390/fi16050167 · Cited by 45
Felizardo, K. R., Lima, M. H. T. D. F., Deizepe, A., Conte, T. U., & Steinmacher, I. (2024) · Proceedings of the ... ACM-IEEE International Symposium on Empirical Software Engineering and Measurement/International Symposium on Empirical Software Engineering and Measurement · doi:10.1145/3674805.3686666 · Cited by 50
Homiar, A., Thomas, J., Ostinelli, E. G., Kennett, J., Friedrich, C., Cuijpers, P. J., Harrer, M., Leucht, S., Miguel, C., Rodolico, A., et al. (2025) · BMJ Mental Health · doi:10.1136/bmjment-2025-301762 · Cited by 44
Yoon, D., Han, C., Kim, D. W., Kim, S., Bae, S., Ryu, J. A., & Choi, Y. (2024) · Journal of Medical Internet Research · doi:10.2196/56614 · Cited by 42
Large language models for automated scholarly paper review: A survey
Zhuang, Z., Chen, J., Xu, H., Jiang, Y., & Lin, J. (2025) · Information Fusion · doi:10.1016/j.inffus.2025.103332 · Cited by 34
Artificial intelligence in food and nutrition evidence: The challenges and opportunities
Bailey, R. L., MacFarlane, A. J., Field, M. S., Tagkopoulos, I., Baranzini, S. E., Edwards, K. M., Rose, C. J., Schork, N. J., Singhal, A., Wallace, B., et al. (2024) · PNAS Nexus · doi:10.1093/pnasnexus/pgae461 · Cited by 36
Purewal, A., Fautsch, K. J., Klasová, J., Hussain, N., & D'Souza, R. S. (2025) · Regional Anesthesia & Pain Medicine · doi:10.1136/rapm-2024-106358 · Cited by 34
Fleurence, R., Dawoud, D., Bian, J., Higashi, M. K., Wang, X., Xu, H., Chhatwal, J., & Ayer, T. (2025) · Value in Health · doi:10.1016/j.jval.2025.06.018 · Cited by 41
Can large language models fully automate or partially assist paper selection in systematic reviews?
Chen, H., Jiang, Z., Liu, X., Xue, C. C., Yew, S. M. E., Sheng, B., Zheng, Y., Wang, X., Wu, Y., Sivaprasad, S., et al. (2025) · British Journal of Ophthalmology · doi:10.1136/bjo-2024-326254 · Cited by 33
Riaz, I. B., Naqvi, S. A. A., Hasan, B., & Murad, M. H. (2024) · Mayo Clinic Proceedings Digital Health · doi:10.1016/j.mcpdig.2024.05.023 · Cited by 31
On the Use of Generative AI for Literature Reviews: An Exploration of Tools and Techniques
Mozelius, P., & Humble, N. (2024) · European Conference on Research Methodology for Business and Management Studies · doi:10.34190/ecrm.23.1.2528 · Cited by 38
Tao, K., Osman, Z. A., Tzou, P. L., Rhee, S., Ahluwalia, V., & Shafer, R. W. (2024) · BMC Medical Research Methodology · doi:10.1186/s12874-024-02253-y · Cited by 35
ASReview LAB v.2: Open-source text screening with multiple agents and a crowd of experts
de Bruin, J., Lombaers, P., Kaandorp, C., Teijema, J., van der Kuil, T., Yazan, B., Dong, A., & van de Schoot, R. (2025) · Patterns · doi:10.1016/j.patter.2025.101318 · Cited by 27
Reason, T., Klijn, S. L., Rawlinson, W. D., Benbow, E., Langham, J., Teitsson, S., Johannesen, K. M., & Malcolm, B. (2025) · PharmacoEconomics - Open · doi:10.1007/s41669-025-00580-4 · Cited by 26
Toward Automated Simulation Research Workflow through LLM Prompt Engineering Design
Liu, Z., Chai, Y., & Li, J. (2025) · Journal of Chemical Information and Modeling · doi:10.1021/acs.jcim.4c01653 · Cited by 24
Optimal large language models to screen citations for systematic reviews
Oami, T., Okada, Y., & Nakada, T. (2025) · Research Synthesis Methods · doi:10.1017/rsm.2025.10014 · Cited by 29
Lee, K., Paek, H., Ofoegbu, N., Rube, S., Higashi, M. K., Dawoud, D., Xu, H., Shi, L., & Wang, X. (2025) · Value in Health · doi:10.1016/j.jval.2025.08.002 · Cited by 23
Rashid, M., Yi, C. S., Sathapanasiri, T., Udayachalerm, S., Boonpattharatthiti, K., Insuk, S., Veettil, S. K., Lai, N. M., Chaiyakunapruk, N., Dhippayom, T., et al. (2025) · Value in Health · doi:10.1016/j.jval.2025.07.001 · Cited by 25
Bio-SIEVE: Exploring Instruction Tuning Large Language Models for Systematic Review Automation
Robinson, A., Thorne, W., Wu, B. P., Pandor, A., Essat, M., Stevenson, M., & Song, X. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2308.06610 · Cited by 31
Bayani, A., Ewane, L. P. E., dos Anjos, D. S. O., Mac-Seing, M., & Nikiema, J. N. (2025) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2025.106048 · Cited by 23
Automated title and abstract screening for scoping reviews using the GPT-4 Large Language ModelPreprint
Wilkins, D. E. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2311.07918 · Cited by 24
Hilkenmeier, F., Pelzer, M., Stierle, C. M. G., & Fink‐Lamotte, J. (2025) · Social Science Computer Review · doi:10.1177/08944393251404052 · Cited by 22
Nykvist, B., Macura, B., Xylia, M., & Olsson, E. (2025) · Environmental Evidence · doi:10.1186/s13750-025-00360-x · Cited by 17
Reason, T., Langham, J., & Gimblett, A. (2024) · Pharmaceutical Medicine · doi:10.1007/s40290-024-00539-6 · Cited by 16
Artificial intelligence in clinical practice: A look at ChatGPT
Deng, J., Heybati, K., Park, Y., Zhou, F., & Bozzo, A. (2024) · Cleveland Clinic Journal of Medicine · doi:10.3949/ccjm.91a.23070 · Cited by 20
Self-citationChatGPT and neurosurgical education: A crossroads of innovation and opportunity
Arfaie, S., Mashayekhi, M. S., Mofatteh, M., Ma, C., Ruan, R., MacLean, M. A., Far, R., Saini, J., Harmsen, I. E., Duda, T., et al. (2024) · Journal of Clinical Neuroscience · doi:10.1016/j.jocn.2024.110815 · Cited by 17
Self-citationOn the road to interactive LLM-based systematic mapping studies
Petersen, K., & Gerken, J. M. (2024) · Information and Software Technology · doi:10.1016/j.infsof.2024.107611 · Cited by 16
How Well Do ChatGPT and Claude Perform in Study Selection for Systematic Review in Obstetrics
Insuk, S., Boonpattharatthiti, K., Booncharoen, C., Chaipitak, P., Rashid, M., Veettil, S. K., Lai, N. M., Chaiyakunapruk, N., & Dhippayom, T. (2025) · Journal of Medical Systems · doi:10.1007/s10916-025-02246-4 · Cited by 15
Jev in Medicine: A Benchmark EvaluationPreprint
Madrid-García, A., & Barbancho, B. M. (2026) · arXiv (Cornell University) · doi:10.5281/zenodo.23008155 · Cited by 16
Madeyski, L., Kitchenham, B. A., & Shepperd, M. (2026) · Information and Software Technology · doi:10.1016/j.infsof.2026.108204 · Cited by 15
Gartlehner, G., Nußbaumer-Streit, B., Hamel, C., Garritty, C., Griebler, U., King, V. J., Devane, D., & Kamel, C. (2025) · Cochrane Evidence Synthesis and Methods · doi:10.1002/cesm.70063 · Cited by 14
Gong, E. J., Bang, C. S., & Shin, Y. S. (2026) · Bioengineering · doi:10.3390/bioengineering13030365 · Cited by 19
Medical Implications of LLM Based Clinical Decision Support Systems in Healthcare
Ogdu, C. U., Gürbüz, S., Karaköse, M., & Hanoğlu, E. (2025) · doi:10.1109/it64745.2025.10930252 · Cited by 18
Do it faster with PICOS: Generative AI-Assisted systematic review screening
Vallamchetla, S. K., Abdelkader, O., Elnaggar, A., Ramadan, D., Islam Shourav, M. M., Riaz, I. B., & Lin, M. P. (2025) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2025.104860 · Cited by 13
SESR-Eval: Dataset for Evaluating LLMs in the Title-Abstract Screening of Systematic Reviews
Huotala, A., Kuutila, M., & Mäntylä, M. (2025) · Proceedings of the ... ACM-IEEE International Symposium on Empirical Software Engineering and Measurement/International Symposium on Empirical Software Engineering and Measurement · doi:10.1109/esem64174.2025.00053 · Cited by 19
Sousa, M. S. A., Peiris, S., Figueiró, M. F., Haby, M. M., Baraldi, A. C., Revéiz, L., & Souza, J. P. (2026) · Systematic Reviews · doi:10.1186/s13643-025-02842-y · Cited by 13
Fleurence, R., Wang, X., Bian, J., Higashi, M. K., Ayer, T., Xu, H., Dawoud, D., & Chhatwal, J. (2025) · Value in Health · doi:10.1016/j.jval.2025.04.2167 · Cited by 21
Ruan, M., Fan, J., Liu, M., Meng, Z., Zhang, X., & Zhang, C. (2025) · BMC Medical Research Methodology · doi:10.1186/s12874-025-02644-9 · Cited by 12
“This may due to the limitations of traditional tools, which often segment text into fixed portions, potentially hindering their ability to capture contextual nuances, particularly in longer text [25].”
How to Write Effective Prompts for Screening Biomedical Literature Using Large Language Models
Colangelo, M. T., Guizzardi, S., Meleti, M., Calciolari, E., & Galli, C. (2025) · BioMedInformatics · doi:10.3390/biomedinformatics5010015 · Cited by 14
Enhancing AI for citation screening in literature reviews: Improving accuracy with ensemble models
Zhang, Z., Nezhad, M. J. M., Gupta, P., Zolnour, A., Azadmaleki, H., Topaz, M., & Zolnoori, M. (2025) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2025.106035 · Cited by 14
Sciurti, A., Migliara, G., Siena, L. M., Isonne, C., Blasiis, M. R. D., Sinopoli, A., Iera, J., Marzuillo, C., Vito, C. D., Villari, P., et al. (2025) · Research Synthesis Methods · doi:10.1017/rsm.2025.10044 · Cited by 14
Evaluating the Efficacy of Large Language Models for Systematic Review and Meta-Analysis ScreeningPreprint
Luo, R., Sastimoglu, Z., Faisal, A. I., & Deen, M. J. (2024) · medRxiv · doi:10.1101/2024.06.03.24308405 · Cited by 17
Krag, C. H., Balschmidt, T., Bruun, F. J., Brejnebøl, M. W., Xu, J. J., Boesen, M., Andersen, M. B., & Müller, F. C. (2024) · medRxiv · doi:10.1101/2024.10.01.24314702 · Cited by 17
Khraisha, Q., Put, S., Kappenberg, J., Warraitch, A., & Hadfield, K. (2023) · arXiv · doi:10.48550/arxiv.2310.17526 · Cited by 268
Use of Large Language Models to Extract Cost-Effectiveness Analysis Data: A Case Study
Gu, X., Zhang, H., Patil, D., Zafarí, Z., Slejko, J. F., & Onukwugha, E. (2025) · Value in Health · doi:10.1016/j.jval.2025.05.008 · Cited by 11
Large language models streamline automated systematic review: A preliminary study
Chen, X., & Xue, Z. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2502.15702 · Cited by 14
The hunt for the last relevant paper: blending the best of humans and AI
van de Schoot, R., Messina Coimbra, B., Evenhuis, T., Lombaers, P., Weijdema, F., de Bruin, L., Neeleman, R., Grandfield, E., Sijbrandij, M., Teijema, J. J., et al. (2025) · European Journal of Psychotraumatology · doi:10.1080/20008066.2025.2546214 · Cited by 6
OSCEai: personalized interactive learning for undergraduate medical education
Guo, E., Ramchandani, R., Park, Y., & Gupta, M. (2024) · Canadian Medical Education Journal · doi:10.36834/cmej.79220 · Cited by 16
Self-citationLarge language model-generated clinical practice guideline for appendicitis
Boyle, A., Huo, B., Sylla, P. A., Calabrese, E., Kumar, S. S., Slater, B. J., Walsh, D. S., & Vosburg, R. W. (2025) · Surgical Endoscopy · doi:10.1007/s00464-025-11723-3 · Cited by 12
Large language models for the screening step in systematic reviews in dentistry
Rokhshad, R., Bagherianlemraski, M., Ehsani, S., Haghighat, S., & Schwendicke, F. (2025) · Journal of Dentistry · doi:10.1016/j.jdent.2025.105877 · Cited by 13
Vembye, M. H., Christensen, J., Bondebjerg, A., & Schytt, F. L. W. (2025) · Psychological Methods · doi:10.1037/met0000769 · Cited by 10
Graumann, O., Xin, W. C., Goudie, A., Blaivas, M., Braden, B., Westerway, S. C., Chammas, M. C., Dong, Y., Gilja, O. H., Hsieh, P. C.-C., et al. (2025) · Ultrasound in Medicine & Biology · doi:10.1016/j.ultrasmedbio.2025.07.008 · Cited by 13
Salehin, I., Sajib, M. T. A., Badhon, N. H., Rifat, M. S. H., Amin, N., & Moon, N. N. (2025) · Engineering Reports · doi:10.1002/eng2.70365 · Cited by 13
Akinseloyin, O., Jiang, X., & Palade, V. (2026) · Biology Methods and Protocols · doi:10.1093/biomethods/bpag006 · Cited by 14
Li, Y., Luan, Z., Liu, Y., Liu, H., Qi, J., & Han, D. (2024) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2024.1454945 · Cited by 9
“From knowledge engineering to machine learning (ML) and deep learning (DL) algorithms, Information extraction (IE) technology (Chen et al., 2021; Guo et al., 2024), which commonly solves sequence annotation tasks such as named entity recognition, has rapidly developed into natural language…”
BackgroundRahman, M. A., & Al-Hazzaa, S. A. F. (2024) · Globecom · doi:10.1109/globecom52923.2024.10901624 · Cited by 11
Xu, S., Zhao, Z., Liu, X., & Meng, X.-L. (2025) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-025-03138-w · Cited by 8
Rüland, A., Andersen, L. H., Hassen, A. K., Kinyanjui, C., Ralfs, A., & Grisci, B. I. (2025) · Scientometrics · doi:10.1007/s11192-025-05396-x · Cited by 9
RelevAI-Reviewer: A Benchmark on AI Reviewers for Survey Paper RelevancePreprint
Couto, P. H., Ho, Q. P., Kumari, N., Rachmat, B. K., Khuong, T. G. H., Ullah, I., & Sun-Hosoya, L. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2406.10294 · Cited by 9
From statistics to deep learning: Using large language models in psychiatric research
Hua, Y., Beam, A., Chibnik, L. B., & Torous, J. (2025) · International Journal of Methods in Psychiatric Research · doi:10.1002/mpr.70007 · Cited by 8
Yu, Z., Fang, L., Ding, Y., Yan, S., Xu, L., Cai, Y., & Liu, X. (2025) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2025.104844 · Cited by 8
Review of Water Use Assessment in Livestock Production Systems and Supply Chains
Drastig, K., & Singh, R. (2025) · Water · doi:10.3390/w17192819 · Cited by 8
Artificial intelligence for clinical trial design, conduct, and analysis: a narrative review
Knapen, D. G., van Kruchten, M., de Groot, D. J. A., Broekman, K. E., & Fehrmann, R. S. N. (2026) · ESMO Real World Data and Digital Oncology · doi:10.1016/j.esmorw.2026.100682 · Cited by 13
Treviño, M., & Arias-Carrión, Ó. (2025) · Preprints.org · doi:10.20944/preprints202506.1895.v1 · Cited by 10
Ramchandani, R., Guo, E., Rakab, E., Rathod, J., Strain, J., Klement, W., Shorr, R., Williams, E., Jones, D., & Gilbert, S. (2025) · PeerJ Computer Science · doi:10.7717/peerj-cs.2822 · Cited by 8
Self-citationAsadi, A. R., Zhang, Y., & Said, H. (2025) · Journal of Ambient Intelligence and Humanized Computing · doi:10.1007/s12652-025-05007-w · Cited by 7
Pratte, M., Thirukumar, S., Zhang, C., Slessarev, M., Basmaji, J., & Prager, R. T. (2025) · Journal of Critical Care · doi:10.1016/j.jcrc.2025.155358 · Cited by 8
Hilkenmeier, F., Stoltenberg, M., & Stierle, C. M. G. (2026) · Systematic Reviews · doi:10.1186/s13643-026-03228-4 · Cited by 8
Park, K. S., & Choi, H. (2024) · Nuclear Medicine and Molecular Imaging · doi:10.1007/s13139-024-00876-z · Cited by 7
Cassell, K., Ologunowa, A., Rastegar-Mojarad, M., Chun, B., Huang, Y.-L., Wang, D., & Cossrow, N. (2025) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2025.1662202 · Cited by 15
“These human errors were likely due to reviewer fatigue, highlighting a potential benefit of automation in terms of better reproducibility and lower risk of bias (Alshami et al., 2023; Guo et al., 2024; Matsui et al., 2024).”
InfluentialSujau, M., Wada, M., Vallée, É., Hillis, N., & Sušnjak, T. (2025) · Machine Learning and Knowledge Extraction · doi:10.3390/make7020028 · Cited by 9
Pan, C., Lu, W., Chen, B., Zhang, G., Yang, Z., & Hao, J. (2025) · JMIR Medical Informatics · doi:10.2196/76252 · Cited by 12
Han, B., Mathrani, A., Wanniarachchi, V. U., & Sušnjak, T. (2026) · Research Synthesis Methods · doi:10.1017/rsm.2026.10110 · Cited by 9
Bentegeac, R., Guellec, B. L., Leblanc, V., Lenain, R., Dauchet, L., Gauthier, V., Gerard, E., Chazard, E., Amouyel, P., Aymes, E., et al. (2026) · Research Synthesis Methods · doi:10.1017/rsm.2026.10079 · Cited by 8
Kwon, C. (2025) · Integrative Medicine Research · doi:10.1016/j.imr.2025.101217 · Cited by 6
Sandner, E., Hu, B., Simiceanu, A., Fontana, L., Jakovljevic, I., Henriques, A., Wagner, A., & Gütl, C. (2024) · International Conference on Foundation and Large Language Models (FLLM) · doi:10.1109/fllm63129.2024.10852425 · Cited by 6
LLM-Guided Weak Supervision Improves Active Learning for Abstract ScreeningPreprint
Akinseloyin, O., Jiang, X., & Palade, V. (2025) · medRxiv · doi:10.1101/2025.08.24.25334314 · Cited by 7
Sugiura, A., Saegusa, S., Jin, Y., Yoshimoto, R., Smith, N. D., Dohi, K., Higuchi, T., & Kozu, T. (2024) · JMIR Formative Research · doi:10.2196/55827 · Cited by 5
Metadata
- Authors
- Guo, E., Gupta, M., Deng, J., Park, Y.-J., Paget, M., & Naugler, C.
- Affiliations
- University of Calgary (CA)
- University of Toronto (CA)
- Corresponding author
- Guo, Eddie
- Venue
- Journal of Medical Internet Research
- Publisher
- JMIR Publications Inc.
- Published
- 2024-01-12
- Volume
- 26
- Pages
- e48996
- ISSN
- 1438-8871 (ISSN-L)
- PMC
- PMC10818236
- OpenAlex
- W4387144848
- Cited by
- 205
- References
- 37
- Citations by year
- 2026: 65 · 2025: 99 · 2024: 37
- FWCI
- 38.15
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- Locations
- Journal of Medical Internet Research (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- arXiv (Cornell University) (submittedVersion, cc-by-nc-nd)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- arXiv (Cornell University)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
- Keywords
- large language models, GPT-4, ChatGPT, systematic review screening, title and abstract screening, interobserver variability, human-AI comparison, AI-assisted literature review
- MeSH terms
- Systematic Reviews as Topic, Data Analysis, Artificial Intelligence, Humans, Language, Natural Language Processing, Problem Solving, Consensus, Biomedical Research, Workflow
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.