Automated Paper Screening for Clinical Reviews Using Large Language Models: Data Analysis Study
Open accessPublished version of a preprint
The DOI matches its Crossref record; 36 of the 37 references it cites were found in a registry or an index.
Abstract
Background: The systematic review of clinical research papers is a labor-intensive and time-consuming process that often involves the screening of thousands of titles and abstracts. The accuracy and efficiency of this process are critical for the quality of the review and subsequent health care decisions. Traditional methods rely heavily on human reviewers, often requiring a significant investment of time and resources. Objective: This study aims to assess the performance of the OpenAI generative pretrained transformer (GPT) and GPT-4 application programming interfaces (APIs) in accurately and efficiently identifying relevant titles and abstracts from real-world clinical review data sets and comparing their performance against ground truth labeling by 2 independent human reviewers. Methods: We introduce a novel workflow using the Chat GPT and GPT-4 APIs for screening titles and abstracts in clinical reviews. A Python script was created to make calls to the API with the screening criteria in natural language and a corpus of title and abstract data sets filtered by a minimum of 2 human reviewers. We compared the performance of our model against human-reviewed papers across 6 review papers, screening over 24,000 titles and abstracts. Results: Our results show an accuracy of 0.91, a macro F1-score of 0.60, a sensitivity of excluded papers of 0.91, and a sensitivity of included papers of 0.76. The interrater variability between 2 independent human screeners was κ=0.46, and the prevalence and bias-adjusted κ between our proposed methods and the consensus-based human decisions was κ=0.96. On a randomly selected subset of papers, the GPT models demonstrated the ability to provide reasoning for their decisions and corrected their initial decisions upon being asked to explain their reasoning for incorrect classifications. Conclusions: Large language models have the potential to streamline the clinical review process, save valuable time and effort for researchers, and contribute to the overall quality of clinical reviews. By prioritizing the workflow and acting as an aid rather than a replacement for researchers and reviewers, models such as GPT-4 can enhance efficiency and lead to more accurate and reliable conclusions in medical research.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures


Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source doesn’t support the claim as written.
In the citing paper
“This finding is consistent with several other studies (Alshami et al., 2023; Guo et al., 2024; Khraisha et al., 2024; Kohandel Gargari et al., 2024; Matsui et al., 2024; Tran et al., 2024), although a smaller number of studies have reported both high sensitivity and high specificity (Landschaft et al., 2024; Li et al., 2024).”
From Analysis of article screening and data extraction performance by an AI systematic literature review platform (2025) · cites this paper as Guo et al., 2024
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“ChatGPT has high sensitivity and accuracy (Guo et al., 2024).”
From Machine Learning-Assisted Systematic Review: A Case Study in Learning Analytics (2025) · cites this paper as Guo et al., 2024
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“LLMs can automate critical tasks for SLRs, including screening abstracts and full texts, assessing risk of bias and extracting data for use in network meta-analyses with high accuracy, all while significantly reducing the time and costs required by using human review [2, 4–7].”
From Automated Mass Extraction of Over 680,000 PICOs from Clinical Study Abstracts Using Generative AI: A Proof-of-Concept Study. (2024) · cites this paper as [6]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“these LLMs have demonstrated promising performance in general text classification tasks [10, 16, 25]”
From Artificial intelligence for the science of evidence synthesis: how good are AI-powered tools for automatic literature screening? (2025) · cites this paper as [25]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“This may due to the limitations of traditional tools, which often segment text into fixed portions, potentially hindering their ability to capture contextual nuances, particularly in longer text [25].”
From Artificial intelligence for the science of evidence synthesis: how good are AI-powered tools for automatic literature screening? (2025) · cites this paper as [25]
Experimental — accuracy not yet measured
2 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Rayyan-a web and mobile app for systematic reviews.
Ouzzani, M., Hammady, H., Fedorowicz, Z., & Elmagarmid, A. (2016) · Systematic Reviews · doi:10.1186/s13643-016-0384-4 · Cited by 26,092
Cochrane Handbook for Systematic Reviews of Interventions
Higgins, J. P. T., Green, S., & Ben Van Den, A. (2020) · International Coaching Psychology Review · doi:10.53841/bpsicpr.2020.15.2.123 · Cited by 5,670
An open source machine learning framework for efficient and transparent systematic reviews
van de Schoot, R., de Bruin, J., Schram, R., Zahedi, P., de Boer, J., Weijdema, F., Kramer, B., Huijts, M., Hoogerwerf, M., Ferdinands, G., et al. (2021) · Nature Machine Intelligence · doi:10.1038/s42256-020-00287-7 · Cited by 1,157 · cited 1×
“This can be particularly beneficial in situations where a large number of papers need to be screened, and human resources are scarce [25].”
BackgroundO’Mara-Eves, A., Thomas, J., McNaught, J., Miwa, M., & Ananiadou, S. (2015) · Systematic Reviews · doi:10.1186/2046-4053-4-5 · Cited by 681
Marshall, I. J., & Wallace, B. C. (2019) · Systematic Reviews · doi:10.1186/s13643-019-1074-9 · Cited by 638 · cited 1×
“Recent developments in artificial intelligence and machine learning have made it possible to semiautomate or fully automate repetitive steps within the systematic review workflow [12-14].”
What is heterogeneity and is it important?
Fletcher, J. (2007) · BMJ · doi:10.1136/bmj.39057.406644.68 · Cited by 546
Best practice guidelines for abstract screening large‐evidence systematic reviews and meta‐analyses
Polanin, J. R., Pigott, T. D., Espelage, D. L., & Grotpeter, J. K. (2019) · Research Synthesis Methods · doi:10.1002/jrsm.1354 · Cited by 401
O’Mara-Eves, A., Thomas, J., McNaught, J., Miwa, M., & Ananiadou, S. (2015) · Systematic Reviews · doi:10.1186/s13643-015-0031-5 · Cited by 353
Semi-automated screening of biomedical citations for systematic reviews
Wallace, B. C., Trikalinos, T. A., Lau, J., Brodley, C., & Schmid, C. H. (2010) · BMC Bioinformatics · doi:10.1186/1471-2105-11-55 · Cited by 351 · cited 2×
“While automation using these tools was found to save time, there was still minimal to substantive risk that there would be missing studies if the tool were fully independent or automated [22,23].”
RobotReviewer: evaluation of a system for automatically assessing bias in clinical trials
Marshall, I. J., Kuiper, J., & Wallace, B. C. (2016) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocv044 · Cited by 317
Research works printed but not linked: 1
- Gates, A, 2019, AHRQ Methods for Effective Health Care
Cited by
Items 101–200 of 268
Evaluating Large Language Models for Radiology Systematic Review Title and Abstract Screening
Dogra, S., Arabshahi, S., Wei, J., Hu, E., Saidenberg, L., Sharma, S., Gu, Z., Siriruchatanon, M., & Kang, S. K. (2025) · Academic Radiology · doi:10.1016/j.acra.2025.08.014 · Cited by 5
Doneva, S. E., de Viragh, S., Hubarava, H., Schandelmaier, S., Briel, M., & Ineichen, B. V. (2025) · Research Synthesis Methods · doi:10.1017/rsm.2025.10031 · Cited by 7
Emami, M., & Shirani, M. (2025) · The Journal of the American Dental Association · doi:10.1016/j.adaj.2025.08.011 · Cited by 5
Hu, B., Tomini, E., Corrin, T., Pussegoda, K., Sandner, E., Henriques, A., Simniceanu, A., Fontana, L., Wagner, A., Brazeau, S., et al. (2025) · Cochrane Evidence Synthesis and Methods · doi:10.1002/cesm.70042 · Cited by 7
Participatory modeling in the AI era
Kolagani, N., Glynn, P. D., Voinov, A. A., Quinn, N. W. T., HELGESON, J. F., & Dyckman, C. S. (2025) · Environmental Modelling & Software · doi:10.1016/j.envsoft.2025.106762 · Cited by 5
Gartlehner, G., Banda, S., Callaghan, M., Chase, J. D., Dobrescu, A. I., Eisele‐Metzger, A., Flemyng, E., Gardner, S., Griebler, U., Helfer, B., et al. (2026) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2026.112390 · Cited by 5
Designing GPT3 prompts to screen articles for systematic reviews of RCTs
Strachan, J. A. (2024) · Health Policy and Technology · doi:10.1016/j.hlpt.2024.100943 · Cited by 5
Exploring possibilities and limits of ChatGPT: Usage in building design studies
Çalışkan, E. B. (2025) · Turkish Journal of Engineering · doi:10.31127/tuje.1591495 · Cited by 5
Tinajero, C. A. C. (2025) · European Journal of Pediatric Surgery · doi:10.1055/a-2722-3871 · Cited by 6
Large language model enhanced framework for systematic reviews and meta-analyses
Shen, J., Luo, Z., Jia, D., Wang, S., Sun, F., & Wu, J. (2025) · BMJ Digital Health & AI · doi:10.1136/bmjdhai-2025-000017 · Cited by 8
Zhou, Y., & Hu, H. (2025) · Frontiers in Research Metrics and Analytics · doi:10.3389/frma.2025.1684137 · Cited by 5
Artificial intelligence-based tools for optimizing surgical research publications
Coronel, B. V., Marescaux, J., & Giménez, M. (2026) · Artificial Intelligence Surgery · doi:10.20517/ais.2025.112 · Cited by 4
Song, Z.-H., Huang, S., Thapa, N., Zhang, X., Park, B.-K., Lu, J., Li, W.-Y., Liu, W.-B., Zhan, B., & Li, J.-F. (2026) · Research Synthesis Methods · doi:10.1017/rsm.2026.10094 · Cited by 5
Kertész, G., Czere, J. T., Zrubka, Z., Gulàcsi, L., & Péntek, M. (2026) · JMIR Formative Research · doi:10.2196/86647 · Cited by 4
Machine learning enables automated screening for systematic reviews and meta-analysis in urology
Menold, H. S., Wieland, V. L. S., Haney, C. M., Uysal, D., Wessels, F., Cacciamani, G. E., Michel, M. S., Seide, S., & Kowalewski, K. (2024) · World Journal of Urology · doi:10.1007/s00345-024-05078-y · Cited by 4
Klang, E., Alper, L., Sorin, V., Barash, Y., Nadkarni, G. N., & Zimlichman, E. (2023) · BJR|Open · doi:10.1093/bjro/tzae022 · Cited by 6
AiReview: An Open Platform for Accelerating Systematic Reviews with LLMs
Mao, X., Leelanupab, T., Potthast, M., Scells, H., & Zuccon, G. (2025) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3726302.3730133 · Cited by 4
Use of AI in Family Medicine Publications: A Joint Editorial From Journal Editors
Schrager, S., Seehusen, D. A., Sexton, S. M., Richardson, C. R., Neher, J. O., Pimlott, N., Bowman, M. A., Rodríguez, J. E., Morley, C. P., Li, L., et al. (2025) · Family Medicine · doi:10.22454/fammed.2025.466696 · Cited by 5
Nordmann, K., Schaller, M., Sauter, S., & Fischer, F. (2025) · JAMIA Open · doi:10.1093/jamiaopen/ooaf098 · Cited by 4
Use of AI in family medicine publications: a joint editorial from journal editors
Schrager, S., Seehusen, D. A., Sexton, S. M., Richardson, C. R., Neher, J. O., Pimlott, N., Bowman, M. A., Rodríguez, J. E., Morley, C. P., Li, L., et al. (2025) · Family Medicine and Community Health · doi:10.1136/fmch-2024-003238 · Cited by 5
Nitturi, V., Flores, Á., & Bauer, D. F. (2025) · Neurosurgery · doi:10.1227/neu.0000000000003450 · Cited by 3
Oami, T., Okada, Y., & Nakada, T. (2026) · Hong Kong Journal of Emergency Medicine · doi:10.1002/hkj2.70085 · Cited by 3
AI-driven fungicide design: From target identification to field application
Hu, H., Qu, Z., Liu, Y., Zhu, L., Mei, Z., & Chen, X. (2026) · Plant Communications · doi:10.1016/j.xplc.2026.101850 · Cited by 3
Lee, K., Datta, S., Paek, H., Rastegar-Mojarad, M., Huang, L., He, L., Wang, S., Wang, J., & Wang, X. (2024) · medRxiv · doi:10.1101/2024.07.03.24309897 · Cited by 6
Fine-tuned large language models for answering questions about full-text biomedical research studiesPreprint
Tao, K., Zhou, J., Osman, Z. A., Ahluwalia, V., Sabatti, C., & Shafer, R. W. (2024) · Research Square · doi:10.1101/2024.10.28.24316263 · Cited by 4
Machine Learning-Assisted Systematic Review: A Case Study in Learning Analytics
Xu, Z., Zhuang, X., & Ma, S. (2025) · Education Sciences · doi:10.3390/educsci15111488 · Cited by 2
Pitre, T., Zeraatkar, D., Granton, J., Busse, J., Rochwerg, B., & Guyatt, G. (2026) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2026.112514 · Cited by 2
Lai, H., Ma, N., Zhao, W., Liu, J., Pan, B., Tian, J., Chen, Y., Ma, B., Shang, H., Liu, J., et al. (2025) · Chinese Science Bulletin (Chinese Version) · doi:10.1360/csb-2025-5395 · Cited by 2
Hodgkinson, I., Barth, M., & Dornack, C. (2025) · Sustainability · doi:10.3390/su172310445 · Cited by 2
Ophtimus-V2-Tx: a compact domain-specific LLM for ophthalmic diagnosis and treatment planning
Kwon, M., Jang, K. J., Baek, S. J., Han, Y. S., Choi, H., Lee, I., & Kim, J. H. (2025) · Scientific Reports · doi:10.1038/s41598-025-27410-1 · Cited by 2
CKD-EHR: Clinical Knowledge Distillation for Electronic Health Records
Wang, J., Ling, H., Zhang, L., Zhang, L., Wang, F., Gao, Y., & Li, Z. (2026) · Biomedical Signal Processing and Control · doi:10.1016/j.bspc.2026.110090 · Cited by 4
Chenggong, X., Keying, H., Zhengquan, D., Xinyi, H., & Bin, W. (2026) · Digital Chinese Medicine · doi:10.1016/j.dcmed.2026.02.001 · Cited by 2
TiAb Review Plugin: A Browser-Based Tool for AI-Assisted Study Selection in Systematic Reviews
KATAOKA, Y., Banno, M., Kyo, M., Nakao, S., Sato, T., Taito, S., Takayama, T., Tsuge, T., Tsujimoto, Y., So, R., et al. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2604.08602 · Cited by 2
Comparing Single-Agent and Multi-Agent Strategies in LLM-Based Title-Abstract Screening
Radeva, I., Noncheva, T., Doukovska, L., & Popchev, I. P. (2026) · Electronics · doi:10.3390/electronics15081661 · Cited by 2
Vivekanantha, P., Kahlon, H., Balogun, O. T., Bouchard, M. D., Sa, D., Ayeni, O. R., & Kay, J. (2026) · Knee Surgery Sports Traumatology Arthroscopy · doi:10.1002/ksa.70412 · Cited by 2
Rodriguez, S., León-Prieto, C., & Rodríguez-Jaime, M. F. (2026) · Journal of Bodywork and Movement Therapies · doi:10.1016/j.jbmt.2026.04.008 · Cited by 2
Comparing supervised machine learning and large language models in title-abstract screening
Aigner, M. F., Ganzinger, M., Probst, P., Rinckens, M., & Pausch, T. M. (2026) · Systematic Reviews · doi:10.1186/s13643-026-03199-6 · Cited by 2
Liu, Y., Yang, R., Liew, J. C. K., Yin, Z., Foote, H. P., Lindsell, C. J., & Hong, C. (2026) · Journal of Healthcare Informatics Research · doi:10.1007/s41666-026-00246-8 · Cited by 4
From Research to Practice in Days, not Decades: Why Leaders Must Act now
Peltonen, L., Topaz, M., & Zhang, Z. (2025) · Journal of Medical Systems · doi:10.1007/s10916-025-02305-w · Cited by 2
Evaluating the performance of a custom GPT in full text screening of a systematic review
Davis, R. C., List, S. S., Chappell, K. G., Madar, A., Henjum, S., & Heen, E. (2026) · Scandinavian Journal of Public Health · doi:10.1177/14034948261423410 · Cited by 2
Tao, K., Osman, Z. A., Tzou, P. L., Rhee, S., Ahluwalia, V., & Shafer, R. W. (2024) · Research Square · doi:10.21203/rs.3.rs-3847311/v1 · Cited by 2
Vembye, M. H., Christensen, J., Bondebjerg, A., & Schytt, F. L. W. (2024) · OSF Preprints (OSF Preprints) · doi:10.31219/osf.io/yrhzm · Cited by 2
Chen, Z., Liu, T., Mo, Y., Fu, Q., Lei, S., Tong, T., & Tang, X. (2025) · medRxiv · doi:10.1101/2025.09.18.25335860 · Cited by 2
Operationalising LLM-assisted screening of literature to support systematic reviewsPreprint
Spillias, S., Avila-Turriago, L., Brown, C. J., Easton, A., Roberts, J., Sievers, M., Swearer, S. E., Taylor, A., Wright, B. R., & Komyakova, V. (2026) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.64898/2026.07.29.741127 · Cited by 2
Medical students' use of large language models: a national survey
Barr, A. A., Rozman, R. C., Liu, K., Pham, M., Klarenbach, Z., Chinna-Meyyappan, A., Hassan, A. Y., Zarychta, M., Ferri, O. E., Al-Khaz’Aly, A., et al. (2026) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2026.106589 · Cited by 1
Self-citationCrop phenology data extraction from research papers using a large language model
Iizumi, T., Yohei, O. N. O., Takimoto, T., & CHAOGEJILATU (2025) · Journal of Agricultural Meteorology · doi:10.2480/agrmet.d-24-00042 · Cited by 1
AIDME: A Scalable, Interpretable Framework for AI-Aided Scoping Reviews
Soprano, M., Modha, S., Roitero, K., Maddalena, E., Viviani, M., Pasi, G., & Mizzaro, S. (2025) · ACM SIGIR International Conference on Theory of Information Retrieval (ICTIR) · doi:10.1145/3731120.3744586 · Cited by 1
Cheng, P.-J., Hu, F., Chen, L.-Y., Liu, J.-Y., Wu, J., & Chen, W. (2025) · Taiwan Journal of Ophthalmology · doi:10.4103/tjo.tjo-d-25-00072 · Cited by 1
Nordmann, K., & Fischer, F. (2025) · BMC Health Services Research · doi:10.1186/s12913-025-13901-4 · Cited by 2
Köhler, J., Harl, M. V., Westner, M. K., & Strahringer, S. (2025) · IEEE Conference on Business Informatics · doi:10.1109/cbi68102.2025.00012 · Cited by 1
Sohoni, N., Sohoni, N. S., Sutherland, R., Sundaresan, V. M., Smani, S., Ananth, P. J., Onofrey, J. A., Aneja, S., Miszczyk, M., Lee, H.-J., et al. (2026) · Urology · doi:10.1016/j.urology.2026.01.008 · Cited by 1
Cal-X: Enhancing Systematic Review Screening with LLMs and Next-Token Likelihood Calibration
Sandner, E., Negovetić, M., Kothari, K. U., Taj, I., Fontana, L., Henriques, A., Jakovljevic, I., Simniceanu, A., Wagner, A., & Gütl, C. (2025) · International Conference on Foundation and Large Language Models (FLLM) · doi:10.1109/fllm67465.2025.11391146 · Cited by 1
Machine learning-assisted abstract screening on learning analytics: a step-by-step tutorial
Xu, Z., Ma, S., Zhuang, X., Adeyemi, A. E., & Kogut, A. (2026) · Systematic Reviews · doi:10.1186/s13643-026-03111-2 · Cited by 1
Li, Y., Plasek, J. M., Du, X., Wang, Y., Zhou, Z., Lian, J., Chuang, Y.-W., Hong, P., Hou, P. C., & Zhou, L. (2026) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocag014 · Cited by 2
Zero-shot interpretable biomedical literature appraisal with generative large language models
Zhou, F., Afzal, M., Saha, A., Parrish, R., Haynes, R. B., Iorio, A., & Lokker, C. (2026) · JAMIA Open · doi:10.1093/jamiaopen/ooag043 · Cited by 1
Optimizing document retrieval using massive text embeddings and LLM prompt engineering
Mitrov, G., Stanoev, B., Trajkovik, V., Stojkoska, B. R., Basnarkov, L., Lameski, P., Kampel, M., & Zdravevski, E. (2026) · Systematic Reviews · doi:10.1186/s13643-026-03155-4 · Cited by 1
Chuquimia, G. H., Velásquez, E. A. G., Sánchez-Luján, B.-I., Jiménez, V. B. J., Maldonado, H. I., Lizana, M. A. T., Flores, G. V. G., Moreno, J. L. R., Bardales, Z. E. P., Torres, L. A. M., et al. (2026) · doi:10.48209/978-65-5417-683-5 · Cited by 1
Keenan, P., & Heavin, C. (2026) · Lecture notes in business information processing · doi:10.1007/978-3-032-25146-6_20 · Cited by 1
Tolend, M., Halabi, R., Ghaouari, K., Lau, Y. C. Y., Alda, M., Hintze, A., Mulsant, B. H., & Ortiz, A. (2026) · The Canadian Journal of Psychiatry · doi:10.1177/07067437261445767 · Cited by 1
Kwok, W. S., Wallbank, G., Hodgson, P., Schräder, T., Shao, L., Elkins, M., Fandim, J. V., Scott, J., Sherrington, C., & Traeger, A. (2026) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2026.112309 · Cited by 1
Ye, Y., Colombo, M., Meessen, J. M. T. A., Hajizadeh, N., Vore, L. D., Sigarchian, H. G., Peters, C., Krovvidi, S., Meresh, S., Hasson, L., et al. (2026) · European Heart Journal - Digital Health · doi:10.1093/ehjdh/ztag070 · Cited by 1
Integration of large language models and evidence-based Chinese medicine: A scoping review
Yao, Y., Liu, H., Yang, D., Luo, X. F., Lai, H., Wang, Z., Chen, Y., & Bian, Z. (2026) · Integrative Medicine Research · doi:10.1016/j.imr.2026.101349 · Cited by 1
Evaluating the Utility of Artificial Intelligence in Conducting Systematic Reviews
Robert Carrier, D. O., Lopez, L., Moya, A., Hernandez, V. H., D’Apuzzo, M. R., & McNamara, C. (2026) · Arthroplasty Today · doi:10.1016/j.artd.2026.102035 · Cited by 1
Pescol, F., Buonocore, T. M., Tibollo, V., Failla, G., Traversi, E., Rovere, M. T. L., Sacchi, L., Ricciardi, W., & Bellazzi, R. (2026) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2026.106509 · Cited by 1
Rahgozar, A., & Mortezaagha, P. (2026) · Artificial Intelligence Review · doi:10.1007/s10462-026-11599-2 · Cited by 1
Jóźwiak, Á. B., Imre, A., Józwiak‐Hagymásy, J., Tittmann, J., Nagy, Á., Kovács, S., Kardas, P., van Boven, J. F. M., Mommers, I., & Ágh, T. (2026) · Expert Review of Pharmacoeconomics & Outcomes Research · doi:10.1080/14737167.2026.2691995 · Cited by 2
Xie, C., Weichang, K., Pi, L., Diaoxin, Q., Yuxuan, Y., Wang, B., & Liang, H. (2026) · Journal of Evidence-Based Medicine · doi:10.1111/jebm.70166 · Cited by 1
Qiu, T., Jiang, H., Chen, M. S., & Lin, T. (2026) · International Journal of Technology Assessment in Health Care · doi:10.1017/s0266462326104073 · Cited by 1
Kurian, N., Cherian, J. M., Reji, R. M., & Varghese, K. G. (2026) · Evidence-Based Dentistry · doi:10.1038/s41432-026-01238-8 · Cited by 1
Ofori-Boateng, R., Aceves‐Martins, M., Wiratunga, N., & Moreno‐García, C. F. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-66538-7_26 · Cited by 1
Use of AI in family medicine publications: a joint editorial from journal editors
Schrager, S., Seehusen, D. A., Sexton, S. M., Richardson, C. R., Neher, J. O., Pimlott, N., Bowman, M., Rodíguez, J., Morley, C. P., Li, L., et al. (2025) · Evidence-Based Practice · doi:10.1097/ebp.0000000000002338 · Cited by 1
Dietrich, E. S. (2025) · Journal of Medical Economics · doi:10.1080/13696998.2025.2488154 · Cited by 1
Jang, W. S., Sultana, S., Yao, Z., Tran, H., Yang, Z., Kwon, S., & Yu, H. (2026) · JMIR AI · doi:10.2196/75561 · Cited by 2
Tao, K., Osman, Z. A., Tzou, P. L., Rhee, S., Ahluwalia, V., & Shafer, R. W. (2024) · Research Square · doi:10.21203/rs.3.rs-3847311/v2 · Cited by 1
Tsybulnik, D. Y., Gillette, J. J., & Heston, T. F. (2025) · medRxiv · doi:10.1101/2025.10.03.25337291 · Cited by 1
From Decks to Decisions: An Auditable AI Framework for Venture CapitalPreprint
Haghani, A., Nur, H., & Pala, S. (2025) · Research Square · doi:10.21203/rs.3.rs-7781525/v1 · Cited by 1
Turki, H., Pasha, N. A., Altammami, A., & Alzahrani, A. (2025) · Research Square · doi:10.21203/rs.3.rs-7660197/v1 · Cited by 1
Kniffert, S., Katthöfer, B., Emprechtinger, R., Pellegrini, P., Funk, E. M., Dhamrait, I. S., Zang, Y., Bornmüller, A., & Toelch, U. (2026) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.64898/2026.01.23.701317 · Cited by 1
Gartlehner, G., Banda, S., Callaghan, M., Chase, J. D., Dobrescu, A. I., Eisele‐Metzger, A., Flemyng, E., Gardner, S., Griebler, U., Helfer, B., et al. (2026) · medRxiv · doi:10.64898/2026.04.13.26350802 · Cited by 1
Rahgozar, A., & Mortezaagha, P. (2026) · arXiv · doi:10.48550/arxiv.2608.14551
“Large language models (LLMs) have recently shown promise as automated screeners [Guo et al., 2024, Wang et al., 2024], achieving competitive recall on benchmark datasets.”
Goldfarb, N., Tal, N., Cohen, I.-C., Elyashar, A., Ellen, M., Puzis, R., & Cohen, O. (2024) · doi:10.2196/preprints.59322
Use of AI in Family Medicine Publications: A Joint Editorial From Journal Editors
Schrager, S., Seehusen, D. A., Sexton, S. M., Richardson, C. R., Neher, J. O., Pimlott, N., Bowman, M. A., Rodríguez, J. E., Morley, C. P., Li, L., et al. (2025) · The Annals of Family Medicine · doi:10.1370/afm.240575
M-PreSS: A Model Pre-training Approach for Study Screening in Systematic ReviewsPreprint
Xu, Z., Davies, P., Millard, L. A. C., Teng, L., Markozannes, G., Erola, P., Seleiro, E. A. P., Higgins, J. P. T., Martin, R. M., Sobczyk-Barad, M., et al. (2025) · medRxiv · doi:10.1101/2025.04.08.25325463 · Cited by 1
Vembye, M. H., Christensen, J., Bondebjerg, A., & Schytt, F. L. W. (2024) · OSF Preprints (OSF Preprints) · doi:10.31219/osf.io/yrhzm_v1
Vembye, M. H., Christensen, J., Bondebjerg, A., & Schytt, F. L. W. (2025) · OSF Preprints (OSF Preprints) · doi:10.31219/osf.io/yrhzm_v2
The STARR Protocol: An Automated LLM Methodology for Enhanced Systematic Literature Review ScreeningPreprint
Baranker, B., Kapoor, A., & Sampath, S. G. (2025) · doi:10.22541/au.174897564.48312932/v1
Zary, N. (2025) · Preprints.org · doi:10.20944/preprints202506.0749.v1
Lee, S., Park, J., Kang, K. H., Han, S., Kim, S., & Seo, M. (2025) · Korean Journal of Environmental Health Sciences · doi:10.5668/jehs.2025.51.3.137
Validating a rapid algorithmic weed hazard ranking method
Buddenhagen, C. E., Bourdôt, G. W., Lamoureaux, S. L., Noble, A. D. L., Dawson, M. I., & Phillips, C. B. (2025) · Pest Management Science · doi:10.1002/ps.70201
Energy Efficiency in Robotics Software: A Systematic Literature Review (2020-2024)
Gupta, A. K. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2508.12170 · Cited by 2
Chen, Z., Liu, T., Mo, Y., Fu, Q., Lei, S., Tong, T., & Tang, X. (2025) · doi:10.2196/preprints.85051
Accelerating Discovery: Rapid Literature Screening with LLMsPreprint
Matalonga, S., Amalfitano, D., Hauck, J. C. R., Solari, M., & Travassos, G. H. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2509.13103
Jaramillo, C., Segura-Osorio, M., Maldonado, K. K. S., & Ríos, I. D. M. (2025) · RECIMUNDO · doi:10.26820/recimundo/9.(4).oct.2025.21-38
Orbea, B. L. F., Medina, R., Solórzano, H. X. P., & Montecé, H. L. C. (2025) · RECIAMUC · doi:10.26820/reciamuc/9.(4).diciembre.2025.48-67
Im, E., Kim, B. Y., Kang, S., & Kim, H. (2025) · JCO Clinical Cancer Informatics · doi:10.1200/cci-25-00233
Shen, P., Yuan, Y., He, X., & Wang, F. (2025) · Global Medical Education · doi:10.1515/gme-2025-0009
Shae, W., Saif, M. S. I., Fife, J., Mudaranthakam, D. P., Pei, D., Harlan-Williams, L. M., Thompson, J. A., & Koestler, D. C. (2025) · JAMIA Open · doi:10.1093/jamiaopen/ooaf156
Increasing Generative AI Effectiveness through Triangulation: The Case for the Future of Work
Strohm, C. J., Matta, V., & Bansal, G. (2025) · AIS Electronic Library (AISeL)
Luo, R. (2025) · MacSphere (McMaster University) · doi:10.71548/11515
Metadata
- Authors
- Guo, E., Gupta, M., Deng, J., Park, Y.-J., Paget, M., & Naugler, C.
- Affiliations
- University of Calgary (CA)
- University of Toronto (CA)
- Corresponding author
- Guo, Eddie
- Venue
- Journal of Medical Internet Research
- Publisher
- JMIR Publications Inc.
- Published
- 2024-01-12
- Volume
- 26
- Pages
- e48996
- ISSN
- 1438-8871 (ISSN-L)
- PMC
- PMC10818236
- OpenAlex
- W4387144848
- Cited by
- 205
- References
- 37
- Citations by year
- 2026: 65 · 2025: 99 · 2024: 37
- FWCI
- 38.15
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- Locations
- Journal of Medical Internet Research (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- arXiv (Cornell University) (submittedVersion, cc-by-nc-nd)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- arXiv (Cornell University)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
- Keywords
- large language models, GPT-4, ChatGPT, systematic review screening, title and abstract screening, interobserver variability, human-AI comparison, AI-assisted literature review
- MeSH terms
- Systematic Reviews as Topic, Data Analysis, Artificial Intelligence, Humans, Language, Natural Language Processing, Problem Solving, Consensus, Biomedical Research, Workflow
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.