Development of Prompt Templates for Large Language Model–Driven Screening in Systematic Reviews
Open accessPublished version of a preprint
The DOI matches its Crossref record; 67 of the 67 references it cites were found in a registry or an index.
Abstract
BACKGROUND: Systematic reviews (SRs) are hindered by the initial rigorous article screen, which delays access to reliable information synthesis. OBJECTIVE: To develop generic prompt templates for large language model (LLM)-driven abstract and full-text screening that can be adapted to different reviews. DESIGN: Diagnostic test accuracy. SETTING: 48 425 citations were tested for abstract screening across 10 SRs. Full-text screening evaluated all 12 690 freely available articles from the original search. Prompt development used the GPT4-0125-preview model (OpenAI). PARTICIPANTS: None. MEASUREMENTS: Large language models were prompted to include or exclude articles based on SR eligibility criteria. Model outputs were compared with original SR author decisions after full-text screening to evaluate performance (accuracy, sensitivity, and specificity). RESULTS: Optimized prompts using GPT4-0125-preview achieved a weighted sensitivity of 97.7% (range, 86.7% to 100%) and specificity of 85.2% (range, 68.3% to 95.9%) in abstract screening and weighted sensitivity of 96.5% (range, 89.7% to 100.0%) and specificity of 91.2% (range, 80.7% to 100%) in full-text screening across 10 SRs. In contrast, zero-shot prompts had poor sensitivity (49.0% abstract, 49.1% full-text). Across LLMs, Claude-3.5 (Anthropic) and GPT4 variants had similar performance, whereas Gemini Pro (Google) and GPT3.5 (OpenAI) models underperformed. Direct screening costs for 10 000 citations differed substantially: Where single human abstract screening was estimated to require more than 83 hours and $1666.67 USD, our LLM-based approach completed screening in under 1 day for $157.02 USD. LIMITATIONS: Further prompt optimizations may exist. Retrospective study. Convenience sample of SRs. Full-text screening evaluations were limited to free PubMed Central full-text articles. CONCLUSION: A generic prompt for abstract and full-text screening achieving high sensitivity and specificity that can be adapted to other SRs and LLMs was developed. Our prompting innovations may have value to SR investigators and researchers conducting similar criteria-based tasks across the medical sciences. PRIMARY FUNDING SOURCE: None.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Other recent work has demonstrated promise for LLMs in data extraction 7,8”
From Automation of Systematic Reviews with Large Language Models (2025) · cites this paper as [8]
In the citing paper
“though these studies rely on self-defined reference standards and evaluate on small datasets.”
From Automation of Systematic Reviews with Large Language Models (2025) · cites this paper as [8]
In the citing paper
“Previous work has shown that ChatGPT 4.1 is very accurate at SR screening”
Citations
Cites
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
Kaugeranna, R., Kaugeranna, E., & Aion, (. S. 4. 6. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.giscience.2023.43 · Cited by 13,690
Cumpston, M., Li, T., Page, M. J., Chandler, J., Welch, V. A., Higgins, J. P., & Thomas, J. (2019) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.ed000142 · Cited by 7,218
THE USE OF CONFIDENCE OR FIDUCIAL LIMITS ILLUSTRATED IN THE CASE OF THE BINOMIAL
CLOPPER, C. J., & PEARSON, E. S. (1934) · Biometrika · doi:10.1093/biomet/26.4.404 · Cited by 5,003
The Levels of Evidence and Their Role in Evidence-Based Medicine
Burns, P. B., Rohrich, R. J., & Chung, K. C. (2011) · Plastic and Reconstructive Surgery · doi:10.1097/prs.0b013e318219c171 · Cited by 2,155 · cited 1×
“We curated BenchSR , a collection of 10 SR data sets comprising more than 170 000 articles, spanning 4 of 6 Oxford Centre for Evidence-Based Medicine question types (32, 33) and 9 different clinical domains (34) ( Appendix Table 1 ).”
Large language models are zero-shot reasoners
Kojima, T., Gu, S. (., Reid, M., Matsuo, Y., & Iwasawa, Y. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-1613 · Cited by 2,013
Large Language Models are Zero-Shot Reasoners
Kojima, T., Gu, S. S., Reid, M., Matsuo, Y., & Iwasawa, Y. (2022) · arXiv · doi:10.48550/arxiv.2205.11916 · Cited by 2,013
Lost in the Middle: How Language Models Use Long Contexts
Liu, N. F., Lin, K., Hewitt, J., Paranjape, A., Bevilacqua, M., Petroni, F., & Liang, P. (2024) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00638 · Cited by 1,612 · cited 1×
“Finally, for lengthy documents (that is, full-text articles), our ISO-ScreenPrompt approach addresses the lost-in-the-middle phenomenon (42 – 44) by repeating and numbering instructions at the start and end of the prompt (43).”
Lost in the Middle: How Language Models Use Long Contexts
Liu, N. F., Lin, K., Hewitt, J., Paranjape, A., Bevilacqua, M., Petroni, F., & Liang, P. (2023) · arXiv · doi:10.48550/arxiv.2307.03172 · Cited by 1,599
Evidence summaries: the evolution of a rapid review approach
Khangura, S., Konnyu, K., Cushman, R., Grimshaw, J., & Moher, D. (2012) · Systematic Reviews · doi:10.1186/2046-4053-1-10 · Cited by 1,232 · cited 1×
“However, SRs are resource intensive, typically requiring 1 year and upwards of $100000 to complete due to the comprehensive processes of conducting detailed searches, screening articles, extracting data, analyzing fi ndings, and report writing (2 – 4).”
Sampling Techniques. 3rd Edition
Cochran, W. G. (1977) · Zenodo · doi:10.5281/zenodo.15338807 · Cited by 1,136
Prompt Programming for Large Language Models: Beyond the Few-Shot Paradigm
Reynolds, L., & McDonell, K. (2021) · Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411763.3451760 · Cited by 952
Self-Consistency Improves Chain of Thought Reasoning in Language Models
Wang, X., Wei, J., Schuurmans, D., Le, Q., Chi, E., Narang, S., Chowdhery, A., & Zhou, D. (2022) · arXiv · doi:10.48550/arxiv.2203.11171 · Cited by 829
Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?
Min, S., Lyu, X., Holtzman, A., Artetxe, M., Lewis, M., Hajishirzi, H., & Zettlemoyer, L. (2022) · Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2022.emnlp-main.759 · Cited by 755
Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?
Min, S., Lyu, X., Holtzman, A., Artetxe, M., Lewis, M., Hajishirzi, H., & Zettlemoyer, L. (2022) · arXiv · doi:10.48550/arxiv.2202.12837 · Cited by 752
Borah, R., Brown, A. W., Capers, P. L., & Kaiser, K. A. (2017) · BMJ Open · doi:10.1136/bmjopen-2016-012545 · Cited by 737 · cited 1×
“However, SRs are resource intensive, typically requiring 1 year and upwards of $100000 to complete due to the comprehensive processes of conducting detailed searches, screening articles, extracting data, analyzing fi ndings, and report writing (2 – 4).”
Waffenschmidt, S., Knelangen, M., Sieben, W., Bühn, S., & Pieper, D. (2019) · BMC Medical Research Methodology · doi:10.1186/s12874-019-0782-0 · Cited by 626
Bobrovitz, N., Ware, H., Ma, X., Li, Z., Hosseini, R., Cao, C., Selemon, A., Whelan, M., Premji, Z., Issa, H., et al. (2023) · The Lancet Infectious Diseases · doi:10.1016/s1473-3099(22)00801-5 · Cited by 614
Self-citationThe Use of Confidence or Fiducial Limits Illustrated in the Case of the Binomial
Clopper, C. J., & Pearson, E. S. (1934) · Biometrika · doi:10.2307/2331986 · Cited by 438
Reducing Workload in Systematic Review Preparation Using Automated Citation Classification
Cohen, A. M., Hersh, W. R., Peterson, K., & Yen, P.-Y. (2006) · Journal of the American Medical Informatics Association · doi:10.1197/jamia.m1929 · Cited by 411
Best practice guidelines for abstract screening large‐evidence systematic reviews and meta‐analyses
Polanin, J. R., Pigott, T. D., Espelage, D. L., & Grotpeter, J. K. (2019) · Research Synthesis Methods · doi:10.1002/jrsm.1354 · Cited by 401 · cited 1×
“This represents the median of estimates we identi fi ed searching the literature (4.3 to 20 minutes for a single full-text article) (29, 30).”
InfluentialPerformance of an Artificial Intelligence Chatbot in Ophthalmic Knowledge Assessment
Mihalache, A., Popovic, M. M., & Muni, R. H. (2023) · JAMA Ophthalmology · doi:10.1001/jamaophthalmol.2023.1144 · Cited by 304
Measuring Massive Multitask Language Understanding
Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., & Steinhardt, J. (2020) · arXiv · doi:10.48550/arxiv.2009.03300 · Cited by 291 · cited 1×
“(41).”
Khraisha, Q., Put, S., Kappenberg, J., Warraitch, A., & Hadfield, K. (2024) · Research Synthesis Methods · doi:10.1002/jrsm.1715 · Cited by 269
Khraisha, Q., Put, S., Kappenberg, J., Warraitch, A., & Hadfield, K. (2023) · arXiv · doi:10.48550/arxiv.2310.17526 · Cited by 268
Global seroprevalence of SARS-CoV-2 antibodies: A systematic review and meta-analysis
Bobrovitz, N., Arora, R. K., Cao, C., Boucher, E., Liu, M., Donnici, C., Yanes-Lane, M., Whelan, M., Perlman-Arrow, S., Chen, J., et al. (2021) · PLOS ONE · doi:10.1371/journal.pone.0252617 · Cited by 246 · cited 1×
“We created a database ( BenchSR ) of 10 previously published SRs spanning 9 unique clinical domains collected through convenience sampling based on Oxford Centre for Evidence-Based Medicine SR question types ( Appendix Table 1 and Supplementary Methods , available at Annals.org) (17 – 26).”
Self-citationMake Your LLM Fully Utilize the Context
An, S., Ma, Z., Lin, Z., Zheng, N., Lou, J., & Chen, W. (2024) · neural information processing systems · doi:10.52202/079017-1986 · Cited by 242
Automated Paper Screening for Clinical Reviews Using Large Language Models
Guo, E., Gupta, M., Deng, J., Park, Y.-J., Paget, M., & Naugler, C. (2023) · arXiv · doi:10.48550/arxiv.2305.00844 · Cited by 230
Make Your LLM Fully Utilize the ContextPreprint
An, S., Ma, Z., Lin, Z., Zheng, N., Lou, J.-G., & Chen, W.-Z. (2024) · Neural Information Processing Systems · doi:10.48550/arxiv.2404.16811 · Cited by 224 · cited 1×
“Finally, for lengthy documents (that is, full-text articles), our ISO-ScreenPrompt approach addresses the lost-in-the-middle phenomenon (42 – 44) by repeating and numbering instructions at the start and end of the prompt (43).”
Gartlehner, G., Affengruber, L., Titscher, V., Noel-Storr, A., Dooley, G., Ballarini, N., & König, F. (2020) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2020.01.005 · Cited by 206
Automated paper screening for clinical reviews using large language models: Data analysis study
Guo, E., Gupta, M., Deng, J., Park, Y.-J., Paget, M., & Naugler, C. (2024) · Journal of Medical Internet Research · doi:10.2196/48996 · Cited by 205 · cited 1×
“However, previous LLM assessments for SR screening relied on basic zero-shot prompting — where LLMs are only given task instructions without examples — and demonstrated low sensitivity and recall or speci fi city that precludes their use (12 – 16).”
Qureshi, R., Shaughnessy, D., Gill, K. A. R., Robinson, K. A., Li, T., & Agai, E. (2023) · Systematic Reviews · doi:10.1186/s13643-023-02243-z · Cited by 201 · cited 1×
“The rise of large language models (LLMs), such as generative pretrained transformer (GPT; OpenAI), creates new horizons for SR screening (9, 10).”
Chai, K. E. K., Lines, R. L. J., Gucciardi, D. F., & Ng, L. (2021) · Systematic Reviews · doi:10.1186/s13643-021-01635-3 · Cited by 190
Can Generalist Foundation Models Outcompete Special-Purpose Tuning? Case Study in Medicine
Nori, H., Lee, Y. T., Zhang, S., Carignan, D., Edgar, R., Fusi, N., King, N., Larson, J., Li, Y., Liu, W., et al. (2023) · arXiv · doi:10.48550/arxiv.2311.16452 · Cited by 178 · cited 1×
“MedPrompt (Microsoft) (11), a collection of prompting techniques to optimize GPT4 performance on medical benchmarks, has shown that well-prompted generalist models can surpass fi ne-tuning methods.”
O’Connor, A. M., Tsafnat, G., Thomas, J., Glasziou, P., Gilbert, S. B., & Hutton, B. (2019) · Systematic Reviews · doi:10.1186/s13643-019-1062-0 · Cited by 167 · cited 1×
“Existing tools only supplement human work fl ows, lack the performance required for independent decision making, and require extensive historical training data that is not available when conducting a new review (7, 8).”
Sui, Y., Zhou, M., Zhou, M., Han, S., & Zhang, D. (2024) · Proceedings of the 17th ACM International Conference on Web Search and Data Mining · doi:10.1145/3616855.3635752 · Cited by 155
Michelson, M., & Reuter, K. (2019) · Contemporary Clinical Trials Communications · doi:10.1016/j.conctc.2019.100443 · Cited by 152
Large Language Models Are Poor Medical Coders — Benchmarking of Medical Code Querying
Soroush, A., Glicksberg, B. S., Zimlichman, E., Barash, Y., Freeman, R., Charney, A. W., Nadkarni, G. N., & Klang, E. (2024) · NEJM AI · doi:10.1056/aidbp2300040 · Cited by 137
Gates, A., Guitard, S., Pillay, J., Elliott, S. A., Dyson, M. P., Newton, A. S., & Hartling, L. (2019) · Systematic Reviews · doi:10.1186/s13643-019-1222-2 · Cited by 133 · cited 2×
“…weighted speci fi city of 85.2% across 10 diverse reviews, surpassing previous tools (for example, Abstrackr [Agency for Healthcare Research and Quality], Rayyan [Rayyan], and RobotAnalyst) (37, 38) and estimates of single human-reviewer performance (86.6% sensitivity and 79.2% speci fi city) (39).”
InfluentialLarger language models do in-context learning differently
Wei, J., Wei, J., Tay, Y., Tran, D., Webson, A., Lu, Y., Chen, X., Liu, H., Huang, D., Zhou, D., et al. (2023) · arXiv · doi:10.48550/arxiv.2303.03846 · Cited by 109
Resource use during systematic review production varies widely: a scoping review
Nussbaumer-Streit, B., Ellen, M., Klerings, I., Sfetcu, R., Riva, N., Mahmić-Kaknjo, M., Poulentzas, G., Martinez, P., Baladia, E., Ziganshina, L. E., et al. (2021) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2021.05.019 · Cited by 98
Performance of ChatGPT in medical examinations: A systematic review and a meta‐analysis
Levin, G., Horesh, N., Brezinov, Y., & Meyer, R. (2024) · BJOG: An International Journal of Obstetrics & Gynaecology · doi:10.1111/1471-0528.17641 · Cited by 90
Leung, A. A., Symonds, C. J., Hundemer, G. L., Ronksley, P. E., Lorenzetti, D. L., Pasieka, J. L., Harvey, A., & Kline, G. A. (2022) · Hypertension · doi:10.1161/hypertensionaha.122.19377 · Cited by 89
Self-citationPerformance of a Large Language Model in Screening Citations
Oami, T., Okada, Y., & Nakada, T.-A. (2024) · JAMA Network Open · doi:10.1001/jamanetworkopen.2024.20496 · Cited by 84 · cited 1×
“Our generic template offers an alternative for investigators seeking to automate screening with minimal upfront effort rather than relying on review-speci fi c customized prompts (16, 36).”
Guillen-Grima, F., Guillen-Aguinaga, S., Guillen-Aguinaga, L., Alas-Brun, R., Onambele, L., Ortega, W., Montejo, R., Aguinaga-Ontoso, E., Barach, P., & Aguinaga-Ontoso, I. (2023) · Clinics and Practice · doi:10.3390/clinpract13060130 · Cited by 72
Tran, V.-T., Gartlehner, G., Yaacoub, S., Boutron, I., Schwingshackl, L., Stadelmaier, J., Sommer, I., Alebouyeh, F., Afach, S., Meerpohl, J., et al. (2024) · Annals of Internal Medicine · doi:10.7326/m23-3389 · Cited by 66
Han, C., Kim, D. W., Kim, S., You, S. C., Park, J. Y., Bae, S., & Yoon, D. (2024) · iScience · doi:10.1016/j.isci.2024.109022 · Cited by 62
Zero-Shot Clinical Trial Patient Matching with LLMs
Wornow, M., Lozano, A., Dash, D., Jindal, J., Mahaffey, K. W., & Shah, N. H. (2025) · NEJM AI · doi:10.1056/aics2400360 · Cited by 59
Zero-Shot Clinical Trial Patient Matching with LLMs
Wornow, M., Lozano, A., Dash, D., Jindal, J., Mahaffey, K. W., & Shah, N. H. (2024) · arXiv · doi:10.48550/arxiv.2402.05125 · Cited by 59
The Systematic Review Toolbox: keeping up to date with tools to support evidence synthesis
Johnson, E. E., O’Keefe, H., Sutton, A., & Marshall, C. (2022) · Systematic Reviews · doi:10.1186/s13643-022-02122-z · Cited by 57
Alvi, M. A., Kwon, B. K., Hejrati, N., Tetreault, L. A., Evaniew, N., Skelly, A. C., & Fehlings, M. G. (2024) · Global Spine Journal · doi:10.1177/21925682231196514 · Cited by 56
Self-citationEfficient Streaming Language Models with Attention Sinks
Xiao, G., Tian, Y., Chen, B., Han, S., & Lewis, M. (2023) · arXiv · doi:10.48550/arxiv.2309.17453 · Cited by 55 · cited 1×
“Finally, for lengthy documents (that is, full-text articles), our ISO-ScreenPrompt approach addresses the lost-in-the-middle phenomenon (42 – 44) by repeating and numbering instructions at the start and end of the prompt (43).”
Samnani, S., Cenzer, I., Kline, G. A., Lee, S. J., Hundemer, G. L., McClurg, C., Pasieka, J. L., Boscardin, W. J., Ronksley, P. E., & Leung, A. A. (2024) · The Journal of Clinical Endocrinology & Metabolism · doi:10.1210/clinem/dgad654 · Cited by 50
Self-citationTeja, B., Berube, M., Pereira, T. V., Law, A. C., Schanock, C., Pang, B., Wunsch, H., Walkey, A. J., & Bosch, N. A. (2024) · American Journal of Respiratory and Critical Care Medicine · doi:10.1164/rccm.202310-1785oc · Cited by 48
Self-citationsrBERT: automatic article classification model for systematic review using BERT
Aum, S., & Choe, S. (2021) · Systematic Reviews · doi:10.1186/s13643-021-01763-w · Cited by 48
Enhancing title and abstract screening for systematic reviews with GPT-3.5 turbo
Gargari, O. K., Mahmoudi, M. H., Hajisafarali, M., & Samiee, R. (2023) · BMJ Evidence-Based Medicine · doi:10.1136/bmjebm-2023-112678 · Cited by 46
Evaluating GPT-4-based ChatGPT’s Clinical Potential on the NEJM Quiz
Ueda, D., Walston, S. L., Matsumoto, T., Deguchi, R., Tatekawa, H., & Miki, Y. (2023) · doi:10.1101/2023.05.04.23289493 · Cited by 45
Evaluating GPT-4-based ChatGPT's clinical potential on the NEJM quiz
Ueda, D., Walston, S. L., Matsumoto, T., Deguchi, R., Tatekawa, H., & Miki, Y. (2024) · BMC Digital Health · doi:10.1186/s44247-023-00058-5 · Cited by 44
Perlman‐Arrow, S., Loo, N., Bobrovitz, N., Yan, T., & Arora, R. K. (2023) · Research Synthesis Methods · doi:10.1002/jrsm.1636 · Cited by 30
Self-citationNashwan, A. J., & Jaradat, J. H. (2023) · Cureus · doi:10.7759/cureus.43023 · Cited by 30 · cited 1×
“The rise of large language models (LLMs), such as generative pretrained transformer (GPT; OpenAI), creates new horizons for SR screening (9, 10).”
Single screening or double screening for study selection in systematic reviews?
Mahtani, K. R., Heneghan, C. J., & Aronson, J. K. (2019) · BMJ Evidence-Based Medicine · doi:10.1136/bmjebm-2019-111269 · Cited by 30
Bobrovitz, N., Heneghan, C. J., Onakpoya, I., Fletcher, B., Collins, D. R. J., Tompson, A., Lee, J. J., Nunan, D., Fisher, R. F., Scott, B. N. V., et al. (2018) · BMC Medicine · doi:10.1186/s12916-018-1104-9 · Cited by 29
Self-citationCalcium during cardiac arrest: A systematic review
Hsu, C. H., Couper, K., Nix, T., Drennan, I. R., Reynolds, J. C., Kleinman, M. E., & Berg, K. M. (2023) · Resuscitation Plus · doi:10.1016/j.resplu.2023.100379 · Cited by 28
Self-citationAssessing the Ability of ChatGPT to Screen Articles for Systematic Reviews
Syriani, E., David, I., & Kumar, G. (2023) · arXiv · doi:10.48550/arxiv.2307.06464 · Cited by 27
Wang, S., Scells, H., Clark, J., Koopman, B., & Zuccon, G. (2022) · Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval · doi:10.1145/3477495.3531748 · Cited by 21
Baczynski, M., Jasani, B., Castro, C. D., Dani, C., Subhedar, N. V., Chandrasekharan, P. K., Lakshminrusimha, S., McNamara, P. J., Bischoff, A. R., & Jain, A. (2023) · Early Human Development · doi:10.1016/j.earlhumdev.2023.105841 · Cited by 18
Direct Evaluation of Chain-of-Thought in Multi-hop Reasoning with Knowledge Graphs
Nguyen, T., Luo, L., Shiri, F., Phung, D., Li, Y.-F., Vu, T.-T., & Haffari, G. (2024) · Findings of the Association for Computational Linguistics ACL 2024 · doi:10.18653/v1/2024.findings-acl.168 · Cited by 17
Cited by
Scherbakov, D., Hubig, N., Jansari, V., Bakumenko, A., & Lenert, L. A. (2024) · arXiv · doi:10.48550/arxiv.2409.04600 · Cited by 156
Scherbakov, D., Hubig, N., Jansari, V., Bakumenko, A., & Lenert, L. A. (2025) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocaf063 · Cited by 154
Artificial intelligence across the cancer care continuum
Riaz, I. B., Khan, M. A., & Osterman, T. J. (2025) · Cancer · doi:10.1002/cncr.70050 · Cited by 40
Automation of Systematic Reviews with Large Language Models
Cao, C., Arora, R., Cento, P., Budak, A., Manta, K., Farahani, E., Cecere, M., Selemon, A., Sang, J., Gong, L. X., et al. (2025) · medRxiv · doi:10.1101/2025.06.13.25329541 · Cited by 33
Self-citationGalli, C., Gavrilova, A. V., & Calciolari, E. (2025) · Information · doi:10.3390/info16050378 · Cited by 36
Metadata
- Authors
- Cao, C., Sang, J., Arora, R., Chen, D., Kloosterman, R., Cecere, M., Gorla, J., Saleh, R., Drennan, I., Teja, B., Fehlings, M., Ronksley, P., Leung, A. A., Weisz, D. E., Ware, H., Whelan, M., Emerson, D. B., Arora, R. K., & Bobrovitz, N.
- Affiliations
- University of Calgary (CA)
- Stripe, San Francisco, California (J.S.)
- Harvard University (US)
- University of Toronto (CA)
- Temerty Faculty of Medicine, University of Toronto, Toronto, Ontario, Canada (D.C., R.K., M.C., J.G., R.S.)
- St. Michael's Hospital (CA)
- Sunnybrook Health Science Centre (CA)
- Vector Institute (CA)
- Corresponding author
- Cao, Christian
- ORCID
- Venue
- Annals of Internal Medicine
- Publisher
- American College of Physicians
- Published
- 2025-03
- Volume
- 178
- Issue
- 3
- Pages
- 389-401
- ISSN
- 0003-4819 (ISSN-L)
- 1539-3704
- OpenAlex
- W4407873728
- Cited by
- 91
- References
- 38
- Citations by year
- 2026: 62 · 2025: 26
- FWCI
- 9.24
- Type
- journal article
- Language
- en
- Open access
- green
- doi.org
- Licence
- cc-by-nc-nd
- www.acpjournals.org/journal/aim/text-and-data-mining (tdm)
- Locations
- Annals of Internal Medicine (publishedVersion)
- medRxiv (submittedVersion, cc-by-nc-nd)
- PubMed (publishedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
- Keywords
- large language models, prompt templates, systematic review screening, abstract screening, full-text screening, GPT-4, zero-shot prompting, diagnostic accuracy, screening sensitivity and specificity, screening costs, eligibility criteria, Claude 3.5, Gemini Pro, GPT-3.5
- MeSH terms
- Systematic Reviews as Topic, Large Language Models, Humans, Sensitivity and Specificity
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.