Transforming literature screening: The emerging role of large language models in systematic reviews
Open accessAbstract
Systematic reviews (SR) synthesize evidence-based medical literature, but they involve labor-intensive manual article screening. Large language models (LLMs) can select relevant literature, but their quality and efficacy are still being determined compared to humans. We evaluated the overlap between title- and abstract-based selected articles of 18 different LLMs and human-selected articles for three SR. In the three SRs, 185/4,662, 122/1,741, and 45/66 articles have been selected and considered for full-text screening by two independent reviewers. Due to technical variations and the inability of the LLMs to classify all records, the LLM’s considered sample sizes were smaller. However, on average, the 18 LLMs classified 4,294 (min 4,130; max 4,329), 1,539 (min 1,449; max 1,574), and 27 (min 22; max 37) of the titles and abstracts correctly as either included or excluded for the three SRs, respectively. Additional analysis revealed that the definitions of the inclusion criteria and conceptual designs significantly influenced the LLM performances. In conclusion, LLMs can reduce one reviewer´s workload between 33% and 93% during title and abstract screening. However, the exact formulation of the inclusion and exclusion criteria should be refined beforehand for ideal support of the LLMs.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Large language models have demonstrated promising performance in general text classification tasks.
“these LLMs have demonstrated promising performance in general text classification tasks [10, 16, 25]”
From Artificial intelligence for the science of evidence synthesis: how good are AI-powered tools for automatic literature screening? (2025) · cites this paper as [10]
The Results (page not provided) report promising LLM performance in classifying titles and abstracts for systematic-review screening. The paper does not establish performance across general text-classification tasks, so the claim extends beyond the setting tested. (LLM, conf=0.85)
Experimental — accuracy not yet measuredLiterature screening is the most time-consuming process in evidence synthesis practice, particularly in the medical field.
“Literature screening is the most time-consuming process in evidence synthesis practice, particularly in the medical field [9–11]”
From Artificial intelligence for the science of evidence synthesis: how good are AI-powered tools for automatic literature screening? (2025) · cites this paper as [10]
The Abstract describes study-selection screening as notably laborious, typically taking several working days per researcher, and discusses systematic reviews used to inform clinical recommendations. It does not compare screening with other evidence-synthesis tasks or establish that screening is the most time-consuming process. (LLM, conf=0.85)
Experimental — accuracy not yet measuredThe four large language models evaluated in this study demonstrated consistently low false positive fraction (all below 4%), with fewer than 20 articles requiring human review.
“Consistent with previous research, the four LLMs evaluated in this study demonstrated consistently low FPF (all below 4%), with fewer than 20 articles requiring human review [10, 16].”
From Artificial intelligence for the science of evidence synthesis: how good are AI-powered tools for automatic literature screening? (2025) · cites this paper as [10]
The Results report that the models generally had higher specificity than recall, and the Discussion reports reduced human screening workload. But the paper does not establish an FPF below 4% or fewer than 20 articles requiring human review; its reported average remaining workloads were 368, 202, and 40 articles across the three reviews. The four models are described as belonging to the citing study, not this source. (LLM, conf=0.82)
Experimental — accuracy not yet measuredInitial studies indicate that LLM performance in identifying relevant publications is highly dependent on the prompt and often results in high differences between sensitivity and specificity, reflecting the tendency of LLMs toward overconfidence or underconfidence.
“Initial studies indicate that LLM performance in identifying relevant publications is highly dependent on the prompt and often results in high differences between sensitivity and speci fi city, re fl ecting the tendency of LLMs toward overcon fi dence or undercon fi dence.”
From AIM review tool: artificial intelligence for smarter systematic review screening (2026) · cites this paper as [35]
Split into 2 atoms — 0 confirmed, 2 partially supported, 0 disputed, 0 misattributed, 0 unverifiable. Expand the claim to see per-atom verdicts.
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
GRADE: an emerging consensus on rating quality of evidence and strength of recommendations
Guyatt, G. H., Oxman, A. D., Vist, G. E., Kunz, R., Falck-Ytter, Y., Alonso-Coello, P., & Schünemann, H. J. (2008) · BMJ · doi:10.1136/bmj.39489.470347.ad · Cited by 23,305 · see citing works
Cochrane Handbook for Systematic Reviews of Interventions
(2019) · doi:10.1002/9781119536604 · Cited by 14,828 · see citing works
Page, M. J., Moher, D., Bossuyt, P. M., Boutron, I., Hoffmann, T. C., Mulrow, C. D., Shamseer, L., Tetzlaff, J. M., Akl, E. A., Brennan, S. E., et al. (2021) · BMJ · doi:10.1136/bmj.n160 · Cited by 11,883 · cited 1× · see citing works
“However, performing a high - quality SR following scientific guidelines, like the Preferred Reporting Items for SRs and Meta - Analyses (PRISMA) is time - consuming ( 2 ).”
Scikit-learn: Machine Learning in Python
(2011) · Journal of Machine Learning Research · doi:10.5555/1953048.2078195 · Cited by 8,539 · see citing works
Large language models in medicine
Thirunavukarasu, A. J., Ting, D. S. J., Elangovan, K., Gutierrez, L., Tan, T. F., & Ting, D. S. W. (2023) · Nature Medicine · doi:10.1038/s41591-023-02448-8 · Cited by 3,944 · see citing works
Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., Küttler, H., Lewis, M., Yih, W.-T., Rocktäschel, T., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.11401 · Cited by 3,101 · see citing works
Waffenschmidt, S., Knelangen, M., Sieben, W., Bühn, S., & Pieper, D. (2019) · BMC Medical Research Methodology · doi:10.1186/s12874-019-0782-0 · Cited by 626 · see citing works
Challenges and opportunities facing medical education.
Densen, P. (2011) · PubMed Central · Cited by 484 · see citing works
The value of a second reviewer for study selection in systematic reviews
Stoll, C. R. T., Izadi, S. R., Fowler, S. A., Green, P. A., Suls, J. M., & Colditz, G. A. (2019) · Research Synthesis Methods · doi:10.1002/jrsm.1369 · Cited by 314 · see citing works
LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale
Dettmers, T., Lewis, M., Belkada, Y., & Zettlemoyer, L. (2022) · arXiv · doi:10.48550/arxiv.2208.07339 · Cited by 305 · see citing works
Systematic Review of Economic Impact Studies of Artificial Intelligence in Health Care
Wolff, J., Pauling, J. K., Keck, A., & Baumbach, J. (2019) · Journal of Medical Internet Research · doi:10.2196/16866 · Cited by 277 · see citing works
Self-citationKhraisha, Q., Put, S., Kappenberg, J., Warraitch, A., & Hadfield, K. (2024) · Research Synthesis Methods · doi:10.1002/jrsm.1715 · Cited by 269 · see citing works
Khraisha, Q., Put, S., Kappenberg, J., Warraitch, A., & Hadfield, K. (2023) · arXiv · doi:10.48550/arxiv.2310.17526 · Cited by 268 · see citing works
Mastering Machine Learning With scikit-learn
Hackeling, G. (2014) · CERN Document Server (European Organization for Nuclear Research) · Cited by 238 · see citing works
LLM-QAT: Data-Free Quantization Aware Training for Large Language Models
Liu, Z., Oğuz, B., Zhao, C., Chang, E., Stock, P., Mehdad, Y., Shi, Y., Krishnamoorthi, R., & Chandra, V. (2024) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2024.findings-acl.26 · Cited by 135 · see citing works
Large Language Models Can Be Easily Distracted by Irrelevant Context
Shi, F., Chen, X., Misra, K., Scales, N., Dohan, D., Chi, E., Schärli, N., & Zhou, D. (2023) · arXiv · doi:10.48550/arxiv.2302.00093 · Cited by 134 · cited 1× · see citing works
“However, LLMs face a significant limitation in their context window, restricting the amount of text they can process simultaneously ( 7 ), which poses challenges when dealing with lengthy scientific articles, particularly in SRs.”
Hasan, B., Saadi, S. M., Rajjoub, N. S., Hegazi, M., Al-Kordi, M., Fleti, F., Farah, M. H., Riaz, I. B., Banerjee, I., Wang, Z., et al. (2024) · BMJ Evidence-Based Medicine · doi:10.1136/bmjebm-2023-112597 · Cited by 72 · see citing works
Transforming clinical trials: the emerging roles of large language models
Ghim, J., & Ahn, S. (2023) · Translational and Clinical Pharmacology · doi:10.12793/tcp.2023.31.e16 · Cited by 61 · see citing works
The global prevalence of overweight and obesity among nurses: A systematic review and meta‐analyses
Sadali, U. B., Kamal, K. K. B. N., Park, J., Chew, H. S. J., & Devi, M. K. (2023) · Journal of Clinical Nursing · doi:10.1111/jocn.16861 · Cited by 36 · see citing works
Keeping Up With the Medical Literature: Why, How, and When?
Kamtchum-Tatuene, J., & Zafack, J. G. (2021) · Stroke · doi:10.1161/strokeaha.121.036141 · Cited by 32 · see citing works
Nashwan, A. J., & Jaradat, J. H. (2023) · Cureus · doi:10.7759/cureus.43023 · Cited by 31 · see citing works
Nussbaum, Z., Morris, J. X., Duderstadt, B., et al. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2402.01613 · Cited by 28 · see citing works
Staying Up to Date and Managing Information Overload
Maggio, L. A., & Artino, A. R. (2018) · Journal of Graduate Medical Education · doi:10.4300/jgme-d-18-00621.1 · Cited by 18 · see citing works
Jennings, M. J., Lochmüller, A., Atalaia, A., et al. (2020) · Journal of Neuromuscular Diseases · doi:10.3233/jnd-200546 · Cited by 15 · see citing works
Self-citationBaumbach, L., Feddern, W., Kretzler, B., Hajek, A., & König, H. (2024) · Sports Medicine - Open · doi:10.1186/s40798-024-00713-9 · Cited by 14 · see citing works
Self-citationLLM-QAT: Data-Free Quantization Aware Training for Large Language Models
Liu, Z., Oguz, B., Zhao, C., Chang, E., Stock, P., Mehdad, Y., Shi, Y., Krishnamoorthi, R., & Chandra, V. (2023) · arXiv · doi:10.48550/arxiv.2305.17888 · Cited by 13 · see citing works
A semiparametric approach for meta‐analysis of diagnostic accuracy studies with multiple cut‐offs
Frömke, C., Kirstein, M., & Zapf, A. (2022) · Research Synthesis Methods · doi:10.1002/jrsm.1579 · Cited by 7 · see citing works
Economic evaluations of musculoskeletal physiotherapy: protocol of a systematic review
Baumbach, L., König, H., Kretzler, B., & Hajek, A. (2022) · BMJ Open · doi:10.1136/bmjopen-2021-058143 · Cited by 5 · see citing works
Self-citationZero-shot Generative Large Language Models for Systematic Review Screening Automation
Wang, S., Scells, H., Zhuang, S., Potthast, M., Koopman, B., & Zuccon, G. (2024) · arXiv · doi:10.48550/arxiv.2401.06320 · Cited by 5 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Automation of Systematic Reviews with Large Language Models
Cao, C., Arora, R., Cento, P., Budak, A., Manta, K., Farahani, E., Cecere, M., Selemon, A., Sang, J., Gong, L. X., et al. (2025) · medRxiv · doi:10.1101/2025.06.13.25329541 · Cited by 33 · see citing works
Galli, C., Gavrilova, A. V., & Calciolari, E. (2025) · Information · doi:10.3390/info16050378 · Cited by 36 · see citing works
Nykvist, B., Macura, B., Xylia, M., & Olsson, E. (2025) · Environmental Evidence · doi:10.1186/s13750-025-00360-x · Cited by 17 · see citing works
Ruan, M., Fan, J., Liu, M., Meng, Z., Zhang, X., & Zhang, C. (2025) · BMC Medical Research Methodology · doi:10.1186/s12874-025-02644-9 · Cited by 12 · see citing works
“However, these LLMs have demonstrated promising performance in general text classification tasks [10, 16, 25].”
InfluentialAIM review tool: artificial intelligence for smarter systematic review screening
Mena, S., Rituerto-González, E., Coutts, F., von Trott, J., Jacobs, G. R., Bryant, L., Moles, L., Sirbu, N., Promet, L., Oliver, D., et al. (2026) · npj Artificial Intelligence · doi:10.1038/s44387-026-00080-8
“With the growing popularityoflargelanguagemodels(LLMs),recentbenchmarkstudieshave evaluatedtheirperformanceintitleandabstractsystematicreview screening 35 – 37 .”
- Authors
- Delgado-Chaves, F. M., Jennings, M. J., Atalaia, A., Wolff, J., Horvath, R., Mamdouh, Z. M., Baumbach, J., & Baumbach, L.
- Affiliations
- Universität Hamburg (DE)
- Columbia University Irving Medical Center (US)
- Inserm Center of Research in Myology, Neuro-Myology Service G.H. Pitié-Salpêtrière, Sorbonne Université
- Syte – Strategy Institute for DigitalHealth
- Department of Clinical Neurosciences, University of Cambridge
- Zagazig University (EG)
- Corresponding author
- Delgado-Chaves, Fernando M.
- ORCID
- Venue
- Proceedings of the National Academy of Sciences
- Publisher
- National Academy of Sciences
- Published
- 2025-01-06
- Volume
- 122
- Issue
- 2
- Pages
- e2411962122-e2411962122
- ISSN
- 0027-8424 (ISSN-L)
- 1091-6490
- PMC
- PMC11745399
- OpenAlex
- W4406110856
- Cited by
- 86
- References
- 29
- Citations by year
- 2026: 56 · 2025: 29 · 2024: 1
- FWCI
- 16.24
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Proceedings of the National Academy of Sciences (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- University of Southern Denmark Research Portal (University of Southern Denmark) (publishedVersion, cc-by)
- Apollo (University of Cambridge) (submittedVersion, cc-by)
- Research Publications (Maastricht University) (submittedVersion, cc-by)
- Funders
- European Commission (101057619)
- Institut National de la Santé et de la Recherche Médicale
- Universiteit Maastricht
- Universität Hamburg
- Syddansk Universitet
- Zagazig University
- Sorbonne Université
- Universitätsklinikum Hamburg-Eppendorf
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
- Keywords
- large language models, systematic reviews, literature screening, title and abstract screening, full-text screening, inclusion criteria, AI-assisted literature review
- MeSH terms
- Humans, Language, Review Literature as Topic, Evidence-Based Medicine
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
