Large Language Models in Systematic Review Screening: Opportunities, Challenges, and Methodological Considerations
Open accessAbstract
Systematic reviews require labor-intensive screening processes—an approach prone to bottlenecks, delays, and scalability constraints in large-scale reviews. Large Language Models (LLMs) have recently emerged as a powerful alternative, capable of operating in zero-shot or few-shot modes to classify abstracts according to predefined criteria without requiring continuous human intervention like semi-automated platforms. This review focuses on the central challenges that users in the biomedical field encounter when integrating LLMs—such as GPT-4—into evidence-based research. It examines critical requirements for software and data preprocessing, discusses various prompt strategies, and underscores the continued need for human oversight to maintain rigorous quality control. By drawing on current practices for cost management, reproducibility, and prompt refinement, this article highlights how review teams can substantially reduce screening workloads without compromising the comprehensiveness of evidence-based inquiry. The findings presented aim to balance the strengths of LLM-driven automation with structured human checks, ensuring that systematic reviews retain their methodological integrity while leveraging the efficiency gains made possible by recent advances in artificial intelligence.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The process followed recent recommendations for AI-assisted synthesis workflows, ensuring transparency and consistency while maintaining full human control over final categorization and interpretation.
“This process followed recent recommendations for AI-assisted synthesis workflows [43,44], ensuring transparency and consistency while maintaining full human control over final categorization and interpretation.”
From A Systematic Literature Review on the Pedagogical Implications and Impact of GenAI on Students' Critical Thinking (2026) · cites this paper as [43]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Rayyan-a web and mobile app for systematic reviews.
Ouzzani, M., Hammady, H., Fedorowicz, Z., & Elmagarmid, A. (2016) · Systematic Reviews · doi:10.1186/s13643-016-0384-4 · Cited by 26,092 · see citing works
Data Structures for Statistical Computing in Python
McKinney, W. (2010) · Proceedings of the Python in Science Conferences · doi:10.25080/majora-92bf1922-00a · Cited by 11,511 · see citing works
Transformers: State-of-the-Art Natural Language Processing
Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., et al. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations · doi:10.18653/v1/2020.emnlp-demos.6 · Cited by 8,741 · see citing works
Cumpston, M., Li, T., Page, M. J., Chandler, J., Welch, V. A., Higgins, J. P., & Thomas, J. (2019) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.ed000142 · Cited by 7,218 · see citing works
Biopython: freely available python tools for computational molecular biology and bioinformatics
Cock, P. J. A., Antao, T., Chang, J. T., Chapman, B. A., Cox, C. J., Dalke, A., Friedberg, I., Hamelryck, T., Kauff, F., Wilczynski, B., et al. (2009) · Bioinformatics · doi:10.1093/bioinformatics/btp163 · Cited by 6,080 · see citing works
Methley, A. M., Campbell, S., Chew-Graham, C., McNally, R., & Cheraghi-Sohi, S. (2014) · BMC Health Services Research · doi:10.1186/s12913-014-0579-0 · Cited by 2,706 · see citing works
Beyond PICO: The SPIDER Tool for Qualitative Evidence Synthesis
Cooke, A., Smith, D., & Booth, A. (2012) · Qualitative Health Research · doi:10.1177/1049732312452938 · Cited by 2,608 · see citing works
Gusenbauer, M., & Haddaway, N. R. (2020) · Research Synthesis Methods · doi:10.1002/jrsm.1378 · Cited by 2,379 · see citing works
Systematic Reviews: Identifying relevant studies for systematic reviews
Dickersin, K., Scherer, R. W., & Lefebvre, C. (1994) · BMJ · doi:10.1136/bmj.309.6964.1286 · Cited by 2,255 · see citing works
Huang, L., Yu, W., Ma, W., Zhong, W., Feng, Z., Wang, H., Chen, Q., Peng, W., Feng, X., Qin, B., et al. (2025) · ACM Transactions on Information Systems · doi:10.1145/3703155 · Cited by 2,142 · see citing works
A Survey of Large Language Models
Zhao, W. X., Zhou, K., Li, J., Tang, T., Wang, X., Hou, Y., Min, Y., Zhang, B., Zhang, J., Dong, Z., et al. (2023) · arXiv · doi:10.48550/arxiv.2303.18223 · Cited by 2,090
Systematic Reviews: Rationale for systematic reviews
Mulrow, C. D. (1994) · BMJ · doi:10.1136/bmj.309.6954.597 · Cited by 1,677 · see citing works
Liu, X., Cruz Rivera, S., Moher, D., Calvert, M. J., Denniston, A. K., The SPIRIT-AI and CONSORT-AI Working Group, SPIRIT-AI and CONSORT-AI Steering Group, Chan, A.-W., Darzi, A., Holmes, C., et al. (2020) · Nature Medicine · doi:10.1038/s41591-020-1034-x · Cited by 1,421 · see citing works
Time to challenge the spurious hierarchy of systematic over narrative reviews?
Greenhalgh, T., Thorne, S., & Malterud, K. (2018) · European Journal of Clinical Investigation · doi:10.1111/eci.12931 · Cited by 1,267 · see citing works
A survey on large language model (LLM) security and privacy: The Good, The Bad, and The Ugly
Yao, Y., Duan, J., Xu, K., Cai, Y., Sun, Z., & Zhang, Y. (2024) · High-Confidence Computing · doi:10.1016/j.hcc.2024.100211 · Cited by 1,083 · see citing works
A Quick Review of Machine Learning Algorithms
Ray, S. (2019) · 2019 International Conference on Machine Learning, Big Data, Cloud and Parallel Computing (COMITCon) · doi:10.1109/comitcon.2019.8862451 · Cited by 1,070 · see citing works
Prompt Engineering with ChatGPT: A Guide for Academic Writers
Giray, L. (2023) · Annals of Biomedical Engineering · doi:10.1007/s10439-023-03272-4 · Cited by 776 · see citing works
Prompt Engineering as an Important Emerging Skill for Medical Professionals: Tutorial
Meskó, B. (2023) · Journal of Medical Internet Research · doi:10.2196/50638 · Cited by 709 · see citing works
Waffenschmidt, S., Knelangen, M., Sieben, W., Bühn, S., & Pieper, D. (2019) · BMC Medical Research Methodology · doi:10.1186/s12874-019-0782-0 · Cited by 626 · see citing works
PubMed and beyond: a survey of web tools for searching biomedical literature
Lu, Z. (2011) · Database · doi:10.1093/database/baq036 · Cited by 574 · see citing works
Cooper, C., Booth, A., Varley-Campbell, J., Britten, N., & Garside, R. (2018) · BMC Medical Research Methodology · doi:10.1186/s12874-018-0545-3 · Cited by 561 · see citing works
Parums, D. V. (2021) · Medical Science Monitor · doi:10.12659/msm.934475 · Cited by 453 · see citing works
Automation of systematic literature reviews: A systematic literature review
van Dinter, R., Tekinerdogan, B., & Catal, C. (2021) · Information and Software Technology · doi:10.1016/j.infsof.2021.106589 · Cited by 432 · see citing works
Best practice guidelines for abstract screening large‐evidence systematic reviews and meta‐analyses
Polanin, J. R., Pigott, T., Espelage, D. L., & Grotpeter, J. K. (2019) · Research Synthesis Methods · doi:10.1002/jrsm.1354 · Cited by 413 · see citing works
Assessing risk of bias due to missing results in a synthesis
Page, M. J., Higgins, J. P., & Sterne, J. A. (2019) · Cochrane Handbook for Systematic Reviews of Interventions · doi:10.1002/9781119536604.ch13 · Cited by 402 · see citing works
The Science of Literature Reviews: Searching, Identifying, Selecting, and Synthesising
Chigbu, U. E., Atiku, S. O., & Plessis, C. C. D. (2023) · Publications · doi:10.3390/publications11010002 · Cited by 387 · see citing works
A review of the PubMed PICO tool: using evidence-based practice in health education
Brown, D. (2020) · Health Promotion Practice · doi:10.1177/1524839919893361 · Cited by 367 · see citing works
A Survey of GPT-3 Family Large Language Models Including ChatGPT and GPT-4
Kalyan, K. S. (2024) · Natural Language Processing Journal · doi:10.1016/j.nlp.2023.100048 · Cited by 335 · see citing works
Selecting Studies for Systemic Review: Inclusion and Exclusion Criteria
Meline, T. J. (2006) · Contemporary Issues in Communication Science and Disorders · doi:10.1044/cicsd_33_s_21 · Cited by 325 · see citing works
RobotReviewer: evaluation of a system for automatically assessing bias in clinical trials
Marshall, I. J., Kuiper, J., & Wallace, B. C. (2016) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocv044 · Cited by 317 · see citing works
A Review of Current Trends, Techniques, and Challenges in Large Language Models (LLMs)
Patil, R., & Gudivada, V. N. (2024) · Applied Sciences · doi:10.3390/app14052074 · Cited by 315 · see citing works
A Review of Current Trends, Techniques, and Challenges in Large Language Models (LLMs)
Patil, R., & Gudivada, A. V. (2024) · doi:10.20944/preprints202402.0357.v1 · Cited by 296 · see citing works
Ullah, E., Parwani, A., Baig, M. M., & Singh, R. (2024) · Diagnostic Pathology · doi:10.1186/s13000-024-01464-7 · Cited by 281 · see citing works
Frandsen, T. F., Nielsen, M. F. B., Lindhardt, C. L., & Eriksen, M. B. (2020) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2020.07.005 · Cited by 276 · see citing works
Khraisha, Q., Put, S., Kappenberg, J., Warraitch, A., & Hadfield, K. (2024) · Research Synthesis Methods · doi:10.1002/jrsm.1715 · Cited by 269 · see citing works
Khraisha, Q., Put, S., Kappenberg, J., Warraitch, A., & Hadfield, K. (2023) · arXiv · doi:10.48550/arxiv.2310.17526 · Cited by 268 · see citing works
Optimal literature search for systematic reviews in surgery
Gooßen, K., Tenckhoff, S., Probst, P., Grummich, K., Mihaljević, A. L., Büchler, M. W., & Diener, M. K. (2017) · Langenbeck s Archives of Surgery · doi:10.1007/s00423-017-1646-x · Cited by 247 · see citing works
Does Prompt Formatting Have Any Impact on LLM Performance?Preprint
He, J., Rungta, M., Koleczek, D., et al. (2024) · arXiv.org · doi:10.48550/arxiv.2411.10541 · Cited by 226 · see citing works
Kim, J. K., Chua, M., Rickard, M., & Lorenzo, A. (2023) · Journal of Pediatric Urology · doi:10.1016/j.jpurol.2023.05.018 · Cited by 225 · see citing works
Error rates of human reviewers during abstract screening in systematic reviews
Wang, Z., Nayfeh, T., Tetzlaff, J., O’Blenis, P., & Murad, M. H. (2020) · PLOS ONE · doi:10.1371/journal.pone.0227742 · Cited by 217 · see citing works
Bramer, W. M., Giustini, D., & Kramer, B. M. R. (2016) · Systematic Reviews · doi:10.1186/s13643-016-0215-7 · Cited by 202 · see citing works
White, J. (2020) · Medical Reference Services Quarterly · doi:10.1080/02763869.2020.1826228 · Cited by 197 · see citing works
Tools to support the automation of systematic reviews: a scoping review.
Khalil, H., Ameen, D., & Zarnegar, A. (2022) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2021.12.005 · Cited by 190 · see citing works
Chai, K. E. K., Lines, R. L. J., Gucciardi, D. F., & Ng, L. (2021) · Systematic Reviews · doi:10.1186/s13643-021-01635-3 · Cited by 190 · see citing works
ExaCT: automatic extraction of clinical trial characteristics from journal publications
Kiritchenko, S., de Bruijn, B., Carini, S., Martin, J., & Sim, I. (2010) · BMC Medical Informatics and Decision Making · doi:10.1186/1472-6947-10-56 · Cited by 188 · see citing works
Evaluation of query expansion using MeSH in PubMed
Lu, Z., Kim, W. K., & Wilbur, W. J. (2008) · Information Retrieval · doi:10.1007/s10791-008-9074-8 · Cited by 177 · see citing works
Searching two or more databases decreased the risk of missing relevant studies: a metaresearch study
Ewald, H., Klerings, I., Wagner, G., Heise, T. L., Stratil, J. M., Lhachimi, S. K., Hemkens, L. G., Gartlehner, G., Armijo-Olivo, S., & Nussbaumer-Streit, B. (2022) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2022.05.022 · Cited by 168 · see citing works
Dennstädt, F., Zink, J., Putora, P. M., Hastings, J., & Čihorić, N. (2024) · Systematic Reviews · doi:10.1186/s13643-024-02575-4 · Cited by 165 · see citing works
Methodology of a systematic review
Linares-Espinós, E., Hernández, V., Domínguez-Escrig, J. L., Fernandéz‐Pello, S., Hevia, V., Mayor, J., Padilla-Fernández, B. Y., & Ribal, M. J. (2018) · Actas Urológicas Españolas (English Edition) · doi:10.1016/j.acuroe.2018.07.002 · Cited by 157 · see citing works
Scherbakov, D., Hubig, N., Jansari, V., Bakumenko, A., & Lenert, L. A. (2024) · arXiv · doi:10.48550/arxiv.2409.04600 · Cited by 156 · see citing works
Lieberum, J.-L., Toews, M., Metzendorf, M.-I., Heilmeyer, F., Siemens, W., Haverkamp, C., Böhringer, D., Meerpohl, J. J., & Eisele-Metzger, A. (2025) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2025.111746 · Cited by 148 · see citing works
Recent Advances in LoRa: A Comprehensive Survey
Sun, Z., Yang, H., Liu, K., Yin, Z., Li, Z., & Xu, W. (2022) · ACM Transactions on Sensor Networks · doi:10.1145/3543856 · Cited by 145 · see citing works
PRISMA AI reporting guidelines for systematic reviews and meta-analyses on AI in healthcare
Cacciamani, G. E., Chu, T. N., Sanford, D. I., Abreu, A., Duddalwar, V., Oberai, A., Kuo, C.-C. J., Liu, X., Denniston, A. K., Vasey, B., et al. (2023) · Nature Medicine · doi:10.1038/s41591-022-02139-w · Cited by 143 · see citing works
Prompt Engineering Paradigms for Medical Applications: Scoping Review
Zaghir, J., Naguib, M., Bjelogrlic, M., Névéol, A., Tannier, X., & Lovis, C. (2024) · Journal of Medical Internet Research · doi:10.2196/60501 · Cited by 143 · see citing works
PubMed and beyond: biomedical literature search in the age of artificial intelligence
Jin, Q., Leaman, R., & Lu, Z. (2024) · EBioMedicine · doi:10.1016/j.ebiom.2024.104988 · Cited by 140 · see citing works
Rawte, V., Chakraborty, S., Pathak, A., Sarkar, A., Tonmoy, S. M. T. I., Chadha, A., Sheth, A., & Das, A. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.155 · Cited by 119 · see citing works
Prompting Is Programming: A Query Language for Large Language Models
Beurer-Kellner, L., Fischer, M., & Vechev, M. (2023) · Proceedings of the ACM on Programming Languages · doi:10.1145/3591300 · Cited by 117 · see citing works
Comparative Evaluation of LLMs in Clinical Oncology
Rydzewski, N. R., Dinakaran, D., Zhao, S. G., Ruppin, E., Türkbey, B., Citrin, D. E., & Patel, K. R. (2024) · NEJM AI · doi:10.1056/aioa2300151 · Cited by 108 · see citing works
Transformers: “The End of History” for Natural Language Processing?
Chernyavskiy, A., Ilvovsky, D., & Nakov, P. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-86523-8_41 · Cited by 98 · see citing works
Yang, M., Adomavičius, G., Burtch, G., & Ren, Y. (2018) · Information Systems Research · doi:10.1287/isre.2017.0727 · Cited by 97 · see citing works
Wu, S., Koo, M., Blum, L., Black, A., Kao, L., Fei, Z., Scalzo, F., & Kurtz, I. (2024) · NEJM AI · doi:10.1056/aidbp2300092 · Cited by 97 · see citing works
On protecting the data privacy of Large Language Models (LLMs) and LLM agents: A literature review
Yan, B., Li, K., Xu, M., Dong, Y., Zhang, Y., Ren, Z., & Cheng, X. (2025) · High-Confidence Computing · doi:10.1016/j.hcc.2025.100300 · Cited by 93 · see citing works
Understanding the Nature of Metadata: Systematic Review
Ulrich, H., Kock-Schoppenhauer, A.-K., Deppenwiese, N., Gött, R., Kern, J., Lablans, M., Majeed, R. W., Stöhr, M. R., Stausberg, J., Varghese, J., et al. (2021) · Journal of Medical Internet Research · doi:10.2196/25440 · Cited by 92 · see citing works
Trustworthy AI: Securing Sensitive Data in Large Language Models
Feretzakis, G., & Verykios, V. S. (2024) · AI · doi:10.3390/ai5040134 · Cited by 92 · see citing works
Dang, H., Mecke, L., Lehmann, F., Goller, S., & Buschek, D. (2022) · arXiv · doi:10.48550/arxiv.2209.01390 · Cited by 92 · see citing works
Transforming literature screening: The emerging role of large language models in systematic reviews
Delgado-Chaves, F. M., Jennings, M. J., Atalaia, A., Wolff, J., Horvath, R., Mamdouh, Z. M., Baumbach, J., & Baumbach, L. (2025) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2411962122 · Cited by 86 · see citing works
Barman, K. G., Wood, N. G., & Pawłowski, P. (2024) · Ethics and Information Technology · doi:10.1007/s10676-024-09778-2 · Cited by 69 · see citing works
Scells, H., Zuccon, G., Koopman, B., Deacon, A. J., Azzopardi, L. A., & Geva, S. (2017) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3132847.3133080 · Cited by 68 · see citing works
Prompt Engineering for Large Language Models
Gao, A. (2023) · doi:10.2139/ssrn.4504303 · Cited by 68 · see citing works
Development of Prompt Templates for Large Language Model–Driven Screening in Systematic Reviews
Cao, C., Sang, J., Arora, R., Chen, D., Kloosterman, R., Cecere, M., Gorla, J., Saleh, R., Drennan, I., Teja, B., et al. (2025) · Annals of Internal Medicine · doi:10.7326/annals-24-02189 · Cited by 67 · see citing works
LitSuggest: a web-based system for literature recommendation and curation using machine learning
Allot, A., Lee, K., Chen, Q., Luo, L., & Lu, Z. (2021) · Nucleic Acids Research · doi:10.1093/nar/gkab326 · Cited by 61 · see citing works
Irugalbandara, C., Mahendra, A., Daynauth, R., Arachchige, T. K., Dantanarayana, J. L., Flautner, K., Tang, L., Kang, Y., & Mars, J. (2024) · IEEE International Symposium on Performance Analysis of Systems and Software (ISPASS) · doi:10.1109/ispass61541.2024.00034 · Cited by 57 · see citing works
FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance
Chen, L., Zaharia, M., & Zou, J. (2023) · arXiv · doi:10.48550/arxiv.2305.05176 · Cited by 51 · see citing works
An Empirical Study on Challenges for LLM Application Developers
Chen, X., Gao, C., Chen, C., Zhang, G., & Liu, Y. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3715007 · Cited by 47 · see citing works
Fagbohun, O., Harrison, R. M., & Dereventsov, A. (2023) · Journal of Artificial Intelligence Machine Learning and Data Science · doi:10.51219/jaimld/oluwole-fagbohun/15 · Cited by 43 · see citing works
A Comprehensive Survey of Bias in LLMs: Current Landscape and Future Directions
Ranjan, R., Gupta, S., & Singh, S. N. (2024) · arXiv · doi:10.48550/arxiv.2409.16430 · Cited by 42 · see citing works
Established search filters may miss studies when identifying randomized controlled trials
Cooper, C., Varley‐Campbell, J., & Carter, P. (2019) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2019.04.002 · Cited by 38 · see citing works
Pichiyan, V., Muthulingam, S., Sathar, G., Nalajala, S., Cherukupally, A., & Das, M. N. (2023) · Procedia Computer Science · doi:10.1016/j.procs.2023.12.074 · Cited by 38 · see citing works
The Promise and Challenges of Using LLMs to Accelerate the Screening Process of Systematic Reviews
Huotala, A., Kuutila, M., Ralph, P., & Mäntylä, M. (2024) · International Conference on Evaluation and Assessment in Software Engineering (EASE) · doi:10.1145/3661167.3661172 · Cited by 35 · see citing works
The Evolution of Large Language Model: Models, Applications and Challenges
Sindhu, B., Prathamesh, R. P., Sameera, M. B., & Kumaraswamy, S. (2024) · International Conference on Current Trends in Advanced Computing (ICCTAC) · doi:10.1109/icctac61556.2024.10581180 · Cited by 34 · see citing works
A Literature Review on Bidirectional Encoder Representations from Transformers
Shreyashree, S., Sunagar, P., Rajarajeswari, S., & Kanavalli, A. (2022) · Lecture notes in networks and systems · doi:10.1007/978-981-16-6723-7_23 · Cited by 32 · see citing works
Methodology of Systematic Reviews and Recommendations
Furlan, J. C., Singh, J. M., Hsieh, J., & Fehlings, M. G. (2010) · Journal of Neurotrauma · doi:10.1089/neu.2009.1146 · Cited by 31 · see citing works
Receiver Operating Characteristic (ROC) Curves: The Basics and Beyond.
Chang, P. W., & Newman, T. B. (2024) · Hospital Pediatrics · doi:10.1542/hpeds.2023-007462 · Cited by 31 · see citing works
The NICE guidelines in the real world: a practical perspective
Dunning, J. (2004) · Emergency Medicine Journal · doi:10.1136/emj.2004.014944 · Cited by 29 · see citing works
Do LLMs Really Adapt to Domains? An Ontology Learning Perspective
Mai, H. T., Chu, C. X., & Paulheim, H. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-77844-5_7 · Cited by 28 · see citing works
Systematic Literature Searching and the Bibliographic Database Haystack
Page, D. (2008) · The Electronic Journal of Business Research Methods · Cited by 27 · see citing works
Decoding Methods in Neural Language Generation: A Survey
Zarrieß, S., Voigt, H., & Schüz, S. (2021) · Information · doi:10.3390/info12090355 · Cited by 26 · see citing works
Prompting Language Models for Linguistic Structure
Blevins, T., Gonen, H., & Zettlemoyer, L. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.367 · Cited by 25 · see citing works
Wu, S. M., Koo, M., Blum, L., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2308.04709 · Cited by 23 · see citing works
Automated Rule-Based Data Cleaning Using NLP
Mavrogiorgos, K., Mavrogiorgou, A., Kiourtis, A., Zafeiropoulos, N., Kleftakis, S., & Kyriazis, D. (2022) · Conference of Open Innovations Association (FRUCT) · doi:10.23919/fruct56874.2022.9953810 · Cited by 22 · see citing works
On Protecting the Data Privacy of Large Language Models (LLMs): A SurveyPreprint
Yan, B., Li, K., Xu, M., et al. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2403.05156 · Cited by 19 · see citing works
Towards a Benchmark for Scientific Understanding in Humans and Machines
Barman, K. G., Caron, S., Claassen, T., & de Regt, H. W. (2024) · Minds and Machines · doi:10.1007/s11023-024-09657-1 · Cited by 18 · see citing works
The Literature Review: Demystifying the Literature Search
Patrick, L. J., & Munro, S. (2004) · The Science of Diabetes Self-Management and Care · doi:10.1177/014572170403000106 · Cited by 17 · see citing works
Heintz, M., Hval, G., Tornes, R. A., Byelyey, N., Hafstad, E., Næss, G. E., & Bakkeli, M. (2023) · Journal of the Medical Library Association JMLA · doi:10.5195/jmla.2023.1482 · Cited by 15 · see citing works
Gosak, L., Štiglic, G., Pruinelli, L., & Vrbnjak, D. (2024) · Journal of Nursing Scholarship · doi:10.1111/jnu.13036 · Cited by 12 · see citing works
Hybrid LLM: Cost-Efficient and Quality-Aware Query Routing
Ding, D., Mallick, A., Wang, C., Sim, R., Mukherjee, S., Ruhle, V., Lakshmanan, L. V. S., & Awadallah, A. H. (2024) · arXiv · doi:10.48550/arxiv.2404.14618 · Cited by 12 · see citing works
Learning to Ask: When LLMs Meet Unclear InstructionPreprint
Wang, W., Shi, J., Wang, C.-Z., Lee, C., Yuan, Y., Huang, J.-T., & Lyu, M. R. (2024) · arXiv.org · doi:10.48550/arxiv.2409.00557 · Cited by 12 · see citing works
Beyond Clouds: Locally Runnable LLMs as a Secure Solution for AI Applications
Kumar, B. V. P., & Ahmed, M. D. S. (2024) · Digital Society · doi:10.1007/s44206-024-00141-y · Cited by 11 · see citing works
CASSAI, A. D., Dost, B., Karapınar, Y. E., Beldağlı, M., Yalin, M. S. O., Turunç, E., Turan, E. İ., & Sella, N. (2025) · Regional Anesthesia & Pain Medicine · doi:10.1136/rapm-2024-106231 · Cited by 11 · see citing works
How to Write Effective Prompts for Screening Biomedical Literature Using Large Language Models
Colangelo, M. T., Guizzardi, S., Meleti, M., Calciolari, E., & Galli, C. (2025) · BioMedInformatics · doi:10.3390/biomedinformatics5010015 · Cited by 11 · see citing works
Self-citationSoAy: A Solution-based LLM API-using Methodology for Academic Information Seeking
Wang, Y., Yu, J., Yao, Z., Zhang, J., Xie, Y., Tu, S., Fu, Y., Feng, Y., Zhang, J., Zhang, J., et al. (2025) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3690624.3709412 · Cited by 11 · see citing works
Extending Context Window of Large Language Models from a Distributional Perspective
Wu, Y., Gu, Y., Feng, X.-C., et al. (2024) · Conference on Empirical Methods in Natural Language Processing · doi:10.48550/arxiv.2410.01490 · Cited by 11
A Comparison of DeepSeek and Other LLMsPreprint
Gao, T., Jin, J., Ke, Z. T., et al. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2502.03688 · Cited by 11 · see citing works
Evaluation of Alignment: Precision, Recall, Weighting and Limitations
Cottam, J. A., Heller, N. C., Ebsch, C. L., Deshmukh, R., Mackey, P., & Chin, G. (2020) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata50022.2020.9378064 · Cited by 8 · see citing works
FirewaLLM: A Portable Data Protection and Recovery Framework for LLM Services
Huang, B., Yu, S., Li, J., Chen, Y., Huang, S., Zeng, S., & Wang, S. (2024) · Communications in computer and information science · doi:10.1007/978-981-97-0844-4_2 · Cited by 8 · see citing works
Taken out of context: On measuring situational awareness in LLMs
Berglund, L., Stickland, A. C., Balesni, M., Kaufmann, M., Tong, M., Korbak, T., Kokotajlo, D., & Evans, O. (2023) · arXiv · doi:10.48550/arxiv.2309.00667 · Cited by 8 · see citing works
Towards Optimizing the Costs of LLM UsagePreprint
Shekhar, S., Dubey, T., Mukherjee, K., et al. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2402.01742 · Cited by 8 · see citing works
Strategies to mitigate hallucinations in large language models
Bhattacharya, R. (2024) · Applied marketing analytics · doi:10.69554/nxxb8234 · Cited by 6 · see citing works
Accuracy of Large Language Models for Literature Screening in Systematic Reviews and Meta-AnalysesPreprint
Dai, Z.-Y., Shen, C., Ji, Y.-L., Li, Z., Wang, Y., & Wang, F. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.4943759 · Cited by 6 · see citing works
Fagbohun, O., Harrison, R. M., & Dereventsov, A. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2402.14837 · Cited by 5 · see citing works
Demystifying Python Package Installation with conda-env-mod
Maji, A. K., Gorenstein, L., & Lentner, G. (2020) · doi:10.1109/hustprotools51951.2020.00011 · Cited by 4 · see citing works
Stuart, D. B. (2023) · Journal of Electronic Resources in Medical Libraries · doi:10.1080/15424065.2023.2275735 · Cited by 4 · see citing works
The Risks Of Human Overreliance On Large Language Models For Critical Thinking
Duenas, T., & Ruiz, D. (2024) · doi:10.13140/rg.2.2.26002.06082 · Cited by 4 · see citing works
Comparison and Analysis of Large Language Models (LLMs)Preprint
Agarwal, L., & Nasim, A. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.4939534 · Cited by 3 · see citing works
Harnessing Python 3.11 and Python Libraries for LLM Development
Grigorov, D. (2024) · Apress eBooks · doi:10.1007/979-8-8688-0540-0_7 · Cited by 3 · see citing works
Extending Context Window of Large Language Models from a Distributional Perspective
Wu, Y., Gu, Y., Feng, X., Zhong, W., Xu, D., Yang, Q., Liu, H., & Qin, B. (2024) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2024.emnlp-main.414 · Cited by 3 · see citing works
Galli, C., Colangelo, M. T., Guizzardi, S., Meleti, M., & Calciolari, E. (2025) · Preprints.org · doi:10.20944/preprints202501.2029.v1 · Cited by 2 · see citing works
Self-citationSchiller, C. A. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2403.09743 · Cited by 2 · see citing works
Prompting Language Models for Linguistic StructurePreprint
Blevins, T., Gonen, H., & Zettlemoyer, L. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2211.07830 · Cited by 2 · see citing works
Huang, W., Poojary, V. B., Hofer, K., & Fazeli, M. S. (2024) · Value in Health · doi:10.1016/j.jval.2024.10.2450 · Cited by 1 · see citing works
An Empirical Study on Challenges for LLM Application DevelopersPreprint
Chen, X., Gao, C., Chen, C., et al. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2408.05002 · Cited by 1 · see citing works
Sumanathilaka, T. G. D. K., Micallef, N., & Hough, J. (2024) · Cronfa (Swansea University) · doi:10.48550/arxiv.2411.18337
Flagged first, then by citations
Research works printed but not linked: 3
- Ashwathy, J.S.; SR, N.; Pyati, T. The Progression of ChatGPT: An Evolutionary Study from GPT-1 to GPT-4. J. Innov. Data Sci. Big Data Manag. 2024, 3, 38-44.
- Safavi-Naini, S.A.A.; Ali, S.; Shahab, O.; Shahhoseini, Z.; Savage, T.; Rafiee, S.; Samaan, J.S.; Shabeeb, R.A.; Ladak, F.; Yang, J.O.; et al. Vision-Language and Large Language Model Performance in Gastroenterology: GPT, Claude, Llama, Phi, Mistral, Gemma, and Quantized Models. arXiv 2024, arXiv:2409.00084. · arXiv
- Gosmar, D.; Dahl, D.A. Hallucination Mitigation using Agentic AI Natural Language-Based Frameworks. arXiv 2025, arXiv:2501.13946. 100. Hassan, M. Measuring the Impact of Hallucinations on Human Reliance in LLM Applications. J. Robot. Process Autom. AI Integr. Work. Optim. 2025, 10, 10-20. 101. · arXiv
Cited by
Sorted by citations
Li, D. X., Pang, Z., Chen, Y., Yang, K., Shao, J., Luo, Y., Zeng, Y., He, C., & Gao, Y. (2025) · Mechanical Systems and Signal Processing · doi:10.1016/j.ymssp.2025.113226 · Cited by 35 · see citing works
Scotti, K. L., Young, S. M. N., Gainey, M. A., & Lan, H. (2025) · Cochrane Evidence Synthesis and Methods · doi:10.1002/cesm.70046 · Cited by 25 · see citing works
“This lack of transparency is particularly problematic for translation, where tool accuracy varies by language and content and where misinterpretation of non ‐ English studies may introduce bias [37, 256, 257].”
Balart, T., Díaz, B., & Shryock, K. (2026) · Algorithms · doi:10.3390/a19030179 · Cited by 15 · see citing works
“This process followed recent recommendations for AI-assisted synthesis workflows [43,44], ensuring transparency and consistency while maintaining full human control over final categorization and interpretation.”
Kamińska, A. M. (2026) · The Journal of Academic Librarianship · doi:10.1016/j.acalib.2026.103218 · Cited by 10 · see citing works
Nawrath, M., Merlina, A., Knight, J., Welch, S. A., Rashidian, M., & Seifert-Dähnn, I. (2026) · Information · doi:10.3390/info17050501 · Cited by 6 · see citing works
“To date, most validation studies have focused on biomedical and environmental sciences, leaving substantial gaps in other fields such as the social sciences, engineering, and interdisciplinary research, where screening criteria and evidence standards may differ markedly [7,23].”
InfluentialOami, T., Okada, Y., & Nakada, T. (2026) · Hong Kong Journal of Emergency Medicine · doi:10.1002/hkj2.70085 · Cited by 3 · see citing works
Navigating the Future of HR NLP
Asim, Z., Susanto, H., Gahramanova, S., & Vasudevan, A. (2025) · IGI Global eBooks · doi:10.4018/979-8-3373-2135-6.ch007 · Cited by 2 · see citing works
Almonacid, J. E. R., Lombardo, C., Romano, M., Quiroga, A., Bianchi, P. C., Hualpa, M., Giai, C., Oviedo, X. M. A., Mansur, R. A. S., Vallejo, M. G., et al. (2025) · Frontiers in Pharmacology · doi:10.3389/fphar.2025.1633382 · Cited by 1 · see citing works
Making Adequate Use of Generative Language Models in Scientific ResearchPreprint
Bustio-Martínez, L., Herrera-Semenets, V., Feregrino-Uribe, C., & van den Berg, J. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5367076
Motahari-Nezhad, H., & CheshmehSohrabi, M. (2025) · Research Square · doi:10.21203/rs.3.rs-7265470/v1
Do SLM agents screen papers as well as LLMs?
Filho, J. G. M. D. F., & Fernandes, S. R. (2025) · Anais do XXII Encontro Nacional de Inteligência Artificial e Computacional (ENIAC 2025) · doi:10.5753/eniac.2025.13929
“Nesse contexto, a aplicac¸˜ao de modelos de linguagem na conduc¸˜ao de RSL tornou-se uma ´area de grande interesse, como aponta [Galli et al. 2025], em especial na etapa de triagem de artigos ( screening ).”
Leitao, R., Podpečan, V., Cunha, L., Vazquez, C., David, F., Imbert, C., Symanczik, S., Sousa, J. P., Bünemann, E. K., Debeljak, M., et al. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.6097235
de la Torre-López, J., Ramírez, A., & Romero, J. R. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.6263105
Shin, W., Hong, J., Lee, S., Park, S., Kim, H. T., & Jo, S. (2026) · Journal of Clinical Medicine · doi:10.3390/jcm15082830
GenAI as a Tool for Scientists and Researchers
Hadzhikoleva, S., Hadzhikolev, E. N., & Yotov, K. (2026) · International Symposium INFOTEH-JAHORINA (INFOTEH) · doi:10.1109/infoteh68759.2026.11477705
“These activities can be significantly optimized through the selection of appropriate GenAI models, prompt customization, and manual validation [5].”
Nair, A. S., & Mistry, T. (2026) · British Journal of Anaesthesia · doi:10.1016/j.bja.2026.03.067
Vivekanantha, P., Son, H., Bernardini, L. C., Bouchard, M. D., Ayeni, O. R., & Kay, J. (2026) · Knee Surgery Sports Traumatology Arthroscopy · doi:10.1002/ksa.70462
“Existing evaluations in general medicine, oncology and rehabilitation have shown that LLMs can approximate human ‐ level inclusion decisions but frequently require extensive prompt tuning and still produce non ‐ negligible amounts of FNs [6].”
Leitao, R., Podpečan, V., Cunha, L. F. N., Vazquez, C., David, F., Imbert, C., Symanczik, S., Sousa, J. P., Bünemann, E. K., Debeljak, M., et al. (2026) · Geoderma · doi:10.1016/j.geoderma.2026.117890
Jóźwiak, Á. B., Imre, A., Józwiak‐Hagymásy, J., Tittmann, J., Nagy, Á., Kovács, S., Kardas, P., van Boven, J. F. M., Mommers, I., & Ágh, T. (2026) · Expert Review of Pharmacoeconomics & Outcomes Research · doi:10.1080/14737167.2026.2691995
“…is often the primary determinant of classification performance, yet prompt design in current practice relies predominantly on manual, iterative refinement by domain experts, a process that is time-consuming, difficult to reproduce, and unlikely to converge on globally optimal solutions [20,23,24].”
Serretti, A. (2026) · Cureus · doi:10.7759/cureus.111193
“Residual hallucination risk also remains [25,26], although schema enforcement (constrained enumerations rather than free generation) and conservative missingness handling (returning ‘NR’) were designed to minimize fabrication and performed well in audited fields.”
Lee, K. H., Kim, H., Yang, D. S., Yoon, W. S., & Rim, C. H. (2026) · Journal of Epidemiology · doi:10.2188/jea.je20260233
Holub, O., Nakonechnyi, A., Metcalf, J., & Allen, R. J. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-30813-9_3
Lai, H., & Li, Y. (2026) · Telematics and Informatics · doi:10.1016/j.tele.2026.102447
Hope in Pediatric Palliative Care: A Systematic Review
Benešová, H., Janoušková, M., & Loučka, M. (2026) · Children · doi:10.3390/children13081117
Árendás, C., Haidegger, T. P., & Zrubka, Z. (2026) · IEEE International Conference on Intelligent Engineering Systems (INES) · doi:10.1109/ines69513.2026.11661098
“While early scoping reviews were fully manual, text-mining tools and machine learning have increasingly been incorporated to prioritise abstracts, though human oversight remains required [7][19].”
Li, L., Soriano, L. D., Kedir, W. M., Hoch, F. L., & Loke, D. K. (2026) · Molecular Biomedicine · doi:10.1186/s43556-026-00498-1
“This challenge is especially pronounced when examining allosteric sites [343–345], as demonstrated in kinases and G protein-coupled receptors, along with protein–protein interactions within signalling cascades.”
Large language models for literature screening in conceptually complex and interdisciplinary reviews
Cheng, N., Dong, H., Han, X., Tan, B., & Zhu, S. (2026) · Aslib Journal of Information Management · doi:10.1108/ajim-06-2025-0363
Li, Y., Guo, J., & Shi, W. (2026) · Artificial Intelligence & Environment · doi:10.66178/aie-0026-0019
Hong, Z., Xi, Y., Zhu, A., Yu, Z., Hao, J., Wang, F., Wei, C., He, C., Zhang, L., Liu, G., et al. (2026) · Diversity · doi:10.3390/d18100600
Merchán-Cruz, E. A., Gabelaia, I., Soe, S. P., Yatskiv, I., & Dmitry, P. (2026) · Informatics · doi:10.3390/informatics13100160
Yahia, A. B., Harrak, E. F. E., Kadir, I., Chisembe, S., Abdallaoui, A., Hmaidi, A. E., & Dehbi, A. (2026) · Array · doi:10.1016/j.array.2026.101321
Sati, A., Hosny, F., Mantha, B. R. K., Dabous, S. A., & Al-Ruzouq, R. I. (2026) · Journal of Computing in Civil Engineering · doi:10.1061/jccee5.cpeng-7708
Pirani, E., Costazza, D., & Berti, P. P. (2026) · medRxiv · doi:10.64898/2026.10.05.26364147
Pietrancosta, M., Wang, S., Wu, Y., Brulotte, V., Harutyunyan, R., Hemmerling, T. M., Morisson, L., & Laferrière-Langlois, P. (2026) · Research Square · doi:10.21203/rs.3.rs-10879359/v1
AI–Assisted Process for Hospital New Drug Inclusion: Development and Validation (Preprint)
Yang, Z., Wu, A., Huang, F., Zhong, G., Zhu, H., Xie, H., He, J. L., Lu, J., Zhong, J., Lan, J., et al. (2026) · JMIR Medical Informatics · doi:10.2196/93972
- Authors
- Galli, C., Gavrilova, A. V., & Calciolari, E.
- Affiliations
- University of Parma (IT)
- University of Milan (IT)
- Corresponding author
- Galli, Carlo
- Venue
- Information
- Publisher
- MDPI AG
- Published
- 2025-05-01
- Volume
- 16
- Issue
- 5
- Pages
- 378
- ISSN
- 2078-2489 (ISSN-L)
- OpenAlex
- W4410000915
- Cited by
- 36
- References
- 102
- Citations by year
- 2026: 29 · 2025: 7
- FWCI
- 6.82
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- Information (publishedVersion, cc-by)
- Preprints.org (acceptedVersion, cc-by)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › General Social Sciences › Computational and Text Analysis Methods
- Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
- Keywords
- large language models, systematic review screening, abstract classification, GPT-4, prompt engineering, human oversight, zero-shot classification, few-shot learning, reproducibility, cost management, biomedical literature
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
