Can ChatGPT Write a Good Boolean Query for Systematic Review Literature Search?
Open accessAbstract
Systematic reviews are comprehensive literature reviews for a highly focused research question. These reviews are considered the highest form of evidence in medicine. Complex Boolean queries are developed as part of the systematic review creation process to retrieve literature, as they permit reproducibility and understandability. However, it is difficult and time-consuming to develop high-quality Boolean queries, often requiring the expertise of expert searchers like librarians. Recent advances in transformer-based generative models have shown their ability to effectively follow user instructions and generate answers based on these instructions. In this paper, we investigate ChatGPT as a means for automatically formulating and refining complex Boolean queries for systematic review literature search. Overall, our research finds that ChatGPT has the potential to generate effective Boolean queries. The ability of ChatGPT to follow complex instructions and generate highly precise queries makes it a tool of potential value for researchers conducting systematic reviews, particularly for rapid reviews where time is a constraint and where one can trade off higher precision for lower recall. We also identify several caveats in using ChatGPT for this task, highlighting that this technology needs further validation before it is suitable for widespread uptake.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Applications of large language models in systematic literature reviews include construction of literature search terms, article screening, data extraction, and article content synthesis/analysis.
“Applications of LLMs in SLRs include construction of literature search terms (Alshami et al., 2023; Wang Z. et al., 2025; Wang et al., 2023), article screening (Akinseloyin et al., 2024; Alshami et al., 2023; Guo et al., 2024; Khraisha et al., 2024; Kohandel Gargari et al., 2024; Landschaft et al., 2024; Li et al., 2024; Matsui et al., 2024; Syriani et al., 2023; Tran et al., 2024; Wang Z. et al.,”
From Analysis of article screening and data extraction performance by an AI systematic literature review platform (2025) · cites this paper as Wang et al., 2023
Cites
Prompt Programming for Large Language Models: Beyond the Few-Shot Paradigm
Reynolds, L., & McDonell, K. (2021) · Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411763.3451760 · Cited by 952 · cited 1× · see citing works
“In some respect, prompt engineering can be seen as a way to program a generative language model through natural language [53].”
BackgroundDesign Guidelines for Prompt Engineering Text-to-Image Generative Models
Liu, V., & Chilton, L. B. (2022) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3491102.3501825 · Cited by 639 · cited 1× · see citing works
“A popular way of guiding model output through prompt engineering is for text-to-image generative models [37, 48].”
BackgroundReducing Workload in Systematic Review Preparation Using Automated Citation Classification
Cohen, A. M., Hersh, W. R., Peterson, K., & Yen, P.-Y. (2006) · Journal of the American Medical Informatics Association · doi:10.1197/jamia.m1929 · Cited by 411 · see citing works
ChatGPT is not all you need. A state-of-the-art review of large generative AI models
Gozalo-Brizuela, R., & Garrido-Merchan, E. C. (2023) · arXiv · doi:10.48550/arxiv.2301.04655 · Cited by 245 · see citing works
Automatic classification of sentences to support Evidence Based Medicine.
Kim, S. N., Martinez, D., Cavedon, L., & Yencken, L. (2011) · BMC Bioinformatics · doi:10.1186/1471-2105-12-s2-s5 · Cited by 214 · see citing works
Automatic text classification to support systematic reviews in medicine
García Adeva, J. J., Pikatza Atxa, J. M., Ubeda Carrillo, M., & Ansuategi Zengotitabengoa, E. (2014) · Expert Systems with Applications · doi:10.1016/j.eswa.2013.08.047 · Cited by 125 · see citing works
Rapid reviews may produce different results to systematic reviews: a meta-epidemiological study
Marshall, I. J., Marshall, R., Wallace, B. C., Brassey, J., & Thomas, J. (2019) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2018.12.015 · Cited by 95 · see citing works
Advanced analytics for the automation of medical systematic reviews
Timsina, P., Liu, J., & El-Gayar, O. (2016) · Information Systems Frontiers · doi:10.1007/s10796-015-9589-7 · Cited by 33 · see citing works
The Impact of Query Refinement on Systematic Review Literature Search: A Query Log Analysis
Scells, H., Forbes, C., Clark, J., Koopman, B., & Zuccon, G. (2022) · Proceedings of the 2022 ACM SIGIR International Conference on Theory of Information Retrieval · doi:10.1145/3539813.3545143 · Cited by 12 · see citing works
Self-citation
Flagged first, then by citations
Cited by
Sorted by citations
Ray, P. P. (2023) · Internet of Things and Cyber-Physical Systems · doi:10.1016/j.iotcps.2023.04.003 · Cited by 2,463 · see citing works
Summary of ChatGPT-Related research and perspective towards the future of large language models
Liu, Y., Han, T., Ma, S., Zhang, J., Yang, Y., Tian, J., He, H., Li, A., He, M., Liu, Z., et al. (2023) · Meta-Radiology · doi:10.1016/j.metrad.2023.100017 · Cited by 855 · see citing works
Opportunities and Challenges for ChatGPT and Large Language Models in Biomedicine and Health
Tian, S., Jin, Q., Yeganova, L., Lai, P.-T., Zhu, Q., Chen, X., Yang, Y., Chen, Q., Kim, W., Comeau, D. C., et al. (2023) · Briefings in Bioinformatics · doi:10.1093/bib/bbad493 · Cited by 359 · see citing works
Khraisha, Q., Put, S., Kappenberg, J., Warraitch, A., & Hadfield, K. (2023) · arXiv · doi:10.48550/arxiv.2310.17526 · Cited by 268 · see citing works
A comprehensive survey of ChatGPT: Advancements, applications, prospects, and challenges
Nazir, A., & Wang, Z. (2023) · Meta-Radiology · doi:10.1016/j.metrad.2023.100022 · Cited by 245 · see citing works
Khraisha, Q., Put, S., Kappenberg, J., Warraitch, A., & Hadfield, K. (2024) · Research Synthesis Methods · doi:10.1002/jrsm.1715 · Cited by 269 · see citing works
Is chatGPT a good nlg evaluator? A preliminary study
Wang, J., Liang, Y., Meng, F., Sun, Z., Shi, H., Li, Z., Xu, J., Qu, J., & Zhou, J. (2023) · Proceedings of the 4th New Frontiers in Summarization Workshop · doi:10.18653/v1/2023.newsum-1.1 · Cited by 235 · see citing works
Screening articles for systematic reviews with ChatGPT
Syriani, E., David, I., & Kumar, G. (2024) · Journal of Computer Languages · doi:10.1016/j.cola.2024.101287 · Cited by 82 · see citing works
Salih, S., Husain, O., Hamdan, M., Abdelsalam, S., Elshafie, H., & Motwakel, A. (2025) · Results in Engineering · doi:10.1016/j.rineng.2024.103837 · Cited by 74 · see citing works
Accelerating clinical evidence synthesis with large language models
Wang, Z., Cao, L., Danek, B., Jin, Q., Lu, Z., & Sun, J. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-01840-7 · Cited by 69 · see citing works
Sumbal, A., Sumbal, R., & Amir, A. (2024) · Journal of Medical Education and Curricular Development · doi:10.1177/23821205241238641 · Cited by 63 · see citing works
A Survey on Student Use of Generative AI Chatbots for Academic Research
Deschenes, A., & McMahon, M. (2024) · Evidence Based Library and Information Practice · doi:10.18438/eblip30512 · Cited by 49 · see citing works
Applying ChatGPT and AI‐Powered Tools to Accelerate Evidence Reviews
Nguyen-Trung, K., Saeri, A. K., & Kaufman, S. (2024) · Human Behavior and Emerging Technologies · doi:10.1155/2024/8815424 · Cited by 47 · see citing works
Mastering medical terminology with ChatGPT and Termbot
Hsu, M.-H. (2024) · Health Education Journal · doi:10.1177/00178969231197371 · Cited by 47 · see citing works
Mahmoudi, H., Chang, D., Lee, H., Ghaffarzadegan, N., & Jalali, M. S. (2025) · JMIR AI · doi:10.2196/68097 · Cited by 35 · see citing works
Bonsu, E., & Baffour-Koduah, D. (2023) · doi:10.2139/ssrn.4387107 · Cited by 26 · see citing works
O’Connor, A. M., Clark, J., Thomas, J., Spijker, R., Kusa, W., Walker, V. R., & Bond, M. (2024) · Systematic Reviews · doi:10.1186/s13643-024-02666-2 · Cited by 21 · see citing works
Cassell, K., Ologunowa, A., Rastegar-Mojarad, M., Chun, B., Huang, Y.-L., Wang, D., & Cossrow, N. (2025) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2025.1662202 · Cited by 6 · see citing works
Zero-shot Generative Large Language Models for Systematic Review Screening Automation
Wang, S., Scells, H., Zhuang, S., Potthast, M., Koopman, B., & Zuccon, G. (2024) · arXiv · doi:10.48550/arxiv.2401.06320 · Cited by 5 · see citing works
Self-citation
- Authors
- Wang, S., Scells, H., Koopman, B., & Zuccon, G.
- Affiliations
- The University of Queensland (AU)
- Leipzig University (DE)
- Commonwealth Scientific and Industrial Research Organisation (AU)
- Venue
- Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval
- Publisher
- ACM
- Published
- 2023-07-18
- Pages
- 1426-1436
- OpenAlex
- W4384641573
- Cited by
- 275
- References
- 9
- Citations by year
- 2026: 46 · 2025: 63 · 2024: 107 · 2023: 59
- FWCI
- 17.11
- Type
- journal article
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) (publishedVersion)
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Funders
- Australian Research Council (DP210104043)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Keywords
- ChatGPT, Boolean queries, query generation, large language models, rapid review, precision-recall trade-off, reproducibility, bibliographic database searching
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
