Retrieval-Augmented Generation–Enabled GPT-4 for Clinical Trial Screening
Open accessPublished version of a preprint
The DOI matches its Crossref record; 12 of the 12 references it cites were found in a registry or an index.
Abstract
Background: Subject screening is a key aspect of all clinical trials; however, traditionally, it is a labor-intensive and error-prone task, demanding significant time and resources. With the advent of large language models (LLMs) and related technologies, a paradigm shift in natural language processing capabilities offers a promising avenue for increasing both quality and efficiency of screening efforts. This study aimed to test the Retrieval-Augmented Generation (RAG) process enabled Generative Pretrained Transformer Version 4 (GPT-4) to accurately identify and report on inclusion and exclusion criteria for a clinical trial. Methods: The Co-Operative Program for Implementation of Optimal Therapy in Heart Failure (COPILOT-HF) trial aims to recruit patients with symptomatic heart failure. As part of the screening process, a list of potentially eligible patients is created through an electronic health record (EHR) query. Currently, structured data in the EHR can only be used to determine 5 out of 6 inclusion and 5 out of 17 exclusion criteria. Trained, but non-licensed, study staff complete manual chart review to determine patient eligibility and record their assessment of the inclusion and exclusion criteria. We obtained the structured assessments completed by the study staff and clinical notes for the past two years and developed a workflow of clinical note-based question answering system powered by RAG architecture and GPT-4 that we named RECTIFIER (RAG-Enabled Clinical Trial Infrastructure for Inclusion Exclusion Review). We used notes from 100 patients as a development dataset, 282 patients as a validation dataset, and 1894 patients as a test set. An expert clinician completed a blinded review of patients' charts to answer the eligibility questions and determine the "gold standard" answers. We calculated the sensitivity, specificity, accuracy, and Matthews correlation coefficient (MCC) for each question and screening method. We also performed bootstrapping to calculate the confidence intervals for each statistic. Results: Both RECTIFIER and study staff answers closely aligned with the expert clinician answers across criteria with accuracy ranging between 97.9% and 100% (MCC 0.837 and 1) for RECTIFIER and 91.7% and 100% (MCC 0.644 and 1) for study staff. RECTIFIER performed better than study staff to determine the inclusion criteria of "symptomatic heart failure" with an accuracy of 97.9% vs 91.7% and an MCC of 0.924 vs 0.721, respectively. Overall, the sensitivity and specificity of determining eligibility for the RECTIFIER was 92.3% (CI) and 93.9% (CI), and study staff was 90.1% (CI) and 83.6% (CI), respectively. Conclusion: GPT-4 based solutions have the potential to improve efficiency and reduce costs in clinical trial screening. When incorporating new tools such as RECTIFIER, it is important to consider the potential hazards of automating the screening process and set up appropriate mitigation strategies such as final clinician review before patient engagement.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Chicco, D., & Jurman, G. (2020) · BMC Genomics · doi:10.1186/s12864-019-6413-7 · Cited by 6,143
Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., Küttler, H., Lewis, M., Yih, W.-T., Rocktäschel, T., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.11401 · Cited by 3,101
Recent Advances in Natural Language Processing via Large Pre-trained Language Models: A Survey
Min, B., Ross, H., Sulem, E., Veyseh, A. P. B., Nguyen, T. H., Sainz, O., Agirre, E., Heintz, I., & Roth, D. (2024) · ACM Computing Surveys · doi:10.1145/3605943 · Cited by 1,387
Chicco, D., Tötsch, N., & Jurman, G. (2021) · BioData Mining · doi:10.1186/s13040-021-00244-z · Cited by 893
Zero-Shot Learning — The Good, the Bad and the Ugly
Xian, Y., Schiele, B., & Akata, Z. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.328 · Cited by 891
Matching patients to clinical trials with large language models
Jin, Q., Wang, Z., Floudas, C. S., Chen, F., Gong, C., Bracken-Clarke, D., Xue, E., Yang, Y., Sun, J., & Lu, Z. (2024) · Nature Communications · doi:10.1038/s41467-024-53081-z · Cited by 287
Matching Patients to Clinical Trials with Large Language Models
Jin, Q., Wang, Z., Floudas, C. S., Chen, F., Gong, C., Bracken-Clarke, D., Xue, E., Yang, Y., Sun, J., & Lu, Z. (2023) · arXiv · doi:10.48550/arxiv.2307.15051 · Cited by 287
Criteria2Query: a natural language interface to clinical databases for cohort definition
Yuan, C., Ryan, P. B., Ta, C., Guo, Y., Li, Z., Hardin, J., Makadia, R., Jin, P., Shang, N., Kang, T., et al. (2019) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocy178 · Cited by 170
Crowdsourcing Ground Truth for Medical Relation Extraction
Dumitrache, A., Aroyo, L., & Welty, C. (2018) · ACM Transactions on Interactive Intelligent Systems · doi:10.1145/3152889 · Cited by 81
Zero-shot interpretable phenotyping of postpartum hemorrhage using large language models
Alsentzer, E., Rasmussen, M. J., Fontoura, R., Cull, A. L., Beaulieu-Jones, B., Gray, K. J., Bates, D. W., & Kovacheva, V. P. (2023) · npj Digital Medicine · doi:10.1038/s41746-023-00957-x · Cited by 69
Zero-Shot Clinical Trial Patient Matching with LLMs
Wornow, M., Lozano, A., Dash, D., Jindal, J., Mahaffey, K. W., & Shah, N. H. (2025) · NEJM AI · doi:10.1056/aics2400360 · Cited by 59
Zero-Shot Clinical Trial Patient Matching with LLMs
Wornow, M., Lozano, A., Dash, D., Jindal, J., Mahaffey, K. W., & Shah, N. H. (2024) · arXiv · doi:10.48550/arxiv.2402.05125 · Cited by 59
Research works printed but not linked: 18
- 10.3233/SHTI220156 · DOI
- 10.1177/1740774514538706 · DOI
- 10.1093/jamia/ocab228 · DOI
- 10.1136/amiajnl-2011-000321 · DOI · Paper page
- 10.1093/jamia/ocx019 · DOI · Paper page
- Gao J Xiao C Glass LM et al. COMPOSE: Cross-modal pseudo-Siamese network for patient trial matching. KDD ‘20: Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining 2020 Aug;803-812. DOI: 10.1145/3394486.3403123. · DOI · Paper page
- Zhang X Xiao C Glass LM et al. DeepEnroll: patient-trial matching with deep embedding and entailment prediction. WWW ‘20: Proceedings of The Web Conference 2020 Apr;1029-1037. DOI: 10.1145/3366423.3380181. · DOI · Paper page
- Open AI Achiam J Adler S et al. GPT-4 Technical Report. March 15 2023 (http://arxiv.org/abs/2303.08774). Preprint. · arXiv
- 10.1038/s41591-023-02448-8 · DOI · Paper page
- den Hamer DM Schoor P Polak TB et al. Improving patient pre-screening for clinical trials: assisting physicians with large language models. April 14 2023 (http://arxiv.org/abs/2304.07396). Preprint. · arXiv
- 10.1001/jamanetworkopen.2023.35924 · DOI · Paper page
- Goel A Gueta A Gilon O et al. LLMs accelerate annotation for medical information extraction. December 4 2023 (http://arxiv.org/abs/2312.02296). Preprint. · arXiv
- 10.1161/circ.148.suppl_1.16521 · DOI
- 10.1055/s-0041-1739195 · DOI
- Neelakantan A Xu T Puri R et al. Text and code embeddings by contrastive pre-training. January 24 2022 (http://arxiv.org/abs/2201.10005). Preprint. · arXiv
- 10.1200/JOP.2012.000646 · DOI · Paper page
- 10.1093/jnci/dju002 · DOI · Paper page
- 10.1016/j.jclinepi.2013.12.005 · DOI
Cited by
Retrieval augmented generation for large language models in healthcare: A systematic review
Amugongo, L. M., Mascheroni, P., Brooks, S., Doering, S., & Seidel, J. (2025) · PLOS Digital Health · doi:10.1371/journal.pdig.0000877 · Cited by 267
“Another study used a multi-layered approach to ensure data privacy and security when using Azure OpenAI for sensitive healthcare data [74].”
Ng, K. K. Y., Matsuba, I., & Zhang, P. C. (2025) · NEJM AI · doi:10.1056/aira2400380 · Cited by 107
Wang, D., Liang, J., Ye, J., Li, J., Li, J., Zhang, Q., Hu, Q., Pan, C., Wang, D., Liu, Z., et al. (2024) · Journal of Medical Internet Research · doi:10.2196/58041 · Cited by 76
Zero-Shot Clinical Trial Patient Matching with LLMs
Wornow, M., Lozano, A., Dash, D., Jindal, J., Mahaffey, K. W., & Shah, N. H. (2025) · NEJM AI · doi:10.1056/aics2400360 · Cited by 59
You, J. G., Hernandez-Boussard, T., Pfeffer, M. A., Landman, A., & Mishuris, R. G. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-01506-4 · Cited by 59
Metadata
- Authors
- Unlu, O., Shin, J., Mailly, C. J., Oates, M. F., Tucci, M. R., Varugheese, M., Wagholikar, K., Wang, F., Scirica, B. M., Blood, A. J., & Aronson, S. J.
- Venue
- NEJM AI
- Publisher
- Massachusetts Medical Society
- Published
- 2024-06-27
- Volume
- 1
- Issue
- 7
- ISSN
- 2836-9386 (ISSN-L)
- OpenAlex
- W4399738410
- Cited by
- 114
- References
- 12
- FWCI
- 9.14
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.