Utilizing Large Language Models for Enhanced Clinical Trial Matching: A Study on Automation in Patient Screening
Open accessPublished version of a preprint
Abstract
Background Clinical trial matching, essential for advancing medical research, involves detailed screening of potential participants to ensure alignment with specific trial requirements. Research staff face challenges due to the high volume of eligible patients and the complexity of varying eligibility criteria. The traditional manual process, both time-consuming and error-prone, often leads to missed opportunities. Recently, large language models (LLMs), specifically generative pre-trained transformers (GPTs), have become impressive and impactful tools. Utilizing such tools from artificial intelligence (AI) and natural language processing (NLP) may enhance the accuracy and efficiency of this process through automated patient screening against established criteria. Methods Utilizing data from the National NLP Clinical Challenges (n2c2) 2018 Challenge, we utilized 202 longitudinal patient records. These records were annotated by medical professionals and evaluated against 13 selection criteria encompassing various health assessments. Our approach involved embedding medical documents into a vector database to determine relevant document sections and then using an LLM (OpenAI's GPT-3.5 Turbo and GPT-4) in tandem with structured and chain-of-thought prompting techniques for systematic document assessment against the criteria. Misclassified criteria were also examined to identify classification challenges. Results This study achieved an accuracy of 0.81, sensitivity of 0.80, specificity of 0.82, and a micro F1 score of 0.79 using GPT-3.5 Turbo, and an accuracy of 0.87, sensitivity of 0.85, specificity of 0.89, and micro F1 score of 0.86 using GPT-4. Notably, some criteria in the ground truth appeared mislabeled, an issue we couldn't explore further due to insufficient label generation guidelines on the website. Conclusion Our findings underscore the potential of AI and NLP technologies, including LLMs, in the clinical trial matching process. The study demonstrated strong capabilities in identifying eligible patients and minimizing false inclusions. Such automated systems promise to alleviate the workload of research staff and improve clinical trial enrollment, thus accelerating the process and enhancing the overall feasibility of clinical research. Further work is needed to determine the potential of this approach when implemented on real clinical data.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Similar errors to those observed in our work have been reported, such as incorrectly identifying patients who meet partial criteria, or assuming a patient with breast cancer does not have lung cancer simply because it is not explicitly mentioned.
“Similar errors to those observed in our work have been reported, such as incorrectly identifying patients who meet partial criteria, or assuming a patient with breast cancer does not have lung cancer simply because it is not explicitly mentioned [36,37].”
From Identifying Deprescribing Opportunities With Large Language Models in Older Adults: Retrospective Cohort Study (2025) · cites this paper as [36]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372 · see citing works
Kaugeranna, R., Kaugeranna, E., & Aion, (. S. 4. 6. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.giscience.2023.43 · Cited by 13,695 · see citing works
Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q., & Zhou, D. (2022) · arXiv · doi:10.48550/arxiv.2201.11903 · Cited by 4,263 · see citing works
Llama 2: Open Foundation and Fine-Tuned Chat Models
Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. (2023) · arXiv · doi:10.48550/arxiv.2307.09288 · Cited by 2,604 · see citing works
Large Language Models are Zero-Shot Reasoners
Kojima, T., Gu, S. S., Reid, M., Matsuo, Y., & Iwasawa, Y. (2022) · arXiv · doi:10.48550/arxiv.2205.11916 · Cited by 2,013 · see citing works
Large language models are zero-shot reasoners
Kojima, T., Gu, S. (., Reid, M., Matsuo, Y., & Iwasawa, Y. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-1613 · Cited by 2,013 · see citing works
Self-Consistency Improves Chain of Thought Reasoning in Language Models
Wang, X., Wei, J., Schuurmans, D., Le, Q., Chi, E., Narang, S., Chowdhery, A., & Zhou, D. (2022) · arXiv · doi:10.48550/arxiv.2203.11171 · Cited by 829 · see citing works
Prompt engineering for healthcare: Methodologies and applications
Wang, J., Shi, E., Yu, S., Wu, Z., Hu, H., Ma, C., Dai, H., Yang, Q., Kang, Y., Wu, J., et al. (2025) · Meta-Radiology · doi:10.1016/j.metrad.2025.100190 · Cited by 219 · see citing works
Adult cancer clinical trials that fail to complete: an epidemic?
Stensland, K. D., McBride, R. B., Latif, A., Wisnivesky, J., Hendricks, R., Roper, N., Boffetta, P., Hall, S. J., Oh, W. K., & Galsky, M. D. (2014) · JNCI Journal of the National Cancer Institute · doi:10.1093/jnci/dju229 · Cited by 161 · see citing works
ChatIE: Zero-Shot Information Extraction via Chatting with ChatGPTPreprint
Wei, X., Cui, X., Cheng, N., Wang, X., Zhang, X., Huang, S., Xie, P., Xu, J., Chen, Y., Zhang, M., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2302.10205 · Cited by 157 · see citing works
Self-Discover: Large Language Models Self-Compose Reasoning StructuresPreprint
Zhou, P., Pujara, J., Ren, X., et al. (2024) · Neural Information Processing Systems · doi:10.48550/arxiv.2402.03620 · Cited by 156 · see citing works
Prompt Engineering for Healthcare: Methodologies and Applications
Wang, J., Shi, E., Yu, S., Wu, Z., Ma, C., Dai, H., Yang, Q., Kang, Y., Wu, J., Hu, H., et al. (2023) · arXiv · doi:10.48550/arxiv.2304.14670 · Cited by 150 · see citing works
Ni, Y., Wright, J., Perentesis, J., Lingren, T., Deleger, L., Kaiser, M., Kohane, I., & Solti, I. (2015) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-015-0149-3 · Cited by 142 · see citing works
Global Public Attitudes About Clinical Research and Patient Experiences With Clinical Trials
Anderson, A., Borfitz, D., & Getz, K. A. (2018) · JAMA Network Open · doi:10.1001/jamanetworkopen.2018.2969 · Cited by 134 · see citing works
Effort Required in Eligibility Screening for Clinical Trials
Penberthy, L. T., Dahman, B. A., Petkov, V. I., & DeShazo, J. P. (2012) · Journal of Oncology Practice · doi:10.1200/jop.2012.000646 · Cited by 96 · see citing works
Cohort selection for clinical trials: n2c2 2018 shared task track 1
Stubbs, A., Filannino, M., Soysal, E., Henry, S., & Uzuner, Ö. (2019) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocz163 · Cited by 73 · see citing works
Oleynik, M., Kugic, A., Kasáč, Z., & Kreuzthaler, M. (2019) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocz149 · Cited by 54 · see citing works
Large Language Models for Healthcare Data Augmentation: An Example on Patient-Trial Matching
Yuan, J., Tang, R., Jiang, X., & Xia, H. (2023) · PubMed Central · doi:10.48550/arxiv.2303.16756 · Cited by 32 · see citing works
Hybrid bag of approaches to characterize selection criteria for cohort identification
Vydiswaran, V. G. V., Strayhorn, A., Zhao, X., Robinson, P., Agarwal, M., Bagazinski, E., Essiet, M., Iott, B. E., Joo, H., Ko, P., et al. (2019) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocz079 · Cited by 22 · see citing works
SELF-DISCOVER: Large Language Models Self-Compose Reasoning Structures
Zhou, P., Pujara, J., Ren, X., Chen, X., Cheng, H.-T., Le, Q. V., Ed, H., Zhou, D., Mishra, S. R., & Zheng, H. (2024) · neural information processing systems · doi:10.52202/079017-4004 · Cited by 17 · see citing works
Barriers to Clinical Trial Accrual: Perspectives of Community-Based Providers
Knelson, L. P., Cukras, A. R., Savoie, J., Agarwal, A., Guo, H., Hu, J., Fell, G. G., Lederman, R. I., Hughes, M. E., Winer, E. P., et al. (2020) · Clinical Breast Cancer · doi:10.1016/j.clbc.2020.05.001 · Cited by 16 · see citing works
Durden, K., Hurley, P., Butler, D. L., Farner, A., Shriver, S. P., & Fleury, M. E. (2023) · Cancer · doi:10.1002/cncr.35044 · Cited by 14 · see citing works
Increasing Clinical Trial Accrual via Automated Matching of Biomarker Criteria
Chen, J. W., Kunder, C. A., Bui, N. Q., Zehnder, J. L., Costa, H. A., & Stehr, H. (2019) · Pacific Symposium on Biocomputing (PSB) · doi:10.1142/9789811215636_0004 · Cited by 8 · see citing works
OpenAI, Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., et al. (2023) · arXiv · doi:10.48550/arxiv.2303.08774
Flagged first, then by citations
Cited by
Sorted by citations
Natural Language Processing in medicine and ophthalmology: A review for the 21st-century clinician
Rojas-Carabali, W. A., Agrawal, R., Gutiérrez-Sinisterra, L., Baxter, S. L., Cifuentes‐González, C., Wei, Y. C., Abisheganaden, J. A., Kannapiran, P., Wong, S., Lee, B. T. K., et al. (2024) · Asia-Pacific Journal of Ophthalmology · doi:10.1016/j.apjo.2024.100084 · Cited by 55 · see citing works
AI for Data Quality Auditing: Detecting Mislabeled Work Zone Crashes Using Large Language Models
Jaradat, S., Acharya, N. K., Shivshankar, S., Alhadidi, T. I., & Elhenawy, M. M. Z. (2025) · Algorithms · doi:10.3390/a18060317 · Cited by 44 · see citing works
Prescribing the Future: The Role of Artificial Intelligence in Pharmacy
Allam, H. (2025) · Information · doi:10.3390/info16020131 · Cited by 43 · see citing works
Perspectives and Experiences With Large Language Models in Health Care: Survey Study
Sumner, J., Wang, Y., Tan, S. Y., Chew, E. H. H., & Yip, A. W. (2025) · Journal of Medical Internet Research · doi:10.2196/67383 · Cited by 24 · see citing works
AI meets informed consent: a new era for clinical trial communication
Waters, M. R. (2025) · JNCI Cancer Spectrum · doi:10.1093/jncics/pkaf028 · Cited by 26 · see citing works
Generative Artificial Intelligence in Medicine
Shanmugam, D., Agrawal, M., Movva, R., Chen, I. Y., Ghassemi, M., Jacobs, M. L., & Pierson, E. (2025) · Annual Review of Biomedical Data Science · doi:10.1146/annurev-biodatasci-103123-095332 · Cited by 34 · see citing works
Owens, D., Nguyen, D. Q., Dohopolski, M. J., Rousseau, J. F., Peterson, E. D., & Návar, A. M. (2025) · Stroke · doi:10.1161/strokeaha.125.051993 · Cited by 19 · see citing works
Self-citationSocrates, V., Wright, D. S., Huang, T., Fereydooni, S., Dien, C., Chi, L., Albano, J., Patterson, B., Sasidhar Kanaparthy, N., Wright, C. X., et al. (2025) · JMIR Aging · doi:10.2196/69504 · Cited by 17 · see citing works
Meyer, B., Kfuri‐Rubens, R., Schmidt, G., Tariq, M., Riedel, C., Recker, F., Riedel, F., Kiechle, M., & Riedel, M. (2025) · International Journal of Gynecology & Obstetrics · doi:10.1002/ijgo.70251 · Cited by 10 · see citing works
Chen, H., Li, X., He, X., Chen, A., McGill, J. M., Webber, E. C., Xu, H., Liu, M., & Bian, J. (2025) · JCO Clinical Cancer Informatics · doi:10.1200/cci-25-00071 · Cited by 14 · see citing works
Mahyoub, M. A., Dougherty, K., & Shukla, A. (2025) · JMIR Medical Informatics · doi:10.2196/67706 · Cited by 7 · see citing works
Large language models for automating clinical trial matching
Layne, E., Olivas, C. G., Hershenhouse, J. S., Ganjavi, C., Cei, F., Gill, I. S., & Cacciamani, G. E. (2025) · Current Opinion in Urology · doi:10.1097/mou.0000000000001281 · Cited by 6 · see citing works
ChatGPT as a decision-support tool for better self-monitoring of hearing
Pastucha, M., Ratuszniak, A., Ganc, M., Piłka, E., Drohobycka, I., Skarżyński, H., & Jędrzejczak, W. W. (2025) · American Journal of Otolaryngology · doi:10.1016/j.amjoto.2025.104711 · Cited by 5 · see citing works
Devi, A., Uttrani, S., Singla, A., Jha, S., Dasgupta, N., Natarajan, S., Punekar, R. S., Pickett, L. A., & Dutt, V. (2024) · International Conference on PErvasive Technologies Related to Assistive Environments (PETRA) · doi:10.1145/3652037.3663922 · Cited by 7 · see citing works
Yigamawano, F. K., Odom-Mabey, A. R., Xue, C., Pandey, H. K., & Kolachalama, V. B. (2025) · The Journal of Prevention of Alzheimer s Disease · doi:10.1016/j.tjpad.2025.100396 · Cited by 4 · see citing works
Geraci, J., Rao, P., Grandinetti, C., Qorri, B., Nadolny, P., Ayalew, K., Bregnhøj, L., Edwards, L., Hofmann, K., Khozin, S., et al. (2025) · Journal of the Society for Clinical Data Management · doi:10.47912/jscdm.426 · Cited by 4 · see citing works
Huh, K. Y., Song, I., Kim, Y., Park, J., Ryu, H., Koh, J., Yu, K., Kim, K. H., & Lee, S. (2025) · Clinical and Translational Science · doi:10.1111/cts.70183 · Cited by 3 · see citing works
Large Language Models Diagnose Facial DeformityPreprint
Lee, J., Xu, X., Kim, D., Deng, H., Kuang, T., Lampen, N., Fang, X. F., Gateño, J., & Yan, P. (2024) · medRxiv · doi:10.1101/2024.07.11.24310274 · Cited by 3 · see citing works
Large Language Model Augmented Clinical Trial ScreeningPreprint
Beattie, J., Owens, D., Návar, A. M., Schmitt, L. G., Taing, K., Neufeld, S., Yang, D. Z., Chukwuma, C., Gul, A., Lee, D. S., et al. (2024) · medRxiv · doi:10.1101/2024.08.27.24312646 · Cited by 3 · see citing works
Self-citationLai, S. M., Malik, A. M., Sathe, T. S., Silvestri, C. J., Manji, G. A., & Kluger, M. D. (2024) · medRxiv · doi:10.1101/2024.09.20.24314053 · Cited by 3 · see citing works
ChatGPT augmented clinical trial screening
Beattie, J., Owens, D., Návar, A. M., Schmitt, L. G., Taing, K., Neufeld, S., Yang, D. X., Chukwuma, C., Gul, A., Lee, D. S., et al. (2025) · Machine Learning Health · doi:10.1088/3049-477x/adbd47 · Cited by 3 · see citing works
Self-citationQi, X. S., Wang, Y., Yang, X., Ren, L., Liu, W., Benedict, S., Xiao, Y., Xing, L., & Naqa, I. M. E. (2026) · Medical Physics · doi:10.1002/mp.70616 · Cited by 2 · see citing works
Shi, H., Zhang, J., & Zhang, K. (2026) · IEEE International Conference on Healthcare Informatics (ICHI) · doi:10.1109/ichi69079.2026.00111 · Cited by 8 · see citing works
Comparative Analysis of Transformer-Based Embedding Models for Clinical Trial Retrieval
Srivastava, S., Devi, A., Kriti, N., Uttrani, S., Punekar, R. S., Pudari, K., Kaushik, S., & Dutt, V. (2025) · International Conference on Innovation and Intelligence for Informatics, Computing, and Technologies (3ICT) · doi:10.1109/3ict68299.2025.11442211 · Cited by 1 · see citing works
Qian, Q. (2026) · JMIR AI · doi:10.2196/95899 · Cited by 1 · see citing works
Haq, N. U., Zahid, M., Azeem, H., Malik, Z. N., Ibrahim, M., Timsal, M., Khan, B., & Rasul, S. (2024) · Developmental medico-life-sciences · doi:10.69750/dmls.01.06.062
Brown, D. L., Novitski, K., Sickler, J. B., Hankla, R. M., Khattak, F., Philpot, E., Durkalski‐Mauldin, V. L., Sayyahmelli, S., Patino, C., Arevalo‐Bermudez, F., et al. (2025) · Journal of the American Heart Association · doi:10.1161/jaha.125.043333
Ong, W., Tan, G., Ting, Y. H., Ge, S., Tan, Y. L., Low, X. Z., Tan, W. C., Makmur, A., Leow, N. W., Jabbar, M. A. D. A., et al. (2026) · Journal of Medical Internet Research · doi:10.2196/86251
Rahutomo, R., Syam, S., Nampira, R. A., & Syaifuddin, A. (2025) · International Conference on Information Management and Technology (ICIMTech) · doi:10.1109/icimtech67074.2025.11265532
Wable, M. S., Adsul, S. S., Bramhane, O. B., Narayan, B. R., Gayke, S. R., & Waghmare, S. M. (2026) · Asian Journal of Pharmaceutical Research · doi:10.52711/2231-5691.2026.00030
- Authors
- Beattie, J., Neufeld, S., Yang, D., Chukwuma, C., Gul, A., Desai, N., Jiang, S., & Dohopolski, M.
- Affiliations
- The University of Texas Southwestern Medical Center (US)
- Corresponding author
- Dohopolski, Michael
- Venue
- Cureus
- Published
- 2024-05-10
- Volume
- 16
- Issue
- 5
- Pages
- e60044-e60044
- ISSN
- 2168-8184 (ISSN-L)
- PMC
- PMC11162699
- OpenAlex
- W4396792104
- Cited by
- 30
- References
- 20
- Citations by year
- 2026: 5 · 2025: 19 · 2024: 6
- FWCI
- 2.80
- Type
- journal article
- Language
- en
- Open access
- diamond
- doi.org
- Licence
- cc-by
- Locations
- Cureus (publishedVersion, cc-by)
- medRxiv (acceptedVersion)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Keywords
- clinical trial matching, patient screening, eligibility criteria, large language models, GPT-4, GPT-3.5-turbo, chain-of-thought prompting, medical records, clinical trial recruitment
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
