Distilling large language models for matching patients to clinical trials
Open accessAbstract
Objective: The objective of this study is to systematically examine the efficacy of both proprietary (GPT-3.5, GPT-4) and open-source large language models (LLMs) (LLAMA 7B, 13B, 70B) in the context of matching patients to clinical trials in healthcare. Materials and methods: The study employs a multifaceted evaluation framework, incorporating extensive automated and human-centric assessments along with a detailed error analysis for each model, and assesses LLMs’ capabilities in analyzing patient eligibility against clinical trial’s inclusion and exclusion criteria. To improve the adaptability of open-source LLMs, a specialized synthetic dataset was created using GPT-4, facilitating effective fine-tuning under constrained data conditions. Results: The findings indicate that open-source LLMs, when fine-tuned on this limited and synthetic dataset, achieve performance parity with their proprietary counterparts, such as GPT-3.5. Discussion: This study highlights the recent success of LLMs in the high-stakes domain of healthcare, specifically in patient-trial matching. The research demonstrates the potential of open-source models to match the performance of proprietary models when fine-tuned appropriately, addressing challenges like cost, privacy, and reproducibility concerns associated with closed-source proprietary LLMs. Conclusion: The study underscores the opportunity for open-source LLMs in patient-trial matching. To encourage further research and applications in this field, the annotated evaluation dataset and the fine-tuned LLM, Trial-LLAMA, are released for public use.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Recent LLM studies for patient-trial matching predominantly use relatively simplistic or synthetic patient notes and clinical trial setups that do not accurately reflect real-world complexity.
“However, these studies predominantly utilize relatively simplistic or synthetic patient notes and clinical trial setups, which do not accurately reflect the complexity of real-world scenarios.”
From PRISM: Patient Records Interpretation for Semantic clinical trial Matching system using large language models (2024) · cites this paper as [19]
The Limitations section (page unavailable) says this paper’s dataset does not fully capture real-world complexity, particularly in patient summaries. The Abstract describes its fine-tuning dataset as synthetic. Neither passage establishes that recent LLM studies predominantly use simplistic or synthetic patient notes and trial setups; that broader claim is unsupported. (LLM, conf=0.85)
Experimental — accuracy not yet measuredSimilar errors to those observed in our work have been reported, such as incorrectly identifying patients who meet partial criteria, or assuming a patient with breast cancer does not have lung cancer simply because it is not explicitly mentioned.
“Similar errors to those observed in our work have been reported, such as incorrectly identifying patients who meet partial criteria, or assuming a patient with breast cancer does not have lung cancer simply because it is not explicitly mentioned [36,37].”
From Identifying Deprescribing Opportunities With Large Language Models in Older Adults: Retrospective Cohort Study (2025) · cites this paper as [37]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Kaugeranna, R., Kaugeranna, E., & Aion, (. S. 4. 6. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.giscience.2023.43 · Cited by 13,695 · see citing works
Large language models encode clinical knowledge
Singhal, K., Azizi, S., Tu, T., Mahdavi, S. S., Wei, J., Chung, H. W., Scales, N., Tanwani, A., Cole-Lewis, H., Pfohl, S., et al. (2023) · Nature · doi:10.1038/s41586-023-06291-2 · Cited by 4,083 · see citing works
Llama 2: Open Foundation and Fine-Tuned Chat Models
Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. (2023) · arXiv · doi:10.48550/arxiv.2307.09288 · Cited by 2,604 · see citing works
Scaling Instruction-Finetuned Language Models
Chung, H. W., Hou, L., Longpre, S., Zoph, B., Tay, Y., Fedus, W., Li, Y., Wang, X., Dehghani, M., Brahma, S., et al. (2022) · arXiv · doi:10.48550/arxiv.2210.11416 · Cited by 1,441 · see citing works
Training Compute-Optimal Large Language Models
Hoffmann, J., Borgeaud, S., Mensch, A., Buchatskaya, E., Cai, T., Rutherford, E., Casas, D. D. L., Hendricks, L. A., Welbl, J., Clark, A., et al. (2022) · arXiv · doi:10.48550/arxiv.2203.15556 · Cited by 929 · see citing works
Capabilities of GPT-4 on Medical Challenge Problems
Nori, H., King, N., McKinney, S. M., Carignan, D., & Horvitz, E. (2023) · arXiv · doi:10.48550/arxiv.2303.13375 · Cited by 509 · see citing works
Evaluating large language models on medical evidence summarization
Tang, L., Sun, Z., Idnay, B., Nestor, J. G., Soroush, A., Elias, P. A., Xu, Z., Ding, Y., Durrett, G., Rousseau, J. F., et al. (2023) · npj Digital Medicine · doi:10.1038/s41746-023-00896-7 · Cited by 423 · see citing works
Opportunities and Challenges for ChatGPT and Large Language Models in Biomedicine and Health
Tian, S., Jin, Q., Yeganova, L., Lai, P.-T., Zhu, Q., Chen, X., Yang, Y., Chen, Q., Kim, W., Comeau, D. C., et al. (2023) · Briefings in Bioinformatics · doi:10.1093/bib/bbad493 · Cited by 359 · see citing works
Jiang, A. Q., Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D. S., Casas, D. D. L., Bressand, F., Lengyel, G., Lample, G., Saulnier, L., et al. (2023) · arXiv · doi:10.48550/arxiv.2310.06825 · Cited by 340 · see citing works
Bai, Y., Kadavath, S., Kundu, S., Askell, A., Kernion, J., Jones, A., Chen, A., Goldie, A., Mirhoseini, A., & Kaplan, J. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2212.08073 · Cited by 326 · see citing works
Instruction Tuning for Large Language Models: A Survey
Zhang, S., Dong, L., Li, X., Zhang, S., Sun, X., Wang, S., Li, J., Hu, R., Zhang, T., Wu, F., et al. (2023) · arXiv · doi:10.48550/arxiv.2308.10792 · Cited by 241 · see citing works
A Survey of Large Language Models in Medicine: Progress, Application, and Challenge
Zhou, H., Liu, F., Gu, B., Zou, X., Huang, J., Wu, J., Li, Y., Chen, S. S., Zhou, P., Liu, J., et al. (2023) · arXiv · doi:10.48550/arxiv.2311.05112 · Cited by 240
GPT-4: a new era of artificial intelligence in medicine
Waisberg, E., Ong, J., Masalkhi, M., Kamran, S. A., Zaman, N., Sarker, P., Lee, A. G., & Tavakkoli, A. (2023) · Irish Journal of Medical Science (1971 -) · doi:10.1007/s11845-023-03377-8 · Cited by 186 · see citing works
Peng, B., Li, C., He, P., Galley, M., & Gao, J. (2023) · arXiv · doi:10.48550/arxiv.2304.03277 · Cited by 183 · see citing works
Challenges in recruitment and retention of clinical trial subjects
Kadam, R., Borde, S., Madas, S., Salvi, S., & Limaye, S. (2016) · Perspectives in Clinical Research · doi:10.4103/2229-3485.184820 · Cited by 179 · see citing works
Criteria2Query: a natural language interface to clinical databases for cohort definition
Yuan, C., Ryan, P. B., Ta, C., Guo, Y., Li, Z., Hardin, J., Makadia, R., Jin, P., Shang, N., Kang, T., et al. (2019) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocy178 · Cited by 170 · see citing works
EliXR: an approach to eligibility criteria extraction and representation
Weng, C., Wu, X., Luo, J., Boland, M. R., Theodoratos, D., & Johnson, S. B. (2011) · Journal of the American Medical Informatics Association · doi:10.1136/amiajnl-2011-000321 · Cited by 143 · see citing works
Electronic Screening Improves Efficiency in Clinical Trial Recruitment
Thadani, S. R., Weng, C., Bigger, J. T., Ennever, J. F., & Wajngurt, D. (2009) · Journal of the American Medical Informatics Association · doi:10.1197/jamia.m3119 · Cited by 139 · see citing works
Self-Instruct: Aligning Language Models with Self-Generated Instructions
Wang, Y., Kordi, Y., Mishra, S., Liu, A., Smith, N. A., Khashabi, D., & Hajishirzi, H. (2022) · arXiv · doi:10.48550/arxiv.2212.10560 · Cited by 118 · see citing works
Self-citationFayter, D., McDaid, C., & Eastwood, A. (2007) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2006.12.013 · Cited by 112 · see citing works
Predicting Low Accrual in the National Cancer Institute’s Cooperative Group Clinical Trials
Bennette, C. S., Ramsey, S. D., McDermott, C. L., Carlson, J. J., Basu, A., & Veenstra, D. L. (2016) · Journal of the National Cancer Institute · doi:10.1093/jnci/djv324 · Cited by 108 · see citing works
Self-citationOpportunities and Challenges in Leveraging Electronic Health Record Data in Oncology
Berger, M. L., Curtis, M. D., Smith, G., Harnett, J., & Abernethy, A. P. (2016) · Future Oncology · doi:10.2217/fon-2015-0043 · Cited by 98 · see citing works
Miotto, R., & Weng, C. (2015) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocu050 · Cited by 97 · see citing works
Effort Required in Eligibility Screening for Clinical Trials
Penberthy, L. T., Dahman, B. A., Petkov, V. I., & DeShazo, J. P. (2012) · Journal of Oncology Practice · doi:10.1200/jop.2012.000646 · Cited by 96 · see citing works
Joseph, G., & Dohan, D. P. (2009) · Contemporary Clinical Trials · doi:10.1016/j.cct.2009.06.009 · Cited by 84 · see citing works
Automated matching software for clinical trials eligibility: Measuring efficiency and flexibility
Penberthy, L., Brown, R., Puma, F., & Dahman, B. A. (2010) · Contemporary Clinical Trials · doi:10.1016/j.cct.2010.03.005 · Cited by 84 · see citing works
Zephyr: Direct Distillation of LM AlignmentPreprint
Tunstall, L., Beeching, E., Lambert, N., Rajani, N. F., Rasul, K., Belkada, Y., Huang, S., von Werra, L., Fourrier, C., Habib, N., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2310.16944 · Cited by 80 · see citing works
How Abilities in Large Language Models are Affected by Supervised Fine-tuning Data Composition
Dong, G., Yuan, H., Lu, K., et al. (2024) · Annual Meeting of the Association for Computational Linguistics (ACL) · doi:10.18653/v1/2024.acl-long.12 · Cited by 50 · see citing works
Large Language Models are Few-Shot Clinical Information ExtractorsPreprint
Agrawal, M., Hegselmann, S., Lang, H., et al. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2205.12689 · Cited by 49 · see citing works
Combining human and machine intelligence for clinical trial eligibility querying
Fang, Y., Idnay, B., Sun, Y., Liu, H., Chen, Z., Marder, K. S., Xu, H., Schnall, R., & Weng, C. (2022) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocac051 · Cited by 29 · see citing works
Cohort selection for clinical trials using deep learning models
Segura-Bédmar, I., & Raez, P. (2019) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocz139 · Cited by 28 · see citing works
Lamberti, M. J., Wilkinson, M., Harper, B., et al. (2018) · Therapeutic Innovation & Regulatory Science · doi:10.1177/2168479017751403 · Cited by 25 · see citing works
Generating medical logic modules for clinical trial eligibility criteria.
Parker, C. G., & Embley, D. W. (2003) · PubMed · Cited by 14 · see citing works
How Abilities in Large Language Models are Affected by Supervised Fine-tuning Data Composition
Dong, G., Yuan, H., Lu, K., Li, C., Xue, M., Liu, D., Wang, W., Yuan, Z., Zhou, C., & Zhou, J. (2023) · arXiv · doi:10.48550/arxiv.2310.05492 · Cited by 5 · see citing works
Generating Medical Logic Modules for Clinical Trial Eligibility
Parker, C. G. (2005) · ScholarsArchive (Brigham Young University) · Cited by 3 · see citing works
How (not) to ensemble LVLMs for VQAPreprint
Alazraki, L., Castrejón, L., Dehghani, M., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2310.06641
OpenAI, Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., et al. (2023) · arXiv · doi:10.48550/arxiv.2303.08774
Flagged first, then by citations
Research works printed but not linked: 16
- Getz K. Enrollment performance: weighing the "facts". Appl Clin Trials. 2012;21(5):24-25.
- Jin Q, Wang Z, Floudas CS, Sun J, Lu Z. Matching patients to clin- ical trials with large language models. arXiv 2307.15051. 2023, preprint: not peer reviewed. Accessed December 6, 2023. 1962 Journal of the American Medical Informatics Association, 2024, Vol. 31, No. 9 · arXiv
- Koopman B, Zuccon G. A test collection for matching patients to clinical trials. In: Proceedings of the 39th International ACM SIGIR Conference on Research and Development in Information Retrieval. SIGIR; 2016:669-672.
- Pradeep R, Li Y, Wang Y, Lin J. Neural query synthesis and domain-specific ranking templates for multi-stage clinical trial matching. In: Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR '22. Association for Computing Machinery; 2022:2325-2330.
- Roberts K, Demner-Fushman D, Voorhees EM, Bedrick S, Hersh WR. Overview of the TREC 2021 clinical trials track. In: Proceed- ings of the Thirtieth Text REtrieval Conference. TREC; 2021.
- Roberts K, Demner-Fushman D, Voorhees EM, Bedrick S, Hersh WR. Overview of the TREC 2022 clinical trials track. In: Soboroff I, Ellis A, eds. Proceedings of the Thirty-First Text REtrieval Con- ference, TREC 2022, Online, November 15-19, 2022. Vol 500- 338 of NIST Special Publication. National Institute of Standards and Technology (NIST); 2022.
- Rybinski M, Nguyen V, Karimi S. A self-learning resource-efficient re-ranking method for clinical trials search. In: Proceedings of the 32nd ACM International Conference on Information and Knowl- edge Management, CIKM '23. Association for Computing Machi- nery; 2023:4249-4253.
- Brown T, Mann B, Ryder N, et al. Language models are few-shot learners. Adv Neural Inf Process Syst. 2020;33:1877-1901.
- Chowdhery A, Narang S, Devlin J, et al. Palm: scaling language modeling with pathways. J Mach Learn Res. 2023;24(240):1-13.
- Van Veen D, Van Uden C, Blankemeier L, et al. Clinical text sum- marization: adapting large language models can outperform human experts. Res Sq. 2023, preprint: not peer reviewed. Preprint published on October 30, 2023.
- Yang J, Liu C, Deng W, et al. Enhancing phenotype recognition in clinical notes using large language models: PhenoBCBERT and PhenoGPT. Patterns. 2024;5(1):100887.
- Gao Y, Li R, Caskey J, et al. Leveraging a medical knowledge graph into large language models for diagnosis prediction. arXiv 2308.14321. 2023, preprint: not peer reviewed. · arXiv
- Mishra S, Khashabi D, Baral C, Hajishirzi H. Cross-task general- ization via natural language crowdsourcing instructions. In: Pro- ceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics; 2022:3470-3487.
- Fleming SL, Lozano A, Haberkorn WJ, et al. MedAlign: a clinician-generated dataset for instruction following with elec- tronic medical records. Proc AAAI Conf Artif Intell. 2024;38 (20):22021-22030.
- Rafailov R, Sharma A, Mitchell E, Ermon S, Manning CD, Finn C. Direct preference optimization: your language model is secretly a reward model. Adv Neural Inf Process Syst. 2024;36.
- All rights reserved. For permissions, please email: journals.permissions@oup.com Journal of the American Medical Informatics Association, 2024, 31, 1953-1963 https://doi.org/10.1093/jamia/ocae073 Research and Applications · DOI
Cited by
Sorted by citations
Matching patients to clinical trials with large language models
Jin, Q., Wang, Z., Floudas, C. S., Chen, F., Gong, C., Bracken-Clarke, D., Xue, E., Yang, Y., Sun, J., & Lu, Z. (2024) · Nature Communications · doi:10.1038/s41467-024-53081-z · Cited by 287 · see citing works
Gupta, S., Basu, A., Nievas, M., Thomas, J., Wolfrath, N., Ramamurthi, A., Taylor, B., Kothari, A. N., Schwind, R., Miller, T. M., et al. (2024) · npj Digital Medicine · doi:10.1038/s41746-024-01274-7 · Cited by 52 · see citing works
Self-citationTransforming Informed Consent Generation Using Large Language Models: Mixed Methods Study
Shi, Q., Luzuriaga, K., Allison, J. J., Oztekin, A., Faro, J. M., Lee, J. L., Hafer, N., McManus, M., & Zai, A. H. (2025) · JMIR Medical Informatics · doi:10.2196/68139 · Cited by 23 · see citing works
Socrates, V., Wright, D. S., Huang, T., Fereydooni, S., Dien, C., Chi, L., Albano, J., Patterson, B., Sasidhar Kanaparthy, N., Wright, C. X., et al. (2025) · JMIR Aging · doi:10.2196/69504 · Cited by 17 · see citing works
End-To-End Clinical Trial Matching with Large Language Models
Ferber, D., Hilgers, L., Wiest, I. C., Leßmann, M.-E., Clusmann, J., Neidlinger, P., Zhu, J., Wölflein, G., Lammert, J., Tschochohei, M., et al. (2024) · arXiv · doi:10.48550/arxiv.2407.13463 · Cited by 12 · see citing works
Gui, X., Lv, H., Wang, X., Lv, L., Xiao, Y., & Wang, L. (2025) · BioData Mining · doi:10.1186/s13040-025-00458-5 · Cited by 7 · see citing works
- Authors
- Nievas, M., Basu, A., Wang, Y., & Singh, H.
- Affiliations
- Triomics Research, Triomics, Inc. , San Francisco, CA 94105, United States
- Triomics Research, Triomics, Inc. , Bengaluru, Karnataka 560102, India
- University of Pittsburgh (US)
- Corresponding author
- Wang, Yanshan
- Venue
- Journal of the American Medical Informatics Association
- Publisher
- Oxford University Press (OUP)
- Published
- 2024-04-19
- Volume
- 31
- Issue
- 9
- Pages
- 1953-1963
- ISSN
- 1067-5027 (ISSN-L)
- 1527-974X
- PMC
- PMC11339497
- OpenAlex
- W4394975332
- Cited by
- 67
- References
- 37
- Citations by year
- 2026: 19 · 2025: 35 · 2024: 13
- FWCI
- 6.26
- Type
- journal article
- Language
- en
- Open access
- green
- www.ncbi.nlm.nih.gov
- Locations
- Journal of the American Medical Informatics Association (publishedVersion)
- PubMed Central (publishedVersion)
- PubMed (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Keywords
- large language models, inclusion and exclusion criteria, open-source LLMs, llamas, GPT-4, GPT-3.5, synthetic data generation, fine-tuning, error analysis, artificial intelligence in healthcare
- MeSH terms
- Clinical Trials as Topic, Humans, Natural Language Processing, Programming Languages, Patient Selection
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
