Cited by 309Artificial Intelligence in Healthcare and EducationMachine Learning in HealthcareInformation Retrieval and Search Behavior

The DOI matches its Crossref record; 56 of the 56 references it cites were found in a registry or an index.

Matching patients to clinical trials with large language models

Open access

Abstract

Patient recruitment is challenging for clinical trials. We introduce TrialGPT, an end-to-end framework for zero-shot patient-to-trial matching with large language models. TrialGPT comprises three modules: it first performs large-scale filtering to retrieve candidate trials (TrialGPT-Retrieval); then predicts criterion-level patient eligibility (TrialGPT-Matching); and finally generates trial-level scores (TrialGPT-Ranking). We evaluate TrialGPT on three cohorts of 183 synthetic patients with over 75,000 trial annotations. TrialGPT-Retrieval can recall over 90% of relevant trials using less than 6% of the initial collection. Manual evaluations on 1015 patient-criterion pairs show that TrialGPT-Matching achieves an accuracy of 87.3% with faithful explanations, close to the expert performance. The TrialGPT-Ranking scores are highly correlated with human judgments and outperform the best-competing models by 43.8% in ranking and excluding trials. Furthermore, our user study reveals that TrialGPT can reduce the screening time by 42.6% in patient recruitment. Overall, these results have demonstrated promising opportunities for patient-to-trial matching with TrialGPT.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1038/s41467-024-53081-z

Cites

56 of 56 references verified0 citation errors · 0 not checkable

Statistics

Figures

Fig. 1 The overall architecture of TrialGPT. a TrialGPT-Retrieval can filter out most of the irrelevant trials from the initial collection and return a list of candidate clinical trials; b For a given patient, TrialGPT-Matching can explain the relevance, generate the evident sentence locations, and predict the eligibility classification for each criterion in a trial; c TrialGPT-Ranking can aggregate the criterion-level predictions by TrialGPT-Matching and use these scores to perform fine-grained ranking to get the final recommended trials.
Fig. 1 The overall architecture of TrialGPT. a TrialGPT-Retrieval can filter out most of the irrelevant trials from the initial collection and return a list of candidate clinical trials; b For a given patient, TrialGPT-Matching can explain the relevance, generate the evident sentence locations, and predict the eligibility classification for each criterion in a trial; c TrialGPT-Ranking can aggregate the criterion-level predictions by TrialGPT-Matching and use these scores to perform fine-grained ranking to get the final recommended trials.CC BY · PubMed Central
Fig. 2 First-stage retrieval results. a Overview of TrialGPT-Retrieval. LLMs first generate a list of keywords for a given patient note. These keywords are used to derive the keyword-level relevant clinical trials, which are then fused to generate a final ranking. b Recalls of relevant clinical trials at different depths for various query types and retrievers. The hybrid retriever combines the results of the BM25 (lexical matching) and the MedCPT (semantic matching) retrievers. Source data are provided as a Source Data file.
Fig. 2 First-stage retrieval results. a Overview of TrialGPT-Retrieval. LLMs first generate a list of keywords for a given patient note. These keywords are used to derive the keyword-level relevant clinical trials, which are then fused to generate a final ranking. b Recalls of relevant clinical trials at different depths for various query types and retrievers. The hybrid retriever combines the results of the BM25 (lexical matching) and the MedCPT (semantic matching) retrievers. Source data are provided as a Source Data file.CC BY · PubMed Central
Fig. 3 Manual evaluations of criterion-level predictions by GPT-4-based TrialGPT-Matching. a The percentage of correct, partially correct, and incorrect relevance explanations generated by TrialGPT-Matching; b Evaluation results of the relevant sentences located by TrialGPT-Matching based on 405 criterion-level annotations where at least one sentence is labeled as relevant in the ground-truth. 95% confidence intervals estimated by bootstrapping are shown as error bars; c The confusion matrices of the eligibility for inclusion criteria predicted by human experts and TrialGPT-Matching; d The confusion matrices of the eligibility for exclusion criteria predicted by human experts and TrialGPT-Matching. Not Incl.: Not included. Not Excl.: Not excluded. No Info.: Not enough information. Not Appl.: Not applicable. Source data are provided as a Source Data file.
Fig. 3 Manual evaluations of criterion-level predictions by GPT-4-based TrialGPT-Matching. a The percentage of correct, partially correct, and incorrect relevance explanations generated by TrialGPT-Matching; b Evaluation results of the relevant sentences located by TrialGPT-Matching based on 405 criterion-level annotations where at least one sentence is labeled as relevant in the ground-truth. 95% confidence intervals estimated by bootstrapping are shown as error bars; c The confusion matrices of the eligibility for inclusion criteria predicted by human experts and TrialGPT-Matching; d The confusion matrices of the eligibility for exclusion criteria predicted by human experts and TrialGPT-Matching. Not Incl.: Not included. Not Excl.: Not excluded. No Info.: Not enough information. Not Appl.: Not applicable. Source data are provided as a Source Data file.CC BY · PubMed Central
Fig. 4 Correlation between differently aggregated TrialGPT scores and the ground-truth patient-trial eligibility labels. a The percentage of inclusion criteria predicted as “included” by TrialGPT; b The percentage of inclusion criteria predicted as “not included”; c The percentage of inclusion criteria predicted as “no relevant information”; d The LLM-aggregated relevance score; e The percentage of exclusion criteria predicted as “excluded”; f The percentage of exclusion criteria predicted as “not excluded”; g The percentage of exclusion criteria predicted as “no relevant information”; h The LLM-aggregated eligibility score. “*” denotes p < 0.05, “**” denotes p < 0.01, “***” denotes p < 0.001, and “n.s.” denotes not significant (p > 0.05) by two-sided independent t-test. There are 60,240 unlabeled, 15,459 irrelevant, 6981 excluded, 647 potential, and 8173 eligible patient-trial pairs. Center line, median; box limits, upper and lower quartiles; whiskers, 1.5x interquartile range; points, outliers. Source data are provided as a Source Data file, including all the exact p values.
Fig. 4 Correlation between differently aggregated TrialGPT scores and the ground-truth patient-trial eligibility labels. a The percentage of inclusion criteria predicted as “included” by TrialGPT; b The percentage of inclusion criteria predicted as “not included”; c The percentage of inclusion criteria predicted as “no relevant information”; d The LLM-aggregated relevance score; e The percentage of exclusion criteria predicted as “excluded”; f The percentage of exclusion criteria predicted as “not excluded”; g The percentage of exclusion criteria predicted as “no relevant information”; h The LLM-aggregated eligibility score. “*” denotes p < 0.05, “**” denotes p < 0.01, “***” denotes p < 0.001, and “n.s.” denotes not significant (p > 0.05) by two-sided independent t-test. There are 60,240 unlabeled, 15,459 irrelevant, 6981 excluded, 647 potential, and 8173 eligible patient-trial pairs. Center line, median; box limits, upper and lower quartiles; whiskers, 1.5x interquartile range; points, outliers. Source data are provided as a Source Data file, including all the exact p values.CC BY · PubMed Central
Fig. 5 Results of the patient-trial matching user study. a Experimental design and actual screening times of each patient-trial pair by two expert annotators; b Comparison of screening time aggregated by different clinical trials; c Comparison of screening time aggregated by different patient cases; d Comparison of screening time aggregated by short cases, long cases, annotators, and all pairs. Numbers in parentheses denote the sample sizes in the corresponding group of the comparison. Within the annotator (e.g., Annotator X or Y), significant tests are conducted by two-sided independent t-test. Other significant tests are two-sided paired t-tests. Trial A, B, C, D, E, and F denote NCT04432597, NCT05012098, NCT04287868, NCT04847466, NCT04719988, and NCT04894370, respectively. Source data are provided as a Source Data file.
Fig. 5 Results of the patient-trial matching user study. a Experimental design and actual screening times of each patient-trial pair by two expert annotators; b Comparison of screening time aggregated by different clinical trials; c Comparison of screening time aggregated by different patient cases; d Comparison of screening time aggregated by short cases, long cases, annotators, and all pairs. Numbers in parentheses denote the sample sizes in the corresponding group of the comparison. Within the annotator (e.g., Annotator X or Y), significant tests are conducted by two-sided independent t-test. Other significant tests are two-sided paired t-tests. Trial A, B, C, D, E, and F denote NCT04432597, NCT05012098, NCT04287868, NCT04847466, NCT04719988, and NCT04894370, respectively. Source data are provided as a Source Data file.CC BY · PubMed Central

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 43 references

Research works printed but not linked: 2

  1. Roberts, K., Demner-Fushman, D., Voorhees, E. M., Bedrick, S. & Hersh, W. R. Overview of the TREC 2021 Clinical Trials Track. In Proc. Thirtieth Text REtrieval Conference (TREC 2021) (2021). · DOI
  2. Roberts, K., Simpson, M. S., Voorhees, E. M. & Hersh, W. R. Overview of the TREC 2015 Clinical Decision Support Track. In Proc. Twenty-Fourth Text REtrieval Conference (TREC 2015) (2015). · DOI

Cited by

Sorted by citations

Items 201–300 of 309

Metadata

Authors
Jin, Q., Wang, Z., Floudas, C. S., Chen, F., Gong, C., Bracken-Clarke, D., Xue, E., Yang, Y., Sun, J., & Lu, Z.
Affiliations
  • National Institutes of Health (US)
  • University of Illinois Urbana-Champaign (US)
  • University of Pittsburgh (US)
  • Albert Einstein College of Medicine (US)
Corresponding author
Lu, Zhiyong
Venue
Nature Communications
Published
2024-11-18
Volume
15
Issue
1
Pages
9074-9074
ISSN
2041-1723 (ISSN-L)
OpenAlex
W4404486407
Cited by
287
References
43
Citations by year
2026: 119 · 2025: 117 · 2024: 50
FWCI
26.74
Type
journal article
Language
en
Open access
Locations
  • Nature Communications (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • PubMed Central (submittedVersion, cc-by)
  • RePEc: Research Papers in Economics (submittedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
  • arXiv (Cornell University) (cc-by)
Funders
U.S. Department of Health & Human Services | National Institutes of Health
Topics
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
  • Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
Keywords
clinical trial recruitment, large language models, explainable AI, synthetic patient data, user study

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review