PRISM: Patient Records Interpretation for Semantic clinical trial Matching system using large language models

Only about 7% of adults participate in cancer clinical trials.

“Only about 7% of adults participate in cancer clinical trials1.”

• cites [1] Unger, J. M., Cook, E., Tai, E. & Bleyer, A. The role of clinical trial participation in cancer research: barriers, evidence, and strategies. Am. Soc. Clin. Oncol. Educ. Book 35, 185–198 (2016). sourceDisputed

  The Introduction estimates that fewer than 5% of adult cancer patients enroll in cancer clinical trials, and the Discussion repeats the <5% figure. The cited paper therefore gives a lower rate than the claimed 7%. (LLM, conf=0.90)

  Checked by a reasoning model — may be wrong.

  Experimental — accuracy not yet measured

The core information pertinent to the inclusion and exclusion criteria of clinical trials is primarily found within unstructured EHRs, such as medical notes.

“The core information pertinent to the inclusion and exclusion criteria of clinical trials are primarily found within unstructured EHRs, such as medical notes12.”

• cites [12] Kong, H.-J. Managing unstructured big data in healthcare system. Healthc. Inform. Res. 25, 1–2 (2019). sourceCould not verify

  The main-text passage says that much medical data is unstructured, including text, but does not discuss where clinical-trial inclusion or exclusion information is found. Searches for trial eligibility criteria and free-text EHR notes found no passage addressing that specific claim. (LLM, conf=0.76)

  Experimental — accuracy not yet measured

Qwen-1.5 14B was the top performer among similar-sized models on the LLM leaderboard when the experiments began.

“We chose this model because it was the top performer among similar-sized models on the LLM leaderboard when our experiments began49.”

• cites [49] Chiang, W. et al. Chatbot arena: an open platform for evaluating llms by human preference. Preprint at arXiv:2403.04132 (2024). sourceCould not verify

  Section 3.2 describes Chatbot Arena’s model comparisons, but it does not identify Qwen-1.5 14B or rank models of similar size. Searches for Qwen-1.5 14B and size-based leaderboard rankings found no passage establishing its position when the citing authors began their experiments. (LLM, conf=0.72)

  Experimental — accuracy not yet measured

Creating queries using trial criteria from structured EHRs often proves challenging in interpretation and scaling.

“Even when dealing with structured EHRs, creating queries using the trial criteria often proves challenging in interpretation and scaling14,15.”

• cites [14] Wong, C. et al. Scaling clinical trial matching using large language models: A case study in oncology. ArXiv (2023). https://arxiv.org/abs/2308.02180. Accessed April 6, 2024. sourceSupported
• cites [15] Yuan, C. et al. Criteria2query: a natural language interface to clinical databases for cohort definition. J. Am. Med Inf. Assoc. 26, 294–305 (2019). sourceSupported

The challenge of patient EHRs surpassing LLM context lengths requires the use of a semantic retrieval engine to manage and extract relevant information from each criterion.

“This challenge requires the use of a semantic retrieval engine to manage and extract relevant information from each criterion46.”

• cites [46] Basu, A., Gupta, S., Taylor, B., Kothari, A. & Singh, H. Onco-retriever: generative classifier for retrieval of EHR records in oncology. Preprint at https://arxiv.org/pdf/2404.06680.pdf (2024). sourcePartly supported

  The Related Work section (page unavailable) proposes semantic retrieval engines to address limitations in searching EHR notes, and §3.1 describes retrieving relevant chunks for predefined concepts. The paper does not attribute that need to patient EHRs exceeding LLM context lengths, nor does it describe extracting information for each criterion. (LLM, conf=0.85)

  Experimental — accuracy not yet measured

• AssumptionRests on an assumption: Each patient’s EHRs usually include hundreds of notes, which often surpass the context length capabilities of most open-source and proprietary LLMs.

Recent LLM studies for patient-trial matching predominantly use relatively simplistic or synthetic patient notes and clinical trial setups that do not accurately reflect real-world complexity.

“However, these studies predominantly utilize relatively simplistic or synthetic patient notes and clinical trial setups, which do not accurately reflect the complexity of real-world scenarios.”

• cites [14] Wong, C. et al. Scaling clinical trial matching using large language models: A case study in oncology. ArXiv (2023). https://arxiv.org/abs/2308.02180. Accessed April 6, 2024. sourcePartly supported

  The Introduction (page unavailable) describes the complexity of real patient records and says this study used structured patient information rather than having an LLM process hundreds of notes. It does not say that recent LLM studies predominantly used simplistic or synthetic patient notes or trial setups; it describes its own oncology trial criteria as elaborate. (LLM, conf=0.82)

  Experimental — accuracy not yet measured
• cites [18] Jin, Q., Wang, Z., Floudas, C. S., Sun, J. & Lu, Z. Matching patients to clinical trials with large language models. Preprint at https://arxiv.org/abs/2307.15051 (2023). sourcePartly supported

  The Discussion (page unavailable) says this paper's evaluation uses paragraph-length patient summaries and omits longitudinal notes, lab values, imaging, geolocation, and recruitment status—limitations relevant to real-world matching. The Abstract mentions some earlier LLM studies using synthetic data for augmentation, but the paper does not establish that prior studies predominantly use simplistic or synthetic patient notes and trial setups. (LLM, conf=0.83)

  Experimental — accuracy not yet measured
• cites [19] Nievas, M., Basu, A., Wang, Y. & Singh, H. Distilling large language models for matching patients to clinical trials. J Am Med Inform Assoc. 31, 1953–1963 (2024). sourcePartly supported

  The Limitations section (page unavailable) says this paper’s dataset does not fully capture real-world complexity, particularly in patient summaries. The Abstract describes its fine-tuning dataset as synthetic. Neither passage establishes that recent LLM studies predominantly use simplistic or synthetic patient notes and trial setups; that broader claim is unsupported. (LLM, conf=0.85)

  Experimental — accuracy not yet measured
• cites [20] Wornow, M. et al. Zero-shot clinical trial patient matching with llms. Preprint at https://arxiv.org/abs/2402.05125 (2024). sourceSupported

The cumbersome task of reviewing patients against trials leads to a lower rate of trial recommendations.

“This cumbersome task leads to a lower rate of trial recommendations6, often due to the intricacy involved in deciphering the eligibility of a patient against the nuanced inclusion and exclusion criteria of these trials7–11.”

• cites [6] Nuttall, A. Considerations for improving patient recruitment into clinical trials. Clinical Leader Newsletter http://vertassets.blob.core.windows.net/download/64c39d7e/64c39d7e-c643-457b-aec2-9ff7b65b3ad2/rdprecruitmentwhitepaper.pdf (2012).Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [7] Kadam, R., Borde, S., Madas, S., Salvi, S. & Limaye, S. Challenges in recruitment and retention of clinical trial subjects. Perspect. Clin. Res. 7, 137–143 (2016). sourcePartly supported

  The Discussion (Interpretations; page unavailable) says complex protocols and extensive selection criteria hinder recruitment, supporting the claim that assessing eligibility can be cumbersome. The paper does not report that this task lowers the rate at which trials are recommended to patients. (LLM, conf=0.78)

  Experimental — accuracy not yet measured
• cites [8] Bennette, C. et al. Predicting low accrual in the national cancer institute’s cooperative group clinical trials. J. Natl Cancer Inst. 108, djv324 (2016). sourceCould not verify

  The Factors Associated With Low Accrual section (page unavailable) links burdensome eligibility criteria and trial complexity to low accrual, but does not address the rate at which clinicians recommend trials. Searches for patient review, recommendations, and referrals found no passage connecting the cumbersome review task to fewer trial recommendations. (LLM, conf=0.76)

  Experimental — accuracy not yet measured
• cites [9] Berger, M., Curtis, M., Smith, G., Harnett, J. & Abernethy, A. Opportunities and challenges in leveraging electronic health record data in oncology. Future Oncol. 12, 1261–1274 (2016). sourceNot checked — no full text of the source
• cites [10] Clinical Research Professionals, A. Tufts analysis: Patient recruitment shortcomings laid at feet of poor provider, researcher engagement (2017).Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [11] Fayter, D., McDaid, C. & Eastwood, A. A systematic review highlights threats to validity in studies of barriers to cancer trial participation. J. Clin. Epidemiol. 60, 990–1001 (2007). sourceNot checked — no full text of the source

For ground truth trials ranked within the top-3, OncoLLM meets 66.7% of criteria compared to GPT-3.5 Turbo's 59%.

“Predictably, the overall criteria Met numbers increased for both models, with OncoLLM (66.7%) outperforming GPT-3.5 Turbo (59%).”

OncoLLM demonstrated superior performance compared to GPT-3.5 Turbo across all three scoring methods for patient-centric ranking.

“In our assessment, OncoLLM demonstrated superior performance compared to GPT-3.5 Turbo across all three scoring methods (Fig. 3).”

GPT-4 achieved 68% accuracy on criteria/question-level answering.

“Although GPT-4 reached a higher accuracy of 68%, it incurs substantially greater computational costs, being approximately more than 100x times larger than OncoLLM (even though there are no official details on the parameter size of GPT-4, it is estimated to be more than a trillion parameters).”

McNemar's test for GPT-3.5 Turbo and OncoLLM confirms the significance of the results with a p-value < 0.05 (0.0013).

“Our statistical analysis conducted using McNemar’s test for GPT-3.5 Turbo and OncoLLM also confirms the significance of the results with a p-value < 0.05 (0.0013).”

GPT-3.5 showed 98% precision and 94% recall in determining the relevance of segments to questions after refining prompts.

“GPT-3.5 showed 98% precision and 94% recall in determining the relevance of segments to questions after refining our prompts.”

OncoLLM (Weighted tier) scored an NDCG of 68% compared to 62.6% for GPT-3.5 Turbo (Iterative tier).

“OncoLLM (Weighted tier) scored an NDCG (Normalized Discounted Cumulative Gain) score of 68% as compared to 62.6% of GPT-3.5 Turbo (Iterative tier).”

OncoLLM and GPT-4 most closely match the actual N/A count determined by human annotators.

“OncoLLM and GPT-4 most closely match the actual N/A count determined by human annotators.”

GPT-3.5 Turbo tends to return N/A responses to simplified questions derived from top-three ranked ground truth trials considerably more often than OncoLLM.

“GPT-3.5 Turbo tends to return “N/A” responses to simplified questions derived from top-three ranked ground truth trials considerably more often than OncoLLM (Figs. 4C, 2).”

OncoLLM exhibited superior performance compared to GPT-3.5 Turbo across all scoring methods for trial-centric ranking, except for the Simple method where both models achieved similar scores.

“In our evaluation, OncoLLM exhibited superior performance compared to GPT-3.5 Turbo across all scoring methods, except for the Simple method where both models achieved similar scores.”

OncoLLM meets 62% of the criteria on average across all 98 patients, compared to GPT-3.5 Turbo's 55.4%.

“We discovered that OncoLLM meets 62% of the criteria on average across all 98 patients, compared to GPT-3.5 Turbo’s score of 55.4% (Fig. 4A).”

Tiering and weighting systems for ranking clinical trials based on criterion-level eligibility outperform existing approaches that do not incorporate domain-specific insight.

“Our experiments showed that these methods outperform existing approaches that do not incorporate this domain-specific insight.”

We propose a novel algorithm for ranking clinical trials that significantly improves the average position of relevant trials compared to a baseline approach.

“We propose a novel algorithm for ranking clinical trials, significantly improving the average position of relevant trials compared to a baseline approach.”

There were significant variations in the responses provided by different annotators, especially for questions where it is difficult to determine whether the available information suffices for accurate criteria assessment.

“We observed significant variations in the responses provided by different annotators, especially for questions where it is difficult to determine whether the available information suffices for accurate criteria assessment.”

In a manual evaluation of 50 trials, 1% of questions were missed by the LLM model.

“Only 1% of questions were missed”

In a manual evaluation of 50 trials, 2% of questions were incorrectly formed.

“merely 2% of questions were incorrectly formed”

The accuracy rate of correctly forming boolean logic at the criterion level was approximately 89%.

“Additionally, we achieved an accuracy rate of ~89% in correctly forming boolean logic at the criterion level.”

Organizing retrieved information chunks chronologically, beginning with the patient's age and enrollment date and supplementing each chunk with note date and type, improved the model's ability to answer temporal questions accurately.

“We observed that this approach benefited the model by enabling it to interpret the patient’s journey chronologically, which enhanced its ability to answer temporal questions accurately (See With and Without Chronological Order of Notes in Supplementary Note 5).”

The information necessary for eligibility screening can span over a year.

“Our findings indicate that the information necessary for eligibility screening can span over a year, underscoring the importance of accessing comprehensive patient notes.”

Adding the trial composition module to the pipeline resulted in significant improvement in the performance of the entire pipeline.

“Our results showed that adding this module in the pipeline resulted in significant improvement in the performance”

The most critical data were often found in the most recent notes.

“However, the most critical data were often found in the most recent notes.”

OncoLLM matches the performance of qualified medical doctors for clinical trial matching.

“show that OncoLLM outperforms GPT-3.5 and matches the performance of qualified medical doctors for clinical trial matching.”

• Also stated in: § Discussion · hedged

  “While there are certain limitations, the performance of our proposed pipeline closely matches that of qualified medical professionals in terms of criteria-level accuracy.”

A primary constraint of the PRISM approach is its reliance solely on unstructured data.

“One of the primary constraints is our reliance solely on unstructured data.”

• AssumptionRests on an assumption: Critical information, particularly laboratory values, is often recorded in structured formats and may not be consistently mentioned in patient notes.

Assigning different weights to various types of inclusion/exclusion criteria is important to achieve optimal performance.

“Additionally, we highlighted the importance of assigning different weights to various types of inclusion/exclusion criteria to achieve optimal performance.”

The accuracy of the end-to-end system, although improved, does not yet meet the ideal standards.

“The accuracy of our end-to-end system, although improved, does not yet meet the ideal standards.”

No patient data or clinical trials used to report the scores were added to the OncoLLM training data.

“Importantly, no patient data or clinical trials that have been used to report the scores here were added to the training data.”

The N/A marginalization approach can resolve Boolean logic even when answers to some questions are unavailable, for example resolving Q1&Q2 to False when Q1 is No regardless of Q2.

“This approach also takes care of the cases when it is possible to answer the Boolean logic even when answers to a few questions are not available. For instance, if the criteria met expression is Q1&Q2, and we know the answer for Q1 is “No”, we do not need the answer to Q2. The expression should resolve to False regardless of the answer to Q2.”

• AssumptionRests on an assumption: The predefined logical tree for each criterion correctly captures the criterion's decision logic.

One major factor contributing to the recruitment bottleneck is the considerable challenge physicians encounter when systematically reviewing each patient against the list of available trials.

“One of the major factors contributing to this recruitment bottleneck is the considerable challenge that physicians encounter when systematically reviewing each patient against the list of available trials.”

• cites [6] Nuttall, A. Considerations for improving patient recruitment into clinical trials. Clinical Leader Newsletter http://vertassets.blob.core.windows.net/download/64c39d7e/64c39d7e-c643-457b-aec2-9ff7b65b3ad2/rdprecruitmentwhitepaper.pdf (2012).Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Empirical evaluations confirm the feasibility of using LLMs to systematically rank clinical trials against patient records and vice-versa.

“Our empirical evaluations confirm the feasibility of using LLMs to systematically rank clinical trials against patient records and vice-versa.”

In oncology, key parameters for clinical trial screening are virtually absent in structured EHRs, such as cancer staging.

“A recent study shows that this is particularly relevant for oncology, where key parameters for clinical trial screening are virtually absent in structured EHRs, such as cancer staging13.”

• cites [13] Shriver, S. et al. Feasibility of institution-agnostic, ehr-integrated regional clinical trial matching. Cancer 130, 60–67 (2024). sourceNot checked — no full text of the source

Smaller models, when fine-tuned appropriately, can surpass the performance of their proprietary counterparts.

“We demonstrate that smaller models, when fine-tuned appropriately, can surpass the performance of their proprietary counterparts.”

Leveraging LLMs, with carefully implemented controls, could significantly shift the paradigm for accruing and enrolling eligible patients in clinical trials.

“Our findings indicate that leveraging LLMs, with carefully implemented controls, could significantly shift the paradigm for accruing and enrolling eligible patients in clinical trials.”