Enhancing hepatopathy clinical trial efficiency: a secure, large language model-powered pre-screening pipeline

Traditional manual screening and automated approaches fall short in key areas.

“While traditional manual screening and automated approaches have attempted to address these challenges, they fall short in key areas [4].”

• cites [4] Ni Y, Wright J, Perentesis J, et al. Increasing the efficiency of trial-patient matching: automated clinical trial eligibility pre-screening for pediatric oncology patients. BMC Med Inf Decis Mak. 2015;15:28. https://doi.org/10.1186/s12911-015-0149-3. [published Online First: 2015/04/17]. sourceSupported

The pipeline showed promising precision-focused results in hepatocellular carcinoma trials (0.878).

“The pipeline showed promising precision-focused results in hepatocellular carcinoma (0.878) and cirrhosis trials (0.843).”

The pipeline achieved high efficiency of 0.44s per task.

“Our pipeline achieved a notable balance of high precision (e.g., 0.921, criteria level) and good overall recall (e.g., ~ 0.82, criteria level), alongside high efficiency (0.44s per task).”

The pipeline achieved a recall of approximately 0.82 at the criteria level.

“Our pipeline achieved a notable balance of high precision (e.g., 0.921, criteria level) and good overall recall (e.g., ~ 0.82, criteria level), alongside high efficiency (0.44s per task).”

Pathway A (majority vote) achieved a recall of 0.822 at the question level.

“Pathway A (majority vote) achieved a precision of 0.877 and recall of 0.822.”

Pathway B's F1 score at the criterion level was 0.820.

“The F1 scores (Pathway A: 0.835, Pathway B: 0.820) also reflect this balance.”

The dataset comprised over 16,000 anonymized admission notes from the Hepatology Department collected over the past 10 years.

“The dataset comprised over 16,000 anonymized admission notes from the Hepatology Department collected over the past 10 years.”

71.3% of Pathway A majority vote responses achieved precision above 0.85.

“a substantial proportion of responses achieved precision above 0.85, with 63.2% for Pathway B and 71.3% for Pathway A’s majority vote.”

Pathway A (majority vote) achieved a precision of 0.877 at the question level.

“Pathway A (majority vote) achieved a precision of 0.877 and recall of 0.822.”

Pathway B demonstrated comparable precision at 0.920 at the criterion level.

“Pathway B demonstrated comparable precision at 0.920 but a lower recall of 0.785.”

Pathway A (majority vote) achieved a recall of 0.819 at the criterion level.

“Pathway A (majority vote) achieved a precision of 0.922 and a recall of 0.819.”

Pathway B achieved a precision of 0.892 at the question level.

“Pathway B yielded a slightly higher precision (0.892) but lower recall (0.793).”

Pathway A's F1 score at the criterion level was 0.835.

“The F1 scores (Pathway A: 0.835, Pathway B: 0.820) also reflect this balance.”

Pathway A (majority vote) achieved a precision of 0.922 at the criterion level.

“Pathway A (majority vote) achieved a precision of 0.922 and a recall of 0.819.”

Pathway A (CRC role) showed high precision in direct match tasks.

“Pathway A (CRC role) showed high precision in direct match tasks (Table 2), maintaining a strong precision-recall balance (Supplemental Table S3).”

63.2% of Pathway B responses achieved precision above 0.85.

“a substantial proportion of responses achieved precision above 0.85, with 63.2% for Pathway B and 71.3% for Pathway A’s majority vote.”

Pathway B achieved a recall of 0.793 at the question level.

“Pathway B yielded a slightly higher precision (0.892) but lower recall (0.793).”

Both pathways demonstrated high annotation consistency, exceeding 97% across all roles and strategies.

“Both pathways demonstrated high annotation consistency, exceeding 97% across all roles and strategies (Fig. 2).”

Pathway B demonstrated lower recall of 0.785 at the criterion level.

“Pathway B demonstrated comparable precision at 0.920 but a lower recall of 0.785.”

Pathway A's JD role achieved the highest individual recall (0.884) at the question level.

“Notably, Pathway A’s JD role achieved the highest individual recall (0.884, Table 1), consistent with its design for comprehensive data retrieval.”

The pipeline achieved a precision of 0.921 at the criteria level.

“Our pipeline achieved a notable balance of high precision (e.g., 0.921, criteria level) and good overall recall (e.g., ~ 0.82, criteria level), alongside high efficiency (0.44s per task).”

For complex Classification tasks, Pathway B achieved precision generally in the 0.85-0.92 range.

“For more complex tasks, such as determining the etiology of liver failure, assessing whether liver cancer is primary, identifying systemic diseases, or verifying HBV antiviral treatment (largely Classification tasks), the Agent-Collab framework (Pathway B) achieved high precision (generally in the 0.85–0.92 range, as detailed in Table 2).”

• AssumptionRests on an assumption: The gold standard labels derived from admission notes are accurate.

Pathway A showed significantly faster processing times than Pathway B.

“Pathway A showed significantly faster processing times, averaging 0.588 s per question for CRC, 0.395 s for JD, and 0.340 s for IE.”

For complex Classification tasks, Pathway B recall was around 0.65-0.78.

“It is important to consider that this high precision in complex scenarios was accompanied by a moderate recall profile (around 0.65–0.78 for Pathway B in these tasks, according to Supplemental Table S3), illustrating a clear precision-recall trade-off where this pathway prioritizes the accuracy of positive identifications in such nuanced scenarios.”

• AssumptionRests on an assumption: The gold standard labels derived from admission notes are accurate.

The pipeline showed promising precision-focused results in cirrhosis trials (0.843).

“The pipeline showed promising precision-focused results in hepatocellular carcinoma (0.878) and cirrhosis trials (0.843).”

Pathway B had the lowest counterfactual inference rate at 0.25%.

“Pathway B had the lowest rate at 0.25%, indicating strong consistency.”

In Pathway A, the counterfactual inference rate was 0.93% for IE.

“In Pathway A, the rates varied among roles, with CRC at 0.82%, IE at 0.93%, and JD at 1.78%.”

Compared to similar studies in Table 4, the pipeline showed competitive or superior performances in both precision and recall.

“Compared to similar studies in Table 4, our pipeline showed competitive or superior performances in both precision and recall, even when using a closed-source LLM.”

• AssumptionRests on an assumption: The gold standard labels derived from admission notes are accurate.

Pathway B precision (0.920) was higher than hybrid baseline (0.737).

“Notably, compared to the included traditional non-LLM baselines, our LLM-driven pathways indicated clear advantages in key metrics such as precision (e.g., Pathway A: 0.922 vs. 0.540 rule-based; Pathway B: 0.920 vs. 0.737 hybrid) and recall (Pathway B: 0.785 vs. 0.560 hybrid).”

• AssumptionRests on an assumption: The gold standard labels derived from admission notes are accurate.

In Pathway A, the counterfactual inference rate was 0.82% for CRC.

“In Pathway A, the rates varied among roles, with CRC at 0.82%, IE at 0.93%, and JD at 1.78%.”

Pathway B recall (0.785) was higher than hybrid baseline (0.560).

“Notably, compared to the included traditional non-LLM baselines, our LLM-driven pathways indicated clear advantages in key metrics such as precision (e.g., Pathway A: 0.922 vs. 0.540 rule-based; Pathway B: 0.920 vs. 0.737 hybrid) and recall (Pathway B: 0.785 vs. 0.560 hybrid).”

• AssumptionRests on an assumption: The gold standard labels derived from admission notes are accurate.

The pipeline achieved precision of 0.878 in hepatocellular carcinoma trials.

“Despite these constraints, the pipeline achieved higher precision in hepatocellular carcinoma (0.878) and cirrhosis trials (0.843), indicating its potential utility in hepatopathy trials.”

• AssumptionRests on an assumption: The admission notes contain sufficient information to determine patient eligibility for the selected clinical trials.
• AssumptionRests on an assumption: The gold standard labels derived from admission notes are accurate.

Pathway A (majority vote) achieved recall of 0.819.

“For instance, Pathway A (majority vote) achieved an F1-score of 0.835, supported by its precision of 0.922 and recall of 0.819 (Table 3).”

• AssumptionRests on an assumption: The gold standard labels derived from admission notes are accurate.

Pathway A average processing time per question was 0.340 s for IE.

“Pathway A showed significantly faster processing times, averaging 0.588 s per question for CRC, 0.395 s for JD, and 0.340 s for IE.”

Pathway A precision (0.922) was higher than rule-based baseline (0.540).

“Notably, compared to the included traditional non-LLM baselines, our LLM-driven pathways indicated clear advantages in key metrics such as precision (e.g., Pathway A: 0.922 vs. 0.540 rule-based; Pathway B: 0.920 vs. 0.737 hybrid) and recall (Pathway B: 0.785 vs. 0.560 hybrid).”

• AssumptionRests on an assumption: The gold standard labels derived from admission notes are accurate.

Pathway A average processing time per question was 0.588 s for CRC.

“Pathway A showed significantly faster processing times, averaging 0.588 s per question for CRC, 0.395 s for JD, and 0.340 s for IE.”

Pathway A typical processing time was 0.44 s per task.

“This was achieved with minimal processing time (a typical of 0.44 s per task, verse traditional manual annotation which take over 10 s per task, representing a 95% reduction [34]).”

Pathway B exhibited a wider range of processing times across questions.

“Pathway B exhibited a wider range due to the possibility of a second evaluation round.”

Trial-level precision averaged 0.72.

“Additionally, we assessed the trial-level precision (aggregation rules detailed in Supplemental Material S7), achieving an average of 0.72, considering the limitations of admission notes in determining patient eligibility.”

• AssumptionRests on an assumption: The admission notes contain sufficient information to determine patient eligibility for the selected clinical trials.
• AssumptionRests on an assumption: The gold standard labels derived from admission notes are accurate.

Pathway B average processing time per question was 2.570 s.

“In contrast, Pathway B took an average of 2.570 s per question (Fig. 4).”

Pathway A (majority vote) achieved precision of 0.922.

“For instance, Pathway A (majority vote) achieved an F1-score of 0.835, supported by its precision of 0.922 and recall of 0.819 (Table 3).”

• AssumptionRests on an assumption: The gold standard labels derived from admission notes are accurate.

The pipeline achieved precision of 0.843 in cirrhosis trials.

“Despite these constraints, the pipeline achieved higher precision in hepatocellular carcinoma (0.878) and cirrhosis trials (0.843), indicating its potential utility in hepatopathy trials.”

• AssumptionRests on an assumption: The admission notes contain sufficient information to determine patient eligibility for the selected clinical trials.
• AssumptionRests on an assumption: The gold standard labels derived from admission notes are accurate.

Pathway A (majority vote) achieved F1-score of 0.835.

“For instance, Pathway A (majority vote) achieved an F1-score of 0.835, supported by its precision of 0.922 and recall of 0.819 (Table 3).”

• AssumptionRests on an assumption: The gold standard labels derived from admission notes are accurate.

Pathway A average processing time per question was 0.395 s for JD.

“Pathway A showed significantly faster processing times, averaging 0.588 s per question for CRC, 0.395 s for JD, and 0.340 s for IE.”

For non-inferential recognition tasks, Pathway A with majority vote offers high recall typically >0.80.

“For non-inferential recognition tasks involving explicit diagnoses and interventions (i.e., Direct Match tasks), pathway A with majority vote is recommend, offering a strong balance of high precision (typically > 0.85 as shown in Table 2) and high recall (typically > 0.80 as shown in Supplemental Table S3) for these tasks”

• AssumptionRests on an assumption: The gold standard labels derived from admission notes are accurate.

For complex reasoning tasks, Pathway B has a moderate recall profile around 0.65-0.78.

“For tasks requiring complex reasoning (i.e., Classification tasks), Pathway B is more suitable, providing excellent precision (generally in the 0.85–0.92 range according to Table 2), ensuring high reliability of positive identifications, alongside a moderate recall profile (around 0.65–0.78 based on Supplemental Table S3) that highlights a clear precision-recall trade-off for these demanding tasks”

• AssumptionRests on an assumption: The gold standard labels derived from admission notes are accurate.

In Pathway A, the counterfactual inference rate was 1.78% for JD.

“In Pathway A, the rates varied among roles, with CRC at 0.82%, IE at 0.93%, and JD at 1.78%.”

For non-inferential recognition tasks (Direct Match), Pathway A with majority vote offers high precision typically >0.85.

“For non-inferential recognition tasks involving explicit diagnoses and interventions (i.e., Direct Match tasks), pathway A with majority vote is recommend, offering a strong balance of high precision (typically > 0.85 as shown in Table 2) and high recall (typically > 0.80 as shown in Supplemental Table S3) for these tasks”

• AssumptionRests on an assumption: The gold standard labels derived from admission notes are accurate.

The majority vote approach reduced Pathway A's overall counterfactual inference rate to 0.77%.

“The majority vote approach effectively reduced Pathway A’s overall counterfactual inference rate to 0.77%, demonstrating the benefit of collective decision-making in minimizing errors.”

Pathway A (particularly CRC and IE roles) excelled in Direct Match tasks, achieving a strong balance of high precision and good recall.

“In contrast, Pathway A (particularly its CRC and IE roles) excelled in direct recognition tasks (Direct Match tasks), achieving a strong balance of both high precision and good recall (as detailed in Table 2 and Supplemental Table S3 respectively).”

• AssumptionRests on an assumption: The gold standard labels derived from admission notes are accurate.

For complex reasoning tasks (Classification), Pathway B provides precision generally in the 0.85-0.92 range.

“For tasks requiring complex reasoning (i.e., Classification tasks), Pathway B is more suitable, providing excellent precision (generally in the 0.85–0.92 range according to Table 2), ensuring high reliability of positive identifications, alongside a moderate recall profile (around 0.65–0.78 based on Supplemental Table S3) that highlights a clear precision-recall trade-off for these demanding tasks”

• AssumptionRests on an assumption: The gold standard labels derived from admission notes are accurate.

LLMs offer a promising avenue to overcome the limitations of BERT-based tools due to superior contextual semantic understanding.

“Large language models (LLMs), with their superior contextual semantic understanding, offer a promising avenue to overcome these limitations [8, 9].”

• cites [8] Lee P, Bubeck S, Petro JJNEJM. Benefits, limits, and risks of GPT-4 as an AI chatbot for medicine. 2023;388(13):1233–39. sourceNot checked — no full text of the source
• cites [9] Lee P, Goldberg C, Kohane I. The AI revolution in medicine: GPT-4 and beyond: Pearson, 2023.Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Both pathways are highly precise at the criterion level.

“These results indicate that while both pathways are highly precise at the criterion level, Pathway A’s majority vote strategy was more effective in capturing a larger proportion of criteria-meeting patients due to its higher recall.”

Pathway A's majority vote strategy was more effective in capturing a larger proportion of criteria-meeting patients due to its higher recall.

“These results indicate that while both pathways are highly precise at the criterion level, Pathway A’s majority vote strategy was more effective in capturing a larger proportion of criteria-meeting patients due to its higher recall.”

The 58 selected criteria were assessable using information typically documented in admission notes.

“We manually reviewed the inclusion and exclusion criteria of six real-world clinical trials on hepatopathy, and carefully selected 58 criteria assessable using information typically documented in admission notes.”

Clinical experts manually reviewed and annotated the converted question sets from 4,000 admission notes to establish a robust gold standard.

“To establish a robust gold standard for all performance evaluations at the question level, clinical experts manually reviewed and annotated the converted question sets from 4,000 admission notes.”

Pathway B excelled in high-precision complex reasoning.

“Pathway B excelled in high-precision complex reasoning (while exhibiting a specific recall profile conducive to accuracy), whereas Pathway A was particularly effective for tasks requiring both robust precision and recall (e.g., direct data extraction), often with faster processing times.”

The reported performance metrics reflect fidelity to admission notes rather than an absolute measure against more comprehensive patient records.

“While this ensures a fair evaluation of the pipeline against expert interpretation of these specific documents, the inherent scope of admission notes means that the reported performance metrics reflect fidelity to this data source rather than an absolute measure against more comprehensive patient records.”

Pathway B ensures high reliability of positive identifications for complex tasks.

“For tasks requiring complex reasoning (i.e., Classification tasks), Pathway B is more suitable, providing excellent precision (generally in the 0.85–0.92 range according to Table 2), ensuring high reliability of positive identifications, alongside a moderate recall profile (around 0.65–0.78 based on Supplemental Table S3) that highlights a clear precision-recall trade-off for these demanding tasks”

The single-center data may limit the generalizability of the findings.

“While covering multiple liver diseases, this may limit the generalizability of our findings.”

Combining advanced LLM capabilities with clinical expertise showed potential for improving overall effectiveness of complex clinical assessments.

“By combining advanced LLM capabilities with clinical expertise, both strategies showed potential for improving the overall effectiveness (considering both precision and recall) of complex clinical assessments (Fig. 5).”

Pathway B's preset stance Agent-Collab strategy effectively managed complex and semantically disordered texts, leading to highly precise outputs.

“Meanwhile, Pathway B’s preset stance Agent-Collab strategy effectively managed complex and semantically disordered texts, leading to highly precise outputs in such scenarios (as shown in Table 2), though this was often balanced with a moderate recall profile that prioritized the certainty of identified cases (Supplemental Table S3).”

The pipeline achieved a 95% reduction in processing time compared to manual annotation.

“This was achieved with minimal processing time (a typical of 0.44 s per task, verse traditional manual annotation which take over 10 s per task, representing a 95% reduction [34]).”

The gold standard was derived from admission notes.

“Our gold standard was also derived from these notes.”

Traditional manual annotation takes over 10 s per task.

“This was achieved with minimal processing time (a typical of 0.44 s per task, verse traditional manual annotation which take over 10 s per task, representing a 95% reduction [34]).”

• cites [34] Wei Q, Franklin A, Cohen T, Xu H. Clinical text annotation - what factors are associated with the cost of time? AMIA Annu Symp Proc. 2018;2018:1552–60. published Online First: 2019/03/01.Not checked — no full text of the source

Pathway A's anthropomorphized experts guided the LLM from general semantic understanding to targeted entity recognition, enhancing performance of weaker open-source LLMs.

“Pathway A’s anthropomorphized experts guided the LLM from general semantic understanding to targeted entity recognition, a critical feature for enhancing the performance of weaker open-source LLMs.”

The data were sourced from a single, regionally influential hepatology center.

“Secondly, the data in this study were sourced from a single, regionally influential hepatology center.”

Pathway B is particularly suited for complex scenarios where accuracy of positive identifications is paramount.

“This fundamental difference in their precision-recall characteristics highlights their complementary strengths: Pathway B is particularly suited for complex scenarios where the accuracy of positive identifications is paramount, supported by its lower counterfactual inference rates.”

• AssumptionRests on an assumption: A lower counterfactual inference rate indicates greater trustworthiness.

LLM-driven pathways indicated clear advantages in precision and recall compared to traditional non-LLM baselines.

“Notably, compared to the included traditional non-LLM baselines, our LLM-driven pathways indicated clear advantages in key metrics such as precision (e.g., Pathway A: 0.922 vs. 0.540 rule-based; Pathway B: 0.920 vs. 0.737 hybrid) and recall (Pathway B: 0.785 vs. 0.560 hybrid).”

There is a clear precision-recall trade-off for complex reasoning tasks.

“For tasks requiring complex reasoning (i.e., Classification tasks), Pathway B is more suitable, providing excellent precision (generally in the 0.85–0.92 range according to Table 2), ensuring high reliability of positive identifications, alongside a moderate recall profile (around 0.65–0.78 based on Supplemental Table S3) that highlights a clear precision-recall trade-off for these demanding tasks”

There is a clear precision-recall trade-off for Pathway B in complex tasks, prioritizing accuracy of positive identifications.

“It is important to consider that this high precision in complex scenarios was accompanied by a moderate recall profile (around 0.65–0.78 for Pathway B in these tasks, according to Supplemental Table S3), illustrating a clear precision-recall trade-off where this pathway prioritizes the accuracy of positive identifications in such nuanced scenarios.”

Patients identified via Pathway B are more reliably accurate.

“This ensures that patients identified via Pathway B (often chosen for its precision in specific complex tasks, despite its focused recall in those tasks as per Supplemental Table S3) are more reliably accurate.”

• AssumptionRests on an assumption: A lower counterfactual inference rate indicates greater trustworthiness.

The pipeline combines different approaches to achieve precision, recall and efficiency across various clinical tasks.

“Designed to operate efficiently in environments with limited IT resources and stringent security requirements, our pipeline combines different approaches to achieve precision, recall and efficiency across various clinical tasks.”

Pathway B's low counterfactual inference rate (0.25%) is a significant advantage for trustworthiness.

“Pathway B exhibited the lowest counterfactual inference rate (0.25%), a significant advantage for trustworthiness.”

• AssumptionRests on an assumption: A lower counterfactual inference rate indicates greater trustworthiness.

Some criteria had a limited number of positive instances.

“Furthermore, some criteria have limited number of positive instances, making recall sensitive to misclassifications, which should be considered when interpreting these specific recall figures.”

The pipeline has potential utility in hepatopathy trials.

“Despite these constraints, the pipeline achieved higher precision in hepatocellular carcinoma (0.878) and cirrhosis trials (0.843), indicating its potential utility in hepatopathy trials.”

• AssumptionRests on an assumption: The admission notes contain sufficient information to determine patient eligibility for the selected clinical trials.
• AssumptionRests on an assumption: The gold standard labels derived from admission notes are accurate.

The pipeline's efficiency, adaptability, and balanced performance profile make it suitable for improving patient recruitment.

“Its efficiency, adaptability, and balanced performance profile make it suitable for improving patient recruitment.”

• AssumptionRests on an assumption: The 4,000 randomly selected records were representative of the overall dataset in terms of distribution of hepatopathy and associated clinical presentations.