Development and validation of an autonomous artificial intelligence agent for clinical decision-making in oncology

May combine two claims.

GPT-4 alone only generates very generic or wrong responses.

“As we have shown, GPT-4 alone only generates very generic or wrong responses.”

• AssumptionRests on an assumption: The 20 simulated patient cases are realistic and representative enough of real-world oncology clinical decision-making for the agent’s performance to be meaningful.

May combine two claims.

For patient W, the model used vision transformer models to assess the patient's mutational status, confirming the presence of a suspected BRAF mutation, and then queried OncoKB for management information.

“for patient W (Supplementary Note 2), the model used vision transformer models to assess the patient’s mutational status, confirming the presence of a suspected BRAF mutation. The model then queried the OncoKB database to retrieve medical information on appropriate management regarding the mutation.”

• AssumptionRests on an assumption: The 20 simulated patient cases are realistic and representative enough of real-world oncology clinical decision-making for the agent’s performance to be meaningful.

May combine two claims.

Llama frequently used superfluous tools, including random calculations on nonsense values or hallucinating tumor locations during imaging analysis, leading to 62 unnecessary tool calls and failures across all 20 patient cases.

“we saw that the Llama model frequently used superfluous tools, for example, performing random calculations on nonsense values or hallucinating (inventing) tumor locations during imaging analysis that did not exist. This led to 62 unnecessary tool calls and failures (‘not required, failed’) across all 20 patient cases evaluated.”

• AssumptionRests on an assumption: The 20 simulated patient cases are realistic and representative enough of real-world oncology clinical decision-making for the agent’s performance to be meaningful.

May combine two claims.

Of 64 required tool invocations, the agent correctly used 56, achieving an overall success rate of 87.5%, with no failures among the required tools.

“Of 64 required tool invocations to fully solve all given patient cases, the agent correctly used 56, achieving an overall success rate of 87.5%, with no failures among the required tools.”

• AssumptionRests on an assumption: The 20 simulated patient cases are realistic and representative enough of real-world oncology clinical decision-making for the agent’s performance to be meaningful.

May combine two claims.

Mixtral frequently disregarded tool use, resulting in fewer than one in ten tools running successfully.

“The major shortcoming of the Mixtral model was its frequent disregard for tool use, resulting in fewer than one in ten tools running successfully.”

• AssumptionRests on an assumption: The 20 simulated patient cases are realistic and representative enough of real-world oncology clinical decision-making for the agent’s performance to be meaningful.

May combine two claims.

For patient G, GPT-4 invoked MedSAM twice to obtain segmentation masks from two images at different time points and used the calculator to determine that the tumor had grown by a factor of 2.14.

“in the tool results for patient G in Supplementary Note 2, GPT-4 invoked the MedSAM tool twice to obtain segmentation masks from two images taken at different time points. Subsequently, GPT-4 used the calculator function with the appropriate values from MedSAM to determine that the tumor had grown by a factor of 2.14.”

• AssumptionRests on an assumption: The 20 simulated patient cases are realistic and representative enough of real-world oncology clinical decision-making for the agent’s performance to be meaningful.

May combine two claims.

In three examples, tools and retrieval enabled the LLM to accurately solve cases, whereas GPT-4 alone either stated it could not solve the patient case and provided hypothetical answers or drew incorrect conclusions.

“in Fig. 2a, we highlight three examples where tools and retrieval enabled the LLM to accurately solve cases, whereas, without these enhancements, GPT-4 either stated that it could not solve the patient case and provided hypothetical answers instead or drew incorrect conclusions”

• AssumptionRests on an assumption: The 20 simulated patient cases are realistic and representative enough of real-world oncology clinical decision-making for the agent’s performance to be meaningful.

May combine two claims.

The agent was capable of resolving issues even when contradictory information was provided in a patient's description, such as discrepancies in reported mutations versus testing results; it pointed out inconsistencies, recommended further genetic confirmation and outlined potential treatment options.

“the agent was capable of resolving issues, even in instances where contradictory information was provided in a patient’s description, such as discrepancies in reported mutations versus results of testing such mutations using tools. In such cases, the agent pointed out these inconsistencies, recommended further genetic confirmation and outlined potential treatment options based on the results (pat”

• AssumptionRests on an assumption: The 20 simulated patient cases are realistic and representative enough of real-world oncology clinical decision-making for the agent’s performance to be meaningful.

May combine two claims.

The agent's responses achieved an 87.2% success rate, with only 14 instances not being covered.

“However, our agent’s responses achieved a 87.2% success rate, with only 14 instances not being covered”

• AssumptionRests on an assumption: The 20 simulated patient cases are realistic and representative enough of real-world oncology clinical decision-making for the agent’s performance to be meaningful.
• AssumptionRests on an assumption: The human experts’ predetermined expected answers and keywords, and their majority-vote evaluations, constitute a valid ground truth for correctness, completeness, helpfulness and citation accuracy.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Built-in guardrails designed to prevent potentially harmful content occasionally led to refusals and to inadequately addressing medical queries.

“Additionally, built-in guardrails designed to prevent potentially harmful content occasionally led to refusals and to inadequately addressing medical queries.”

May combine two claims.

Both models struggle to identify necessary tools for a given patient context, with 18.8% of required tools remaining unused by Llama and 42.2% for Mistral.

“First, both models struggle to identify necessary tools for a given patient context (18.8% of required tools remain unused by Llama and even 42.2% for Mistral).”

• AssumptionRests on an assumption: The 20 simulated patient cases are realistic and representative enough of real-world oncology clinical decision-making for the agent’s performance to be meaningful.

May combine two claims.

Both open-weight models Llama-3 70B and Mixtral 8x7B demonstrate only extremely poor function-calling performance.

“Overall, our findings reveal that both open-weight models demonstrate only extremely poor function-calling performance.”

• AssumptionRests on an assumption: The 20 simulated patient cases are realistic and representative enough of real-world oncology clinical decision-making for the agent’s performance to be meaningful.

May combine two claims.

Of 257 citations, 194 (75.5%) were accurately aligned with the model's assertions, 59 (23.0%) were unrelated, and 4 (1.6%) were in conflict with the model's statement.

“of the 257 citations provided in the models’ responses, 194 (75.5%) were accurately aligned with the model’s assertions, while 59 (23.0%) were found to be unrelated and merely 4 (1.6%) were found to be in conflict with the model’s statement.”

• AssumptionRests on an assumption: The 20 simulated patient cases are realistic and representative enough of real-world oncology clinical decision-making for the agent’s performance to be meaningful.
• AssumptionRests on an assumption: The human experts’ predetermined expected answers and keywords, and their majority-vote evaluations, constitute a valid ground truth for correctness, completeness, helpfulness and citation accuracy.

May combine two claims.

For Llama and Mixtral, overall tool success rates were 39.1% and 7.8%, respectively.

“Consequently, for Llama and Mixtral, the overall success rates were low, registering only 39.1% (Llama) and 7.8% (Mixtral) (‘required, successful’).”

• AssumptionRests on an assumption: The 20 simulated patient cases are realistic and representative enough of real-world oncology clinical decision-making for the agent’s performance to be meaningful.

Classified as probably not a substantial claim. May combine two claims.

The decision regarding the use and timing of tools is dependent only on the agent's reasoning, unlike the retrieval phase which was manually enforced at each invocation.

“Unlike the retrieval phase, which we manually enforced at each invocation, the decision regarding the use and timing of tools is dependent only on the agent’s reasoning.”

May combine two claims.

Of 245 assessable statements, 223 (91.0%) were factually correct, 16 (6.5%) were incorrect, and 6 (2.4%) were flagged as potentially harmful.

“resulting in a total of 245 assessable statements. Our evaluations found 223 (91.0%) of these to be factually correct and 16 (6.5%) to be incorrect, while 6 (2.4%) were flagged as potentially harmful.”

• AssumptionRests on an assumption: The 20 simulated patient cases are realistic and representative enough of real-world oncology clinical decision-making for the agent’s performance to be meaningful.
• AssumptionRests on an assumption: The human experts’ predetermined expected answers and keywords, and their majority-vote evaluations, constitute a valid ground truth for correctness, completeness, helpfulness and citation accuracy.

May combine two claims.

GPT-4 can track the results from MedSAM back to the original image date because it has access to image file names that are based on date, ensuring accurate decision-making of lesion development over time.

“GPT-4 can track the results from MedSAM back to the original image date, as it has access to image file names that are based on date. This capability ensures accurate decision-making of lesion development over time.”

Probably not a substantial claim: may not pass S4 (based).

All necessary information for calling the designated tools is derivable or producible from the given patient context.

“All necessary information for calling the designated tools is derivable or producible from the given patient context.”

Probably not a substantial claim: may not pass S3 (load-bearing).

In oncology, LLMs can serve as a rapid and reliable reference tool, for example by providing medical recommendation suggestions from official medical guidelines.

“In oncology, where clinical decision-making is increasingly complex, LLMs can serve as a rapid and reliable reference tool, for instance, by providing medical recommendation suggestions from official medical guidelines4.”

• cites [4] Ferber, D. et al. GPT-4 for information retrieval and comparison of medical oncology guidelines. NEJM AI 1, AIcs2300235 (2024). sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

Several multimodal AI systems have been introduced, including models that analyze radiology images together with clinical data or integrate histopathology with genomic or text-based information.

“Recently, several such AI systems have been introduced6. Examples include models that analyze radiology images together with clinical data7 or integrate information from histopathology with genomic8 or text-based information9.”

• cites [6] Lipkova, J. et al. Artificial intelligence for multimodal data integration in oncology. Cancer Cell 40, 1095–1110 (2022). sourceNot checked yet

  Not checked in the free run.

• cites [7] Khader, F. et al. Multimodal deep learning for integrating chest radiographs and clinical parameters: a case for transformers. Radiology 309, e230806 (2023). sourceNot checked yet

  Not checked in the free run.

• cites [8] Chen, R. J. et al. Pan-cancer integrative histology-genomic analysis via multimodal deep learning. Cancer Cell 40, 865–878 (2022). sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

GPT-4 has achieved a passing score on the United States Medical Licensing Examination and can provide detailed explanations for its responses.

“GPT-4 has achieved a passing score on the United States Medical Licensing Examination and is able to provide detailed explanations for its responses3.”

• cites [3] Nori, H., King, N., McKinney, S. M., Carignan, D. & Horvitz, E. Capabilities of GPT-4 on medical challenge problems. Preprint at https://doi.org/10.48550/arXiv.2303.13375 (2023). sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S4 (based). May combine two claims.

The distribution of human diseases is wide and complex and is not captured in current performance benchmarks, where models are predominantly evaluated on a single specific task at a time.

“The distribution of human diseases is wide and complex, which is not captured in current performance benchmarks, where these models are predominantly evaluated on a single specific task at a time.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Previous work has shown that limitations of LLMs can partially be overcome by enriching them with domain-specific information through fine-tuning or retrieval-augmented generation.

“Previous work has shown that some of these limitations can partially be overcome by enriching LLMs with domain-specific information. This can be achieved through fine-tuning14 or retrieval-augmented generation (RAG)15”

• cites [14] Chen, Z. et al. MEDITRON-70B: scaling medical pretraining for large language models. Preprint at https://doi.org/10.48550/arXiv.2311.16079 (2023). sourceNot checked yet

  Not checked in the free run.

• cites [15] Lewis, P. et al. Retrieval-augmented generation for knowledge-intensive NLP tasks. In NIPS '20: Proc. 34th International Conference on Neural Information Processing Systems 793, 9459–9474 (2020).Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

RAG temporarily enhances an LLM's knowledge by incorporating relevant text excerpts from authoritative sources into the model.

“retrieval-augmented generation (RAG)15, a process that temporarily enhances an LLM’s knowledge by incorporating relevant text excerpts from authoritative sources into the model, such as medical guidelines16 or textbooks.”

• cites [15] Lewis, P. et al. Retrieval-augmented generation for knowledge-intensive NLP tasks. In NIPS '20: Proc. 34th International Conference on Neural Information Processing Systems 793, 9459–9474 (2020).Not checked yet

  Not checked in the free run.

• cites [16] Zakka, C. et al. Almanac—retrieval-augmented language models for clinical medicine. NEJM AI 1, AIoa2300068 (2024). sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

The future of medical artificial intelligence is multimodal.

“the future of medical artificial intelligence (AI) is multimodal5.”

• cites [5] Acosta, J. N., Falcone, G. J., Rajpurkar, P. & Topol, E. J. Multimodal biomedical AI. Nat. Med. 28, 1773–1784 (2022). sourceNot checked yet

  Not checked in the free run.

May combine two claims.

The study does not possess annotated ground-truth segmentation masks for comparison to MedSAM but uses clinical endpoints such as progressive disease or tumor shrinkage as the primary metric to measure outcomes.

“we do not possess annotated ground-truth segmentation masks for comparison to MedSAM but use clinical endpoints such as progressive disease or tumor shrinkage as our primary metric to measure outcomes”

May combine two claims.

A notable restriction is the provision of only a singular slice of radiology images and the yet limited capabilities of GPT-4V in interpreting medical images.

“One notable restriction for instance lies in the provision of only a singular slice of radiology images and the yet limited capabilities of GPT-4V (ref. 32) in interpreting medical images.”

• cites [32] OpenAI. GPT-4V(ision) system card. Available online at https://cdn.openai.com/papers/GPTV_System_Card.pdf (2023).Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

The investigators were not blinded to allocation during experiments and outcome assessment.

“The investigators were not blinded to allocation during experiments and outcome assessment; however, they were blinded to the model responses while establishing the ground truth for the necessity of tool use and the completeness of the responses.”

May combine two claims.

Combining precision medicine tools with an LLM enhances its capabilities in problem solving, aligning with the concept of using LLMs as 'reasoning engines'.

“Our results demonstrate that combining precision medicine tools with an LLM enhances its capabilities in problem solving, aligning with the concept of using LLMs as ‘reasoning engines’ (ref. 30).”

• cites [30] Truhn, D., Reis-Filho, J. S. & Kather, J. N. Large language models should be used as scientific reasoning engines, not knowledge databases. Nat. Med. 29, 2983–2984 (2023). sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

The model was programmatically configured to incorporate citations for each statement (maximum two consecutive sentences) using DSPy suggestions.

“the model was programmatically configured to incorporate citations for each statement (as defined as a maximum of two consecutive sentences) using DSPy suggestions49.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The investigators were blinded to the model responses while establishing the ground truth for the necessity of tool use and the completeness of the responses.

“The investigators were not blinded to allocation during experiments and outcome assessment; however, they were blinded to the model responses while establishing the ground truth for the necessity of tool use and the completeness of the responses.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Vision transformers in the study were trained and tested on data from The Cancer Genome Atlas (TCGA).

“vision transformers in our study were trained and tested on data from The Cancer Genome Atlas (TCGA)”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Arasteh et al. used the integrated data analysis tools of an LLM to analyze scientific data, achieving results on par with human researchers.

“Arasteh et al. used the integrated data analysis tools of an LLM to analyze scientific data, achieving results on par with human researchers19.”

• cites [19] Tayebi Arasteh, S. et al. Large language models streamline automated machine learning for clinical studies. Nat. Commun. 15, 1603 (2024). sourceNot checked yet

  Not checked in the free run.

May combine two claims.

Integrating three core elements (a reasoning engine, a knowledge database and tools) enables effectively addressing the limitation of GPT-4 alone, greatly improving accuracy and reliability.

“However, integrating three core elements (a reasoning engine, a knowledge database and tools) enables us to effectively address this limitation, greatly improving accuracy and reliability.”

Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

The modular approach offers far superior explainability compared to a large, generalist black-box model, as physicians can investigate the output from each individual tool separately.

“Crucially, this modular approach also offers far superior explainability compared to a large, generalist black-box model, as physicians can investigate the output from each individual tool separately.”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

Individual tools require better independent optimization and validation.

“it is essential to recognize that individual tools require better independent optimization and validation.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Reproducibility of the study's findings may be affected by silent changes implemented to the model by its developers and potential future model deprecations.

“Reproducibility of the study’s findings may be affected by silent changes implemented to the model by its developers, which may not always be publicly disclosed, as well as by potential future model deprecations.”

Probably not a substantial claim: may not pass S3 (load-bearing).

In instances of refusal, each sample was rerun in newly instantiated settings until no such refusals occurred.

“In these instances, we reran each sample in newly instantiated settings until no such refusals occurred.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Previous work already showed that LLMs can reliably identify temporal differences in medical documents.

“Previous work already showed that LLMs can reliably identify temporal differences in medical documents4.”

• cites [4] Ferber, D. et al. GPT-4 for information retrieval and comparison of medical oncology guidelines. NEJM AI 1, AIcs2300235 (2024). sourceNot checked yet

  Not checked in the free run.

May combine two claims.

The pipeline is able to autonomously handle multiple steps, including determining the need for specific tools, locating relevant data, understanding chronological order, sending requests, receiving results and integrating these into next steps, without human supervision.

“we show that our pipeline is able to autonomously handle multiple steps, such as determining the need for specific tools, locating the relevant data, understanding their chronological order, sending requests to the appropriate tool, receiving results and integrating these into the next steps of its decision-making”

• Also stated in: § Radiology tools improve GPT-4’s treatment accuracy · plain

  “All of these steps are managed fully autonomously, without human supervision, by the LLM agent.”

• AssumptionRests on an assumption: The 20 simulated patient cases are realistic and representative enough of real-world oncology clinical decision-making for the agent’s performance to be meaningful.

Probably not a substantial claim: may not pass S3 (load-bearing).

No data were excluded in the analysis.

“No data were excluded in the analysis.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

Some radiology images available online may have been included in the pretraining of GPT-4V.

“some radiology images available online may have been included in the pretraining of GPT-4V.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The vision tool approach is currently restricted to single-slice images that need to be taken at the same magnification.

“A potential limitation of the vision tool approach is that, for both scenarios, we currently restrict the use to single-slice images that additionally need to be taken at the same magnification.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The overall analysis was approved by the Ethics Commission of the Medical Faculty of the Technical University Dresden (BO-EK-444102022).

“The overall analysis was approved by the Ethics Commission of the Medical Faculty of the Technical University Dresden (BO-EK-444102022).”

Probably not a substantial claim: may not pass S3 (load-bearing).

The evaluation is currently confined to a single interaction.

“despite being implemented as a multiturn chat agent, our evaluation is currently confined to a single interaction.”

May combine two claims.

GPT-4 with tools and RAG can deliver highly accurate, helpful and well-cited responses, enhancing its ability to handle complex medical scenarios and provide support for clinical decision-making.

“Overall, our results show that GPT-4 with tools and RAG can deliver highly accurate, helpful and well-cited responses, which enhances its ability to handle complex medical scenarios and provide support for clinical decision-making.”

• AssumptionRests on an assumption: The 20 simulated patient cases are realistic and representative enough of real-world oncology clinical decision-making for the agent’s performance to be meaningful.

Probably not a substantial claim: may not pass S3 (load-bearing).

The study used generalist embedding, retrieval and reranking models.

“we used generalist embedding, retrieval and reranking models”

May combine two claims.

Existing biomedical benchmarks and evaluation datasets are designed for one or two data modalities and are restricted to closed question-and-answer formats.

“Existing biomedical benchmarks and evaluation datasets are designed for one or two data modalities26 and are restricted to closed question-and-answer formats.”

• cites [26] He, X., Zhang, Y., Mou, L., Xing, E. & Xie, P. PathVQA: 30000+ questions for medical visual question answering. Preprint at https://doi.org/10.48550/arXiv.2003.10286 (2020). sourceNot checked yet

  Not checked in the free run.

Classified as probably not a substantial claim. May combine two claims.

For all benchmarks, the majority vote was reported; in cases of a tie, the most adverse outcome was selected according to a hierarchical schema.

“For all benchmarks, we reported the majority vote. In cases of a tie, we selected the most adverse outcome, adhering to a hierarchical schema: correct, irrelevant and wrong for citations and correct, wrong and harmful for the correctness evaluation.”

Probably not a substantial claim: may not pass S2 (checkable).

By prioritizing clinical endpoints to validate the agent pipeline, the study can offer new, directly relevant evidence and metrics to address the current scarcity of objective measures for AI agents in healthcare.

“Overall, we believe that, by prioritizing clinical endpoints to validate our agent pipeline, we can offer new, directly relevant evidence and metrics to address the current scarcity of objective measures for AI agents in healthcare.”

Classified as probably not a substantial claim.

Experiments were limited to n = 1 per case because of stringent rate and access limitations at the time of experimentation.

“Experiments were limited to n = 1 per case because of stringent rate and access limitations at the time of experimentation, when access to GPT-4V was available only as a preview.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

The current agent architecture is a static design choice.

“Moreover, our current agent architecture is a static design choice.”

Probably not a substantial claim: may not pass S3 (load-bearing).

In-house CT/MRI slices were selected by an experienced radiologist from the team who was blinded to the rest of the study at the time of image selection.

“Our in-house slices were selected by an experienced radiologist from our team, who was blinded to the rest of the study at the time of image selection.”

Probably not a substantial claim: may not pass S4 (based).

Current regulatory restrictions make GPT-4 unsuitable for real-world settings because of its cloud-based nature, which necessitates transferring sensitive patient data to proprietary commercial servers.

“current regulatory restrictions make GPT-4 unsuitable because of its cloud-based nature, which necessitates transferring sensitive patient data to proprietary commercial servers.”

May combine two claims.

The approach demonstrates a high capability in delivering accurate answers, supported by relevant citations from authoritative sources, which facilitates quick fact-checking as the agent returns the exact original text from the guideline documents.

“We show that our approach demonstrates a high capability in delivering accurate answers, supported by relevant citations from authoritative sources, which also facilitates quick fact-checking as the agent returns the exact original text from the guideline documents.”

Probably not a substantial claim: may not pass S2 (checkable).

Our study provides evidence that agent-based generalist medical AI is within reach.

“Herein, our study could serve as a blueprint, providing evidence that agent-based generalist medical AI is within reach.”

Probably not a substantial claim: may not pass S3 (load-bearing).

No statistical methods were used to predetermine sample sizes.

“No statistical methods were used to predetermine sample sizes.”

Classified as probably not a substantial claim.

All evaluations were performed independently by four certified clinicians with expertise in oncology.

“All evaluations described here were performed independently by four certified clinicians with expertise in oncology.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Experiments were not randomized, as only one test group (the AI agent) existed in this proof-of-concept study.

“Experiments were not randomized, as only one test group (the AI agent) existed in this proof-of-concept study.”

Classified as probably not a substantial claim.

The work is considered a proof-of-concept study for agentic AI workflows in oncology.

“We, therefore, consider our work as a proof-of-concept study for agentic AI workflows in oncology and encourage future efforts to develop larger-scale benchmark cases.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The patient scenarios were restricted to oncological use cases only.

“we restricted our patient scenarios to oncological use cases only”

Probably not a substantial claim: may not pass S3 (load-bearing).

Xiong et al. demonstrated that domain-specific models can enhance retrieval performance.

“Xiong et al.37 demonstrated that domain-specific models can enhance retrieval performance.”

• cites [37] Xiong, G., Jin, Q., Lu, Z. & Zhang, A. Benchmarking retrieval-augmented generation for medicine. Preprint at https://doi.org/10.48550/arXiv.2402.13178 (2024). sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

For rare terms, exact match keyword searches can outperform similarity searches in embedding space and both can be combined (hybrid search).

“for rare terms (such as rare diseases), exact match keyword searches can outperform similarity searches in embedding space and both can even be combined (also termed hybrid search38).”

• cites [38] Bruch, S., Gai, S. & Ingber, A.An analysis of fusion functions for hybrid retrieval. ACM Trans. Inf. Syst. 42, 1–35 (2023).Not checked yet

  Not checked in the free run.

Classified as probably not a substantial claim. May combine two claims.

This study did not include confidential patient information; all research procedures were conducted exclusively on publicly accessible, anonymized patient data and in accordance with the Declaration of Helsinki.

“This study did not include confidential patient information. All research procedures were conducted exclusively on publicly accessible, anonymized patient data and in accordance with the Declaration of Helsinki, maintaining all relevant ethical standards.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

The results are expected to remain reproducible when using other current state-of-the-art models.

“Despite these factors, the results are expected to remain reproducible when using other current state-of-the-art models.”

Classified as probably not a substantial claim.

The RAG framework synergizes the generative capabilities of LLMs with document retrieval to provide domain-specific medical knowledge (context) to a model.

“We leveraged RAG to synergize the generative capabilities of LLMs with document retrieval to provide domain-specific medical knowledge (context) to a model.”

• cites [15] Lewis, P. et al. Retrieval-augmented generation for knowledge-intensive NLP tasks. In NIPS '20: Proc. 34th International Conference on Neural Information Processing Systems 793, 9459–9474 (2020).Not checked yet

  Not checked in the free run.

Classified as probably not a substantial claim. May combine two claims.

Recent datasets by Zakka et al. target open-ended responses and LongHealth focuses on patient-related content.

“Recent advancements have been made with the introduction of new datasets by Zakka et al.16, targeting the enhancement of open-ended responses, and LongHealth27, focusing on patient-related content.”

• cites [16] Zakka, C. et al. Almanac—retrieval-augmented language models for clinical medicine. NEJM AI 1, AIoa2300068 (2024). sourceNot checked yet

  Not checked in the free run.

• cites [27] Adams, L. et al. LongHealth: a question answering benchmark with long clinical documents. Preprint at https://doi.org/10.48550/arXiv.2401.14490 (2024). sourceNot checked yet

  Not checked in the free run.

Classified as probably not a substantial claim.

The reranking step filters out passages that exhibit falsely high similarity (low distance) in embedding space but are contextually irrelevant.

“This step filters out passages that exhibit falsely high similarity (low distance) in embedding space but are contextually irrelevant.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Each case provided a comprehensive but entirely fictional patient profile.

“Each case provided a comprehensive but entirely fictional patient profile, which includes a concise medical history overview encompassing diagnoses, notable medical events and previous treatments.”

Classified as probably not a substantial claim.

GROBID enables the effective parsing of medical documents, preserving text hierarchy and generating essential metadata such as document and journal titles, authorship, pagination, publication dates and download URLs.

“Through its particular training on scientific and technical articles, GROBID enables the effective parsing of medical documents, preserving text hierarchy and generating essential metadata such as document and journal titles, authorship, pagination, publication dates and download URLs.”

• cites [46] Ide, N. & Véronis, J. (eds) Text Encoding Initiative (Springer, 1995). sourceNot checked yet

  Not checked in the free run.