The DOI matches its Crossref record; 63 of the 64 references it cites were found in a registry or an index; none of the 25 sources its claims cite is checked for support yet.
Almanac — Retrieval-Augmented Language Models for Clinical Medicine
Open accessAbstract
BACKGROUND: Large language models (LLMs) have recently shown impressive zero-shot capabilities, whereby they can use auxiliary data, without the availability of task-specific training examples, to complete a variety of natural language tasks, such as summarization, dialogue generation, and question answering. However, despite many promising applications of LLMs in clinical medicine, adoption of these models has been limited by their tendency to generate incorrect and sometimes even harmful statements. METHODS: We tasked a panel of eight board-certified clinicians and two health care practitioners with evaluating Almanac, an LLM framework augmented with retrieval capabilities from curated medical resources for medical guideline and treatment recommendations. The panel compared responses from Almanac and standard LLMs (ChatGPT-4, Bing, and Bard) versus a novel data set of 314 clinical questions spanning nine medical specialties. RESULTS: Almanac showed a significant improvement in performance compared with the standard LLMs across axes of factuality, completeness, user preference, and adversarial safety. CONCLUSIONS: Our results show the potential for LLMs with access to domain-specific corpora to be effective in clinical decision-making. The findings also underscore the importance of carefully testing LLMs before deployment to mitigate their shortcomings. (Funded by the National Institutes of Health, National Heart, Lung, and Blood Institute.).
Overview
AI snapshot
Written from the paper’s full text.
TL;DR
Almanac is a retrieval-augmented LLM framework for open-ended clinical question answering that grounds responses in curated sources and provides citations. In a clinician-graded evaluation on the novel ClinicalQA benchmark, Almanac outperformed ChatGPT-4, Bing, and Bard on factuality, completeness, preference, and adversarial safety.
Key findings
- Across ClinicalQA, Almanac had the best mean ranks (lower is better): factuality 1.96±0.06, completeness 1.85±0.06, and preference 1.87±0.06, with significant differences from counterparts by Nemenyi testing (P<0.01); Bard performed significantly worse.
- Almanac provided correct and trustworthy citations for 91% of ClinicalQA questions, compared with 82% for Bing; ChatGPT-4 citations were mostly nonexistent or unrelated, and Bard either relied on intrinsic knowledge or refused to cite sources.
- In adversarial prompting, Almanac scored 100% correct versus Bard 77%, Bing 70%, and ChatGPT-4 7%. For errors of input, ChatGPT-4 and Almanac caught missing inputs, whereas Bard and Bing refused to perform clinical calculations.
- On the LiveQA test set of 102 questions, Almanac obtained an average score of 2.85, versus the best-performing model reported in the associated publication at 0.637, described as a 347% improvement.
Methods
- Almanac combines a vector database (Qdrant v1.3.0) initialized with more than 500 clinical calculators from MDCalc, a browser limited to PubMed, UpToDate, and BMJ Best Practices, a retriever using OpenAI text-embedding-ada-002 with threshold λ=0.83 and n=10, and gpt-4-0613 with an 8192-token maximum. Articles were chunked into 1000-token passages.
- The evaluation used ClinicalQA, a novel benchmark of 314 open-ended clinical questions generated by a panel of eight board-certified clinicians and two health care practitioners with an average of 10.5 years of experience across nine medical specialties. A subset of 25 questions was used for adversarial prompting, and five clinical calculation vignettes were used for errors of input.
- Clinician panelists independently ranked outputs from Almanac, ChatGPT-4, Bing, and Bard on factuality, completeness, and preference. Statistical significance was assessed with the Friedman test at P<0.01 followed by Nemenyi post hoc pairwise comparisons; citation quality was scored binary, and LiveQA responses were manually scored 0–3.
Limitations
- The paper states that Almanac has limitations in ranking information sources by criteria such as evidence level, study type, and publication date; it uses general-purpose retrievers and language models that could be fine-tuned; ClinicalQA metrics are subjective and rely on human graders, posing scalability challenges; and integration into health care should be cautious because of flaws in generating accurate responses and handling questions lacking straightforward answers.
- The paper also notes that off-the-shelf adversarial resilience becomes less effective as adversarial prompts decrease in word count, requiring careful λ tuning to balance true- and false-positive findings, and that Almanac still struggled with specific resources such as providing a clinical or physician recommendation.
- Evident from the text: complete blinding was not possible because of differing prose styles; alternate retrieval strategies were not explored; adversarial evaluation used only a subset of 25 ClinicalQA questions; and all evaluations were conducted before July 30, 2023, with specific model versions.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
0 of 25 cited sources checked · 0 supported · 0 flagged
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We note that for Almanac, the addition of the adversarial prompt lowered the average score between the query and the retrieved articles below the threshold (λ), resulting in the system abstaining from responding to a given prompt.”
§ Almanac -Retrieval-Augmented Language Models for Clinical Medicine > Results
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesA result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We evaluated our approach on a novel data set of clinical questions and showed that our framework achieves significant improvements in factuality, completeness, preference, and adversarial safety compared with baselines as assessed by a panel of board-certified physicians and health care workers.”
§ Almanac -Retrieval-Augmented Language Models for Clinical Medicine > Discussion
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesA result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Regarding adversarial safety, Almanac's performance greatly superseded that of Bard, Bing, and ChatGPT-4 in adversarial prompting (100% vs. 77, 70, and 7%, respectively).”
§ Almanac -Retrieval-Augmented Language Models for Clinical Medicine > Results
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesA result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“These results were echoed by the Nemenyi P values (P<0.01) in Figure 2 showing significant differences between Almanac and its counterparts.”
§ Almanac -Retrieval-Augmented Language Models for Clinical Medicine > Results
Load-bearingStrength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesThe paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“We note that although Almanac was able to provide more up-to-date answers than the ones provided at times, it still struggled with specific resources, such as providing a clinical or physician recommendation.”
§ Almanac -Retrieval-Augmented Language Models for Clinical Medicine > Results
Strength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passesA result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For citations, Almanac was able to provide correct and trustworthy citations for 91% of the ClinicalQA questions, with missed marks because of an inability to provide reliable sources when relying on its intrinsic knowledge.”
§ Almanac -Retrieval-Augmented Language Models for Clinical Medicine > Results
Load-bearingStrength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passes- Rests on an assumption: The predefined sources used by Almanac are trustworthy.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We evaluated our approach on a novel data set of clinical questions and showed that our framework achieves significant improvements in factuality, completeness, preference, and adversarial safety compared with baselines as assessed by a panel of board-certified physicians and health care workers.”
§ Almanac -Retrieval-Augmented Language Models for Clinical Medicine > Discussion
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesA result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For errors of input, although Bard and Bing refused to perform any clinical calculations, both ChatGPT-4 and Almanac were able to catch missing inputs (Supplementary Appendix).”
§ Almanac -Retrieval-Augmented Language Models for Clinical Medicine > Results
Strength:plain· not yet calibratedCausal:direct· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passesA result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Almanac Outperformed Counterparts across the Scope of ClinicalQA.”
§ Almanac -Retrieval-Augmented Language Models for Clinical Medicine > Discussion
Load-bearingStrength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesA result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For citations, Almanac was able to provide correct and trustworthy citations for 91% of the ClinicalQA questions, with missed marks because of an inability to provide reliable sources when relying on its intrinsic knowledge.”
§ Almanac -Retrieval-Augmented Language Models for Clinical Medicine > Results
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passes
- Probably not a substantial claim. 15
- It may state two separate claims as one. 24
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“They are not without flaws, however, particularly in generating accurate responses and handling questions that lack straightforward answers in their data sources.”
§ Almanac -Retrieval-Augmented Language Models for Clinical Medicine > Discussion
Load-bearingStrength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesProbably not a substantial claim. It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“On LiveQA, Almanac obtained an average score of 2.85 across the 102 questions, well above the best-performing model reported in the article with an average score of 0.637 (a 347% improvement).”
§ Almanac -Retrieval-Augmented Language Models for Clinical Medicine > Results
Load-bearingStrength:boosted· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Similar trends were observed for the weighted mean (with lower ranks penalized more heavily) and inverse-weighted mean (with higher ranks weighed more heavily), with a noted slightly better weighted mean performance by ChatGPT-4 in anesthesia (factuality, 5.23±0.15 vs. 5.60±0.22)and matched performances in cardiology (factuality, 6.20±0.11vs. 6.20±0.14).”
§ Almanac -Retrieval-Augmented Language Models for Clinical Medicine > Results
Strength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passesIt may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“we fed each question into a series of models: Almanac, ChatGPT-4 (May 24, 2023 version), Bing (June 28, 2023), Bard (June 28, 2023), Galactica 120B, RAG, and BioMedLM (formerly known as PubMed GPT).We then retrieved their answers.Because of the very poor performance by the last three models (Supplementary Appendix), only Almanac, ChatGPT-4, Bing, and Bard were used for further evaluation.”
§ Almanac -Retrieval-Augmented Language Models for Clinical Medicine > CLINICALQA EVALUATION
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Although ChatGPT-4 citations were mostly plagued by nonexistent or unrelated web pages, Bard either relied on its intrinsic knowledge or refused to cite sources, despite being prompted to do so.”
§ Almanac -Retrieval-Augmented Language Models for Clinical Medicine > Results
Strength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passesIt may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“We note that although Almanac was able to provide more up-to-date answers than the ones provided at times, it still struggled with specific resources, such as providing a clinical or physician recommendation.”
§ Almanac -Retrieval-Augmented Language Models for Clinical Medicine > Results
Strength:hedged· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Despite slightly worse results in the performance of Bing compared with ChatGPT-4 on the ClinicalQA data set, the Nemenyi heat maps showed no significant differences in their performances.”
§ Almanac -Retrieval-Augmented Language Models for Clinical Medicine > Results
Strength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passesIt may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For errors of input, although Bard and Bing refused to perform any clinical calculations, both ChatGPT-4 and Almanac were able to catch missing inputs (Supplementary Appendix).”
§ Almanac -Retrieval-Augmented Language Models for Clinical Medicine > Results
Strength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passesIt may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“However, despite providing sources for every question on the data set, Bing achieved a performance of 82% because of unreliable sources, including personal blogs and online forums.”
§ Almanac -Retrieval-Augmented Language Models for Clinical Medicine > Results
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Interestingly, Bard was also able to output the correct response along with the adversarial output despite being prompted not to, whereas Bing complied or refused to respond altogether.”
§ Almanac -Retrieval-Augmented Language Models for Clinical Medicine > Results
Strength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passesIt may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Across the full scope of ClinicalQA, Almanac outperformed its counterparts, with a mean rank of 1.96±0.06 in factuality, 1.85±0.06 in completeness, and 1.87±0.06 in preference, with similar overall performances observed within each specialty (Table 3 and Supplementary Appendix).”
§ Almanac -Retrieval-Augmented Language Models for Clinical Medicine > Results
Load-bearingStrength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“As such, using data sets with public-facing evaluations hinders objective experimentation, thereby undermining any subsequently drawn conclusions.”
§ Almanac -Retrieval-Augmented Language Models for Clinical Medicine > DATA SET
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
- Rests on an assumption: The training data for many proprietary models are often kept confidential, introducing an extra layer of uncertainty in estimating data contamination.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“Graders were made aware of the study's goals, but they were not cognizant of the models being tested or familiar with the prose styles adopted by each model.”
§ Almanac -Retrieval-Augmented Language Models for Clinical Medicine > CLINICALQA EVALUATION
Load-bearingStrength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesProbably not a substantial claim. It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Moreover, because these systems leverage the knowledge encoded within their weights to answer clinical queries, their outputs become contingent on the assumption that correct information outweighs misinformation within their training data set.”
§ Almanac -Retrieval-Augmented Language Models for Clinical Medicine > Discussion
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✗ Based: failsProbably not a substantial claim. The paper may give no reason why it holds.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“It is also worth noting that although our ClinicalQA benchmark shows promise, the evaluation metrics are subjective and rely on human graders, posing challenges for scalability.”
§ Almanac -Retrieval-Augmented Language Models for Clinical Medicine > Discussion
Strength:plain· not yet calibratedCausal:direct· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passesIt may state two separate claims as one.
In the paper
“The latter models were prompted with the standard”
§ Almanac -Retrieval-Augmented Language Models for Clinical Medicine > CLINICALQA EVALUATION
What the sources say
- Not matchedcites [56]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
Strength:plain· not yet calibratedCausal:direct· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
- Not matched
None of the sources cited for this claim could be matched to a paper in Crossref, so there is nothing to check it against.
In the paper
“However, despite many promising applications of LLMs in clinical medicine, adoption of these models has been limited by their tendency to generate incorrect and sometimes even harmful statements.”
- Also stated in: § Background · plain
“However, because of the nature of the training objective of predicting the next token in a sentence, large language models (LLMs) can be prone to generating incorrect statements, a phenomenon known as hallucinations.8,9”
§ Almanac -Retrieval-Augmented Language Models for Clinical Medicine > Abstract
What the sources say
- Not matchedcites [8]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not matchedcites [9]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
- Also stated in: § Background · plain
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“The evaluators were explicitly instructed to distinguish between the performance of different models rather than assigning them equivalent scores.”
§ Almanac -Retrieval-Augmented Language Models for Clinical Medicine > CLINICALQA EVALUATION
Strength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“To stay abreast of the shifting landscape of evidence-based practices, physicians often make use of point-of-care tools to improve outcomes.20”
§ Almanac -Retrieval-Augmented Language Models for Clinical Medicine > Background
What the sources say
- Not matchedcites [20]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
Strength:plain· not yet calibratedCausal:direct· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
- Not matched
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“These websites are carefully curated to ensure high-quality content in response to queries.”
§ Almanac -Retrieval-Augmented Language Models for Clinical Medicine > ARCHITECTURE
Strength:plain· not yet calibratedCausal:direct· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✗ Based: failsProbably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
None of the sources cited for this claim could be matched to a paper in Crossref, so there is nothing to check it against.
In the paper
“Moreover, studies have shown that these models may reproduce social biases and make statements that reinforce gender, racial, and religious stereotypes.10,11”
§ Almanac -Retrieval-Augmented Language Models for Clinical Medicine > Background
What the sources say
- Not matchedcites [10]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not matchedcites [11]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
Strength:hedged· not yet calibratedCausal:conditional· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
- Not matched
None of the sources cited for this claim could be matched to a paper in Crossref, so there is nothing to check it against.
In the paper
“In addition to showing poor to weak positive clinical correlation, [43][44][45][46][47][48] U.S. Medical Licensing Examination-style questions fail to encapsulate the full scope of actual clinical scenarios encountered by medical professionals.They often portray patient scenarios as neat clinical vignettes, bypassing the intricate series of microdecisions that constitute real patient care.”
§ Almanac -Retrieval-Augmented Language Models for Clinical Medicine > DATA SET
What the sources say
- Not matchedcites [43]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not matchedcites [44]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not matchedcites [45]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not matchedcites [46]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not matchedcites [47]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not matchedcites [48]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
- Not matched
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“With this framework, large language model (LLM) outputs remain grounded in truth while providing a reliable way of fact-checking.”
§ Almanac -Retrieval-Augmented Language Models for Clinical Medicine > Discussion
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✗ Based: failsProbably not a substantial claim. The paper may give no reason why it holds. It may state two separate claims as one.
- Rests on an assumption: The predefined sources used by Almanac are trustworthy.
None of the sources cited for this claim could be matched to a paper in Crossref, so there is nothing to check it against.
In the paper
“Although existing data sets, such as MultiMedQA, 30 MedMCQA, 40 and PubMedQA, 41 serve as valid benchmarks for assessing reading comprehension and knowledge recall of biomedical language models, their use as benchmarks in open-ended clinical QA tasks poses two clear problems: data contamination and poor clinical proxies.”
§ Almanac -Retrieval-Augmented Language Models for Clinical Medicine > DATA SET
What the sources say
- Not matchedcites [30]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not matchedcites [40]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not matchedcites [41]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
- Not matched
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“By supplementing responses with passages from predefined sources, our grounded system is able to dampen explainability concerns by enabling clinicians to independently verify outputs from trustworthy sources, leading to significant improvements over general retrieval systems.”
§ Almanac -Retrieval-Augmented Language Models for Clinical Medicine > Discussion
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
- Rests on an assumption: The predefined sources used by Almanac are trustworthy.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“Although efforts were made to ensure unbiased grading (e.g., arbitrary answer formatting, answer order shuffling) to blind physicians to the answer's provenance, complete blinding was not possible because of the prose styles adopted by each system.”
§ Almanac -Retrieval-Augmented Language Models for Clinical Medicine > CLINICALQA EVALUATION
Strength:plain· not yet calibratedCausal:direct· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“This threshold was on the basis of an analysis of retriever scores for a sample of 25 clinical questions from an external data set.”
§ Almanac -Retrieval-Augmented Language Models for Clinical Medicine > ARCHITECTURE
Strength:plain· not yet calibratedCausal:direct· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“Despite strong performances on medical question-answer data sets, such as MedQA, 59 these models failed to translate to real-world clinical scenarios because of the potential for harm from unmitigated hallucinations, benchmarks that do not accurately reflect clinically relevant tasks, and concerns of data contamination between train-test splits.”
§ Almanac -Retrieval-Augmented Language Models for Clinical Medicine > Discussion
What the sources say
- Not matchedcites [59]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
- Not matched
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“There was no overlap between graders, as all were asked to evaluate their own submitted questions in accordance with their specialties and fellowships.”
§ Almanac -Retrieval-Augmented Language Models for Clinical Medicine > CLINICALQA EVALUATION
Strength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“Because of the considerable burden and information overload experienced by health care workers in their day-to-day responsibilities, it is not uncommon for medical errors to occur because of human error and poor documentation.43”
§ Almanac -Retrieval-Augmented Language Models for Clinical Medicine > •
What the sources say
- Not matchedcites [43]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
Strength:plain· not yet calibratedCausal:direct· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
- Not matched
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“We noted that each retrieved passage is processed independently of other chunks, and alternate retrieval strategies were not explored.”
§ Almanac -Retrieval-Augmented Language Models for Clinical Medicine > ARCHITECTURE
Strength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
In the paper
“Current LLM evaluation metrics rely on automated methods, such as bilingual evaluation understudy, 55 but they fail to fully capture the complexity and nuances of medical retrieval tasks.”
§ Almanac -Retrieval-Augmented Language Models for Clinical Medicine > CLINICALQA EVALUATION
What the sources say
- Not matchedcites [55]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
Load-bearingStrength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
- Not matched
The paper relies on this without support.
As clinical evidence continues to grow, curated content becomes less accessible and more confined to error-prone search tools and time-consuming appraisal techniques that fail to address the unique needs of individual patients.
“However, as clinical evidence continues to grow, curated content becomes less accessible and more confined to error-prone search tools and time-consuming appraisal techniques that fail to address the unique needs of individual patients.”
The paper relies on this without support.
The training data for many proprietary models are often kept confidential, introducing an extra layer of uncertainty in estimating data contamination.
“More concerning, the training data for many proprietary models are often kept confidential, introducing an extra layer of uncertainty in estimating data contamination.”
The paper relies on this without support.
The predefined sources used by Almanac are trustworthy.
“By supplementing responses with passages from predefined sources, our grounded system is able to dampen explainability concerns by enabling clinicians to independently verify outputs from trustworthy sources”
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Implementations of RAG in the medical field include Zakka et al. (2024), who use a web browser along with an unspecified corpus of medical text and guidelines to pre-retrieve relevant information, which is then passed to a LLM. The system demonstrated an 18% increase in factuality compared to ChatGPT.”
From Grounding Large Language Models in Clinical Evidence: A Retrieval-Augmented Generation System for Querying UK NICE Clinical Guidelines (2025) · cites this paper as Zakka et al. (2024)
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“One study used adversarial prompting, defined as intentionally adding harmful directives to a prompt, to evaluate the safety of the RAG system.28”
The cited source says what the claim attributes to it.
In the citing paper
“Almanac (Zakka et al., 2024) uses RAG frameworks to convert clinical QA tasks into search and retrieval processes, which use LLMs for knowledge distillation from authoritative medical sources to minimize hallucination risks.”
From Trustworthy Medical Question Answering: An Evaluation-Centric Survey (2025) · cites this paper as Zakka et al., 2024
In the citing paper
“retrieval-augmented generation (RAG)15, a process that temporarily enhances an LLM’s knowledge by incorporating relevant text excerpts from authoritative sources into the model, such as medical guidelines16 or textbooks.”
From Development and validation of an autonomous artificial intelligence agent for clinical decision-making in oncology (2025) · cites this paper as [16]
In the citing paper
“Recent advancements have been made with the introduction of new datasets by Zakka et al.16, targeting the enhancement of open-ended responses, and LongHealth27, focusing on patient-related content.”
From Development and validation of an autonomous artificial intelligence agent for clinical decision-making in oncology (2025) · cites this paper as [16]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Below-threshold retrieval leads to both an answer and abstention
MajorThe methods say Almanac answers from intrinsic knowledge when no retrieved articles exceed the threshold. The results say an adversarial prompt lowered retrieval scores below that threshold and caused Almanac to abstain instead.
“If no articles from the database exceeded the match threshold, the language model defaulted to answering using its intrinsic knowledge.”
“the addition of the adversarial prompt lowered the average score between the query and the retrieved articles below the threshold (λ), resulting in the system abstaining from responding to a given prompt.”
LiveQA test-set citation points to a medical-exams dataset
MinorThe methods cite reference 57 for LiveQA, but reference 57 describes a question-answering dataset from medical exams. The LiveQA reference is numbered 55.
“we further evaluated it on the LiveQA 57 test set composed of 102 questions.”
“57. Jin D, Pan E, Oufattole N, Weng W-H, Fang H, Szolovits P. What disease does this patient have? A large-scale open domain question answering dataset from medical exams.”
Step-by-step prompting citation points to a different topic
MinorThe text cites reference 56 for the claim about step-by-step prompting, but that reference concerns language-knowledge graph pretraining.
“The latter models were prompted with the standard "think carefully and step by step" before each question, which has been shown to generally lead to more accurate outputs.56”
“56. Yasunaga M, Bosselut A, Ren H, et al. Deep bidirectional languageknowledge graph pretraining.”
Browser-domain citations point to different sources
MinorThe browser section cites references 52-54 for PubMed, UpToDate, and BMJ Best Practices, respectively. Reference 52 is BMJ Best Practice, while references 53 and 54 are unrelated papers.
“For our experiments, we limited our model's access to three standard domains: PubMed, 52 UpToDate, 53 and BMJ Best Practices.54”
“52. BMJ Publishing Group. BMJ Best Practice. 2023 (https://bestpractice.bmj.com).”
BLEU citation points to a LiveQA paper
MinorThe evaluation section cites reference 55 for bilingual evaluation understudy, but reference 55 is about the LiveQA task. The BLEU paper appears as reference 53.
“Current LLM evaluation metrics rely on automated methods, such as bilingual evaluation understudy, 55”
“55. Ben Abacha A, Agichtein E, Pinter Y, Demner-Fushman D. Overview of the Medical Question Answering Task at TREC 2017 LiveQA.”
Citations
Cites
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
BLEU: a method for automatic evaluation of machine translation
Papineni, K., Roukos, S., Ward, T., & Zhu, W.-J. (2001) · Proceedings of the 40th Annual Meeting on Association for Computational Linguistics - ACL '02 · doi:10.3115/1073083.1073135 · Cited by 21,344
BioBERT: a pre-trained biomedical language representation model for biomedical text mining
Lee, J., Yoon, W., Kim, S., Kim, D., Kim, S., So, C. H., & Kang, J. (2020) · Bioinformatics · doi:10.1093/bioinformatics/btz682 · Cited by 7,218
The National Center for Biotechnology Information
Benson, D. A., Boguski, M. S., Lipman, D. J., & Ostell, J. (1990) · Genomics · doi:10.1016/0888-7543(90)90583-g · Cited by 5,109
Survey of Hallucination in Natural Language Generation
Ji, Z., Lee, N., Frieske, R., Yu, T., Su, D., Xu, Y., Ishii, E., Bang, Y. J., Madotto, A., & Fung, P. (2023) · ACM Computing Surveys · doi:10.1145/3571730 · Cited by 5,025
Training language models to follow instructions with human feedback
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. (2022) · arXiv · doi:10.48550/arxiv.2203.02155 · Cited by 4,208
Large Language Models Encode Clinical Knowledge
Singhal, K., Azizi, S., Tu, T., Mahdavi, S. S., Wei, J., Chung, H. W., Scales, N., Tanwani, A., Cole-Lewis, H., Pfohl, S., et al. (2022) · arXiv · doi:10.48550/arxiv.2212.13138 · Cited by 4,083
SciBERT: A Pretrained Language Model for Scientific Text
Beltagy, I., Lo, K., & Cohan, A. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1371 · Cited by 3,218
Jin, D., Pan, E., Oufattole, N., Weng, W.-H., Fang, H., & Szolovits, P. (2020) · Applied Sciences · doi:10.48550/arxiv.2009.13081 · Cited by 2,339
Flagged first, then by citations
Research works printed but not linked: 3
- Lewis P Perez E Piktus A et al. Retrieval-augmented generation for knowledge-intensive NLP tasks. NIPS’20: Proceedings of the 34th International Conference on Neural Information Processing Systems December 2020 (https://proceedings.neurips.cc/paper/2020/file/6b493230205f780e1bc26945df7481e5-Paper.pdf).
- Wolters Kluwer. UpToDate. 2023 (https://www.uptodate.com).
- BMJ Publishing Group. BMJ Best Practice. 2023 (http://bestpractice.bmj.com).
Cited by
Sorted by citations
Items 201–300 of 430
ChemLit-QA: a human evaluated dataset for chemistry RAG tasks
Wellawatte, G. P., Guo, H., Lederbauer, M., Borisova, A., Hart, M., Brucka, M., & Schwaller, P. (2025) · Machine Learning Science and Technology · doi:10.1088/2632-2153/adc2d6 · Cited by 7
Samaan, J. S., Margolis, S., Srinivasan, N., Srinivasan, A., Yeo, Y. H., Anand, R., Samaan, F. S., Mirocha, J., Safavi‐Naini, S. A. A., Kurdi, B. E., et al. (2024) · medRxiv · doi:10.1101/2024.07.27.24310809 · Cited by 4
Stepping With Caution: Large Language Models for Consulting Infectious Diseases
Ray, P. P. (2024) · Clinical Infectious Diseases · doi:10.1093/cid/ciae442 · Cited by 4
The Black Box Dilemma: Challenges in Human-AI Collaboration in ML-CDSS
Jain, R. K., Srirambhatla, R., Kessler, J., & Goel, R. (2024) · The American Journal of Bioethics · doi:10.1080/15265161.2024.2377105 · Cited by 4
Kunze, K. N. (2024) · Arthroscopy The Journal of Arthroscopic and Related Surgery · doi:10.1016/j.arthro.2024.09.013 · Cited by 4
Embracing Foundation Models for Advancing Scientific Discovery
Guo, S., Shariatmadari, A. H., Xiong, G., & Zhang, A. (2024) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata62323.2024.10825618 · Cited by 5
Wu, K.-C., Chew, F. Y., Cheng, K.-L., Shen, W., Yeh, P.-C., Kao, C. H., Guo, W., & Chang, S. (2025) · Bioengineering · doi:10.3390/bioengineering12070698 · Cited by 3
Zhou, S., & Yu, S. (2025) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-025-03204-3 · Cited by 5
Yap, W., Cheng, S.-C., Lin, C.-H., Hsiao, I., Tsai, T., Yap, W.-L., Chen, W. P.-Y., Lin, C., & Huang, S.-M. (2025) · Bioengineering · doi:10.3390/bioengineering12121345 · Cited by 5
Deng, L., Chen, L., Liu, M., Wang, X., Qi, Y., Shao, C., & Jiang, T. (2025) · Information Processing & Management · doi:10.1016/j.ipm.2025.104435 · Cited by 4
Large language models and conditional rules in clinical decision support systems
Sivasothy, S., Bingham, A., Logothetis, I., Barnett, S., Abdelrazek, M., Lückhoff, C., Mathew, J., Vasa, R., & Mouzakis, K. (2026) · Health Information Science and Systems · doi:10.1007/s13755-026-00428-z · Cited by 6
DxDirector: an agentic large language model driving the full-process clinical diagnosis
Xu, S., Huang, X., Wei, Z., Pang, L., Shen, H., & Cheng, X. (2026) · Nature Communications · doi:10.1038/s41467-026-71928-5 · Cited by 7
MedSynapse-V: Bridging Visual Perception and Clinical Intuition via Latent Memory Evolution
Zhu, C., Wang, Y., Zeng, J., Jiang, J., & Lin, J. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-37550-6_32 · Cited by 4
Safety and diagnostic accuracy of large-language model application of PECARN head injury algorithm
Simper, M. A., Ahmad, F. A., Abraham, J., Maddox, T. M., Flotken, J. A., & Rudloff, J. R. (2026) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2026.106466 · Cited by 3
Nordman, J., Najjuuko, C., Jeschke, N., Snyders, R. E., Guillamet, C. V., Shapiro, J., Leone, C. L., Dethloff, M., Babcock, H. M., Reich, P., et al. (2026) · Clinical Infectious Diseases · doi:10.1093/cid/ciag301 · Cited by 3
Trust-Aware KG-RAG: a framework for trustworthy question answering for engineering equipment
Shi, Q. W., Yang, Y., Di, J., Li, L., Liu, J., Hu, Y., Xiao, J., & Hu, Z. (2026) · Advanced Engineering Informatics · doi:10.1016/j.aei.2026.104801 · Cited by 4
Safety and security of large language models in healthcare
Clusmann, J., Freyer, O., Ostermann, M., Ferber, D., Laleh, N. G., Hilgers, L., Kolbinger, F. R., Schneider, C. V., Downing, A., Wekenborg, M., et al. (2026) · Nature · doi:10.1038/s41586-026-10687-1 · Cited by 5
Alnefaie, S., Atwell, E., & Alsalka, M. A. (2024) · doi:10.1109/icnlp60986.2024.10692797 · Cited by 3
Mora-Delgado, J., Ramos-Ruperto, L., Pardilla, M. J., Sicilia, M. L. M., Rodríguez‐González, A., Sempere, J. M., & Puchades, R. (2025) · Revista Clínica Española · doi:10.1016/j.rce.2025.502413 · Cited by 4
LLM-based kidney disease diagnostic framework for Pathologists
Syeda, M. Z., Bukhari, S. U. K., Hussain, M., Khan, W. A., & Shah, S. S. H. (2024) · Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC) · doi:10.1109/embc53108.2024.10782599 · Cited by 3
Kang, B., Lee, S., Bae, H., & Kim, C.-E. (2024) · 동의생리병리학회지 · doi:10.15188/kjopp.2024.04.38.2.49 · Cited by 3
Coen, E., Fiol, G. D., Kaphingst, K. A., Borsato, E. P., Shannon, J., Smith, H. S., Masino, A. J., & Allen, C. G. (2024) · doi:10.2196/preprints.65848 · Cited by 3
Wang, W., Xu, H., & Zhang, Y. (2025) · Communications in computer and information science · doi:10.1007/978-3-031-94150-4_40 · Cited by 2
Jarchow, H., Bobrowski, C., Falk, S. S. I., Hermann, A., Kulaga, A. Y., Põder, J.-C., Unfried, M., Usanov, N., Zendeh, B., Kennedy, B. K., et al. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-01996-2 · Cited by 4
Implementing a RAG-Based Chatbot in Industrial Operations: A Real-World Case Study
Bahçeci, A. Z., Demirok, F., Kayaoğlu, M., & Topçu, B. (2025) · Innovations in Intelligent Systems and Applications Conference (ASYU) · doi:10.1109/asyu67174.2025.11208259 · Cited by 2
Bai, T., Jiang, K., Yu, Y., Qie, S., Wang, C., Wang, B., & Zhang, W. (2026) · Future Internet · doi:10.3390/fi18020095 · Cited by 2
Medical multi-recall embedding: Adaptive retrieval for diverse evidence in medical RAG systems
Li, C., Jing, F., Li, H., Dong, Z., Xu, Z., Zou, H., Wang, Q., He, J., Qian, Y., Cao, B., et al. (2026) · Information Processing & Management · doi:10.1016/j.ipm.2026.104695 · Cited by 3
Utilizing Metadata for Better Retrieval-Augmented Generation
Yousuf, R. B., Xu, S., Sharma, M., Neeser, A., Latimer, C., & Ramakrishnan, N. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-21289-4_20 · Cited by 6
MedSegAgent: A Universal and Scalable Multi-Agent System for Instructive Medical Image Segmentation
Huang, Z., Wang, H., Ye, J., Ji, Y., Hu, X., Liu, L., Yang, Z., Li, W., Hu, M., Su, Y., et al. (2026) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2026.3677444 · Cited by 2
Med-R 2 : Crafting Trustworthy LLM Physicians via Retrieval and Reasoning of Evidence-Based Medicine
Lu, K., Zheng, L., Pan, D., Zhang, S., Dong, G., Leng, H., Cui, B., wu, Z., & Zhang, W. (2026) · International World Wide Web Conference (WWW) · doi:10.1145/3774904.3792076 · Cited by 21
Lee, J., Shang, T., Baik, J. Y., Duong-Tran, D., Yang, S., Li, L., & Shen, L. (2026) · IEEE International Conference on Healthcare Informatics (ICHI) · doi:10.1007/s41666-026-00241-z · Cited by 16
Tutorial: guidance on the use of large language models for medical research
Jin, Q., Wan, N., Leaman, R., Tian, S., Wang, Z., Yang, Y., Wang, Z., Xiong, G., Lai, P.-T., Zhu, Q. L., et al. (2026) · Nature Protocols · doi:10.1038/s41596-026-01408-z · Cited by 10
Zhao, Y., Miao, Y., Guo, R., Luo, Y., Wang, H., & Wu, Y. X. (2026) · Journal of Medical Internet Research · doi:10.2196/90046 · Cited by 2
Myers, S., Dligach, D., Miller, T. A., Barr, S., Landefeld, J., Gao, Y., Churpek, M. M., Mayampurath, A. M., & Afshar, M. (2026) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocag139 · Cited by 9
Yung, K.-W., Sivaraj, J., Stoyanov, D. V., Loukogeorgakis, S., & Mazomenos, E. B. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-72086-4_55 · Cited by 2
Charting the future of cardiology with large language model artificial intelligence
Wehbe, R. M. (2024) · Nature Reviews Cardiology · doi:10.1038/s41569-024-01105-y · Cited by 2
Large language models for infectious diseases require evidence generation and regulation
Gao, C., Satheakeerthy, S., Guo, C. W., et al. (2025) · Internal Medicine Journal · doi:10.1111/imj.70072 · Cited by 2
Wang, Z. J., Tsang, T. H., Yu, F.-Y., Fang, Y., Chen, S., Sun, F., Mo, P. K. H., & Wong, K.-C. (2025) · Journal of Medical Internet Research · doi:10.2196/76849 · Cited by 2
Reproducible Generative AI Evaluation for Healthcare: A Clinician-in-the-Loop ApproachPreprint
Livingston, L., Featherstone-Uwague, A., Barry, A., Barretto, K., Morey, T., Herrmannová, D., & Avula, V. (2025) · medRxiv · doi:10.1101/2025.03.04.25323131 · Cited by 2
Designing for the Future of Information Access with Generative Information Retrieval
Murdock, V., Lee, C., & Hersh, W. (2024) · The information retrieval series · doi:10.1007/978-3-031-73147-1_9 · Cited by 2
Feasibility of Automated Precharting using GPT-4 in New Specialty Referrals.
Liang, A. S., Banda, J. M., Savage, T., Pandya, A., Carey, R., Megwalu, U. C., Chang, M. T., Dash, D., Corbin, C. K., Sharma, A., et al. (2025) · Europe PMC (PubMed Central) · Cited by 1
Yokoe, T., Roversi, G., Sevivas, N., Kamei, N., Diniz, P., & Pereira, H. (2025) · Journal of Experimental Orthopaedics · doi:10.1002/jeo2.70393 · Cited by 1
Lu, K., Mehdinia, S., Man, K., Wong, C. W., Mao, A., & Eftekhari, Z. (2025) · JCO Clinical Cancer Informatics · doi:10.1200/cci-25-00011 · Cited by 2
Empowering SMEs with SustainWater Bot to advance urban water sustainability
Arslan, M., Munawar, S., & Riaz, Z. (2025) · Sustainable Cities and Society · doi:10.1016/j.scs.2025.106793 · Cited by 2
A Recommendation System for Stress Management at the Workplace Using RAG-Based LLM
Mavrogiannis, P., Panagopoulos, C., Menychtas, A., Gallos, P., & Maglogiannis, I. (2025) · IFIP advances in information and communication technology · doi:10.1007/978-3-032-05681-8_26 · Cited by 1
Momin, E. A., & Mansoor, H. (2025) · Cureus · doi:10.7759/cureus.95083 · Cited by 1
Wang, G., Zhang, Y., Wang, W., Zhu, Y., Lu, W., Wang, C., Bi, H., & Yang, X. (2025) · JMIR AI · doi:10.2196/76372 · Cited by 2
Junqi, B. A. I., Ning, D., You, Y., & Chen, J. (2026) · Buildings · doi:10.3390/buildings16010196 · Cited by 1
Jin, L., Zhao, C., Han, D., Zuo, Z., Liang, Y., Liao, Y., & Tohti, T. (2026) · Knowledge and Information Systems · doi:10.1007/s10115-025-02676-z · Cited by 2
Xuan, T. N., Thanh, V. N., Duy, T. D. N., Hoang, S. T. H., Nguyen, G. B., & Ngoc, T. N. T. (2026) · Lecture notes of the Institute for Computer Sciences, Social Informatics and Telecommunications Engineering · doi:10.1007/978-3-032-14055-5_16 · Cited by 1
TRACE: Topical Reasoning with Adaptive Contextual Experts
Ye, J., Xin, Q., Zhang, C., & Qi, H. (2026) · Big Data and Cognitive Computing · doi:10.3390/bdcc10010031 · Cited by 1
Vision-Language Transformers for Medical Report Synthesis: A Multimodal Approach
Suram, K., J., H. A. T., Kurian, A., Augustine, K., & Rehana Majeed, A. (2025) · International Conference for Advancement in Technology (ICONAT) · doi:10.1109/iconat66879.2025.11362879 · Cited by 1
Toward integrated sleep health: multimodal AI in Hang Hao Meng agent
Cai, M., Liang, S., Zhen, S., Wei, S., Sun, T., Liu, J., Mao, H., & Zhang, J. (2026) · npj Digital Medicine · doi:10.1038/s41746-026-02432-9 · Cited by 1
TAXObot: A Chatbot for the taxonomy of marine organisms
Sahu, N., Baburajan, R., & Saravanane, N. (2026) · Expert Systems with Applications · doi:10.1016/j.eswa.2026.131785 · Cited by 2
Safranek, C. W., Socrates, V., Wright, D., et al. (2026) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-026-03381-9 · Cited by 1
Kunze, K. N., & Barad, A. V. (2026) · Arthroscopy The Journal of Arthroscopic and Related Surgery · doi:10.1002/arj.70094 · Cited by 2
Li, J., Chang, Y., Liu, Y., Wang, Y., Ou, B., Lin, S., Tang, S.-K., & Zheng, L. (2026) · Frontiers in Medicine · doi:10.3389/fmed.2026.1804478 · Cited by 1
Hack, S., Bolis, E., Coccapani, M., D’Angelo, B., Lancieri, A., & Karni, R. J. (2026) · Journal of Emergency Medicine · doi:10.1016/j.jemermed.2026.04.018 · Cited by 1
Evaluation of ChatGPT-5.2 responses to frequently asked questions about benign bone tumors
Ayhan, B., Yoğurt, S. B., & Ayhan, Z. D. (2026) · Frontiers in Oncology · doi:10.3389/fonc.2026.1826346 · Cited by 2
Léon, M., Feng, R., Quiroz-Flores, M., Pelletier, G. J., Bethencourt, D. M., Shibata, M., He, H., & Ruaengsri, C. (2026) · Frontiers in Digital Health · doi:10.3389/fdgth.2026.1769467 · Cited by 1
Mali, Y., Zeng, Z., Heo, K., Zhang, G., Kashyap, A., Chen, J., Keramatian, K., Saraf, G., Solmi, M., Tam, E., et al. (2026) · The Canadian Journal of Psychiatry · doi:10.1177/07067437261457232 · Cited by 2
A Safety-Constrained Architecture for Adaptive Clinical Decision Support
Wendakoon, K., & Tripathi, N. (2026) · Communications in computer and information science · doi:10.1007/978-3-032-28819-6_21 · Cited by 1
Evaluation of ChatGPT-4o’s and DeepSeek R1’s responses to urological problems: a comparative study
Lu, H., Zhang, Y., Wang, Z., Zhao, Y., Liu, J., Qiu, D., & Zhang, Y. (2026) · BMC Urology · doi:10.1186/s12894-026-02228-9 · Cited by 2
A roadmap for medical large language models: a review of foundations, applications, and challenges
Sha, Y.-Y., Yu, L., Lin, Z.-H., Kaur, A., Lou, Y., Gornale, S. S., Zhang, T., Wong, L. S., Wang, Z., Yan, Y., et al. (2026) · Military Medical Research · doi:10.1016/j.mmr.2026.100050 · Cited by 2
RAG in clinical practice: a cautionary tale of AI ‘Truthfulness’
Jung, H., CHO, K., Jun, T. J., & Kim, Y. (2026) · npj Health Systems · doi:10.1038/s44401-026-00115-x · Cited by 3
Ryzhenko, V. P., Бурлака, Б. С., Belenichev, I., Burlaka, K., Aliyeva, O., Bukhtiyarova, N., Halabitska, I., Petakh, P., & Kamyshnyi, O. (2026) · Frontiers in Medicine · doi:10.3389/fmed.2026.1898857 · Cited by 1
Artificial Intelligence in Rhinology
Ayoub, N., & Glicksman, J. T. (2024) · Otolaryngologic Clinics of North America · doi:10.1016/j.otc.2024.04.010 · Cited by 1
Evaluating Transferability in Retrieval Tasks: An Approach Using MMD and Kernel Methods
Dai, M., Raffiee, A. H., Jain, A., & Correa, J. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.02113 · Cited by 2
Wang, Q., Wu, L.-I., & Li, G. (2024) · IEEE International Conference on Medical Artificial Intelligence (MedAI) · doi:10.1109/medai62885.2024.00051 · Cited by 1
What’s the data say? An LLM-based system for interrogating experimental data
Craig, D. B., & Drăghici, S. (2024) · IEEE International Conference on Bioinformatics and Biomedicine (BIBM) · doi:10.1109/bibm62325.2024.10821725 · Cited by 1
Hidayaturrahman, H., & Prawira, I. (2024) · International Conference on Informatics, Multimedia, Cyber and Information System (ICIMCIS) · doi:10.1109/icimcis63449.2024.10956838 · Cited by 1
Instruções de Uso de Medicamentos Suportadas por RAG em Grandes Modelos de Linguagem
dos Reis, D. N., Reis, Z. S. N., & da Rocha, L. C. D. (2025) · Revista Eletrônica de Iniciação Científica em Computação · doi:10.5753/reic.2025.6039 · Cited by 1
Customizing GPT-4 for clinical information retrieval from standard operating proceduresPreprint
Muti, H. S., Loeffler, C. M. L., Leßmann, M.-E., Stueker, E. H., Kirchberg, J., Bonin, M. V., Kolditz, M., Ferber, D., Egger‐Heidrich, K., Merboth, F., et al. (2024) · medRxiv · doi:10.1101/2024.06.24.24309221 · Cited by 2
Tieliwaerdi, X., Abuduweili, A., Saleh, S., Mutabi, E., Rosenberg, M., & Liu, E. (2024) · medRxiv · doi:10.1101/2024.07.10.24310247 · Cited by 1
Ong, J. C. L., Michael, C., Ng, N., Elangovan, K., Tan, N. Y. T., Jin, L., Xie, Q., Ting, D. S. W., Rodríguez-Monguió, R., Bates, D. W., et al. (2024) · medRxiv · doi:10.1101/2024.09.13.24313606 · Cited by 1
Commentary: The Future of Generative Artificial Intelligence in Radiology
Madhuripan, N. (2024) · Journal of Computer Assisted Tomography · doi:10.1097/rct.0000000000001667 · Cited by 1
Zhong, W., Liu, Y., Liu, Y., et al. (2024) · doi:10.2196/preprints.69929 · Cited by 1
Implementing a context-augmented large language model to guide precision cancer medicinePreprint
Jun, H., Tanaka, Y., Johri, S., Carvalho, F. L. F., Jordan, A. C., Labaki, C., Nagy, M. R., O’Meara, T. A., Παππά, Θ., Pimenta, E. M., et al. (2025) · medRxiv · doi:10.1101/2025.05.09.25327312 · Cited by 1
Baldwin, J. D., Dinh, C., Mukerji, A., Sanghavi, N., & Gombar, S. (2025) · medRxiv · doi:10.1101/2025.07.01.25330655 · Cited by 1
Haddadian, K., Chen, S., & Shor, R. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5357219 · Cited by 1
Liu, W., Chen, X., Jiang, Z., Zhao, L., Jiang, K., Tang, R., Li, W., You, M., Zhou, H., Chen, H., et al. (2025) · Research Square · doi:10.21203/rs.3.rs-8147049/v1 · Cited by 2
Mali, Y., Zeng, Z., Heo, K., Zhang, G., Kashyap, A., Chen, J., Keramatian, K., Saraf, G., Solmi, M., Tam, E. M., et al. (2025) · medRxiv · doi:10.64898/2025.11.30.25341311 · Cited by 1
Evidence-Graded Decision Authorization for Safe Clinical AI: A Constrained Reasoning FrameworkPreprint
Lin, C., Lin, J., & Lin, Y.-S. (2026) · medRxiv · doi:10.64898/2026.05.19.26353565 · Cited by 2
Gropper, A. (2026) · NEJM AI · doi:10.1056/aip2600289 · Cited by 1
Liao, Y., Franco, Z., Lizarraga Mazaba, J. E., & Chen, K. (2026) · Proceedings of the 17th ACM International Conference on Bioinformatics, Computational Biology and Health Informatics · doi:10.1145/3807503.3819453 · Cited by 1
Deceptive Grounding: Entity Attribution Failure in Clinical Retrieval-Augmented Generation
Caruzzo, C., Yoo, D., & Kim, T. S. (2026) · arXiv · doi:10.48550/arxiv.2607.09349
Lewis, M., Thio, S., Roberts, A., Siju, C., Mukit, W., Kuruvilla, R., Jiang, Z. J., Möller-Grell, N., Borakati, A., Dobson, R. J., et al. (2025) · arXiv · doi:10.48550/arxiv.2510.02967
Retracted: Augmenting Content Retrieval Through Machine Learning
Pavan, P. S., Sripriya, T., Vikas, B., Shivank, Parmar, Y., & Acharjya, K. (2023) · International Conference on Smart Generation Computing, Communication and Networking (SMART GENCON) · doi:10.1109/smartgencon60755.2023.10441896
RetractedCan Large Language Models Help Healthcare?
Miyamoto, Y. (2024) · Journal of Atherosclerosis and Thrombosis · doi:10.5551/jat.ed273
The effectiveness of Large Language Models with RAG for auto-annotating phenotype descriptionsPreprint
Kainer, D. (2024) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2024.11.24.625102
Deb, B., Fradley, M. G., Cook, S. C., & Barnes, G. D. (2024) · CJC Open · doi:10.1016/j.cjco.2024.11.020
Breast-Crag: A Breast Cancer Large Language Model Leveraging Retrieval-Augmented GenerationPreprint
An, J., Chen, Z., Wang, Q., Liu, J., Sun, Y., Lü, X., & Zheng, H. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.5052341
Reliable Typical Case Diagnosis via Optimized Retrieval-Augmented Generation Techniques
Zhang, K., Wang, B., Yuan, C., Feng, C., & Shi, G. (2025) · Communications in computer and information science · doi:10.1007/978-981-96-4298-4_19
Integrating Large Language Models into Space Radiology: Opportunities and Challenges
Güneş, Y. C., Çamur, E., & Cesur, T. (2025) · Wilderness and Environmental Medicine · doi:10.1177/10806032251337674
Park, H., Rees, M., Kruger, N., Fuse, K., Castro, V. M., Gainer, V. S., Wattanasin, N., Benoit, B., Wagholikar, K. B., & Murphy, S. N. (2025) · medRxiv · doi:10.1101/2025.04.29.25326696 · Cited by 1
Development of a CDH-Specific Conversational Assistant Using RAG
Klotzman, V., Lopes, C. V., Schomberg, J., Armanyous, Y. S., Linden, D., Ma, I., Giron, A., Yu, P. T., Ahmad, H., Goodman, L. F., et al. (2025) · Communications in computer and information science · doi:10.1007/978-3-031-88346-0_3
Staple, B. L., Staple, E. M., Wallace, C., & Staple, B. D. (2025) · Research Square · doi:10.21203/rs.3.rs-6442642/v1
He, M., Song, J., Ren, S., Zhang, Y., Du, J., Feng, J., Wu, R., & Shen, B. (2025) · Research Square · doi:10.21203/rs.3.rs-6601903/v1
Murali, S., Kamath, S. S., & Sugumaran, V. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-97141-9_16
Wu, W., Xu, D., Liu, L., Wang, B., Zhao, Y., Cheng, X., & Li, X. (2025) · Applied Sciences · doi:10.3390/app15169226
Metadata
- Authors
- Zakka, C., Shad, R., Chaurasia, A., Dalal, A. R., Kim, J. L., Moor, M., Fong, R., Phillips, C., Alexander, K., Ashley, E., Boyd, J., Boyd, K., Hirsch, K., Langlotz, C., Lee, R., Melia, J., Nelson, J., Sallam, K., Tullis, S., Vogelsong, M. A., Cunningham, J. P., & Hiesinger, W.
- Affiliations
- Stanford Medicine (US)
- University of Pennsylvania Health System (US)
- Stanford University (US)
- Department of Radiology and Biomedical Informatics, Stanford Medicine, Stanford, CA
- Johns Hopkins University (US)
- Columbia University (US)
- Corresponding author
- Zakka, Cyril
- ORCID
- Zakka, Cyril
- Shad, Rohan
- Chaurasia, Akash
- Dalal, Alex R.
- Kim, Jennifer L.
- Moor, Michael
- Fong, Robyn
- Phillips, Curran
- Alexander, Kevin
- Ashley, Euan
- Boyd, Jack
- Boyd, Kathleen
- Hirsch, Karen
- Langlotz, Curt
- Lee, Rita
- Melia, Joanna
- Nelson, Joanna
- Sallam, Karim
- Tullis, Stacey
- Vogelsong, Melissa Ann
- Cunningham, John Patrick
- Hiesinger, William
- Venue
- NEJM AI
- Publisher
- Massachusetts Medical Society
- Published
- 2024-01-25
- Volume
- 1
- Issue
- 2
- ISSN
- 2836-9386 (ISSN-L)
- PMC
- PMC10857783
- OpenAlex
- W4391221150
- Cited by
- 424
- References
- 26
- Citations by year
- 2026: 163 · 2025: 159 · 2024: 99
- FWCI
- 39.53
- Type
- journal article
- Language
- en
- Open access
- green
- www.ncbi.nlm.nih.gov
- Licence
- other-oa
- Locations
- NEJM AI (publishedVersion)
- PubMed (publishedVersion)
- PubMed Central (acceptedVersion, other-oa)
- Funders
- National Institutes of Health
- National Heart, Lung, and Blood Institute (R01HL157235)
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Keywords
- retrieval-augmented language models, large language models, medical question answering, clinical guidelines, treatment guidelines, factuality, clinical decision support, almanacs
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-10-29, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-29 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
