The DOI matches its Crossref record; 63 of the 64 references it cites were found in a registry or an index; none of the 25 sources its claims cite is checked for support yet.
Almanac — Retrieval-Augmented Language Models for Clinical Medicine
Open accessAbstract
BACKGROUND: Large language models (LLMs) have recently shown impressive zero-shot capabilities, whereby they can use auxiliary data, without the availability of task-specific training examples, to complete a variety of natural language tasks, such as summarization, dialogue generation, and question answering. However, despite many promising applications of LLMs in clinical medicine, adoption of these models has been limited by their tendency to generate incorrect and sometimes even harmful statements. METHODS: We tasked a panel of eight board-certified clinicians and two health care practitioners with evaluating Almanac, an LLM framework augmented with retrieval capabilities from curated medical resources for medical guideline and treatment recommendations. The panel compared responses from Almanac and standard LLMs (ChatGPT-4, Bing, and Bard) versus a novel data set of 314 clinical questions spanning nine medical specialties. RESULTS: Almanac showed a significant improvement in performance compared with the standard LLMs across axes of factuality, completeness, user preference, and adversarial safety. CONCLUSIONS: Our results show the potential for LLMs with access to domain-specific corpora to be effective in clinical decision-making. The findings also underscore the importance of carefully testing LLMs before deployment to mitigate their shortcomings. (Funded by the National Institutes of Health, National Heart, Lung, and Blood Institute.).
Overview
AI snapshot
Written from the paper’s full text.
TL;DR
Almanac is a retrieval-augmented LLM framework for open-ended clinical question answering that grounds responses in curated sources and provides citations. In a clinician-graded evaluation on the novel ClinicalQA benchmark, Almanac outperformed ChatGPT-4, Bing, and Bard on factuality, completeness, preference, and adversarial safety.
Key findings
- Across ClinicalQA, Almanac had the best mean ranks (lower is better): factuality 1.96±0.06, completeness 1.85±0.06, and preference 1.87±0.06, with significant differences from counterparts by Nemenyi testing (P<0.01); Bard performed significantly worse.
- Almanac provided correct and trustworthy citations for 91% of ClinicalQA questions, compared with 82% for Bing; ChatGPT-4 citations were mostly nonexistent or unrelated, and Bard either relied on intrinsic knowledge or refused to cite sources.
- In adversarial prompting, Almanac scored 100% correct versus Bard 77%, Bing 70%, and ChatGPT-4 7%. For errors of input, ChatGPT-4 and Almanac caught missing inputs, whereas Bard and Bing refused to perform clinical calculations.
- On the LiveQA test set of 102 questions, Almanac obtained an average score of 2.85, versus the best-performing model reported in the associated publication at 0.637, described as a 347% improvement.
Methods
- Almanac combines a vector database (Qdrant v1.3.0) initialized with more than 500 clinical calculators from MDCalc, a browser limited to PubMed, UpToDate, and BMJ Best Practices, a retriever using OpenAI text-embedding-ada-002 with threshold λ=0.83 and n=10, and gpt-4-0613 with an 8192-token maximum. Articles were chunked into 1000-token passages.
- The evaluation used ClinicalQA, a novel benchmark of 314 open-ended clinical questions generated by a panel of eight board-certified clinicians and two health care practitioners with an average of 10.5 years of experience across nine medical specialties. A subset of 25 questions was used for adversarial prompting, and five clinical calculation vignettes were used for errors of input.
- Clinician panelists independently ranked outputs from Almanac, ChatGPT-4, Bing, and Bard on factuality, completeness, and preference. Statistical significance was assessed with the Friedman test at P<0.01 followed by Nemenyi post hoc pairwise comparisons; citation quality was scored binary, and LiveQA responses were manually scored 0–3.
Limitations
- The paper states that Almanac has limitations in ranking information sources by criteria such as evidence level, study type, and publication date; it uses general-purpose retrievers and language models that could be fine-tuned; ClinicalQA metrics are subjective and rely on human graders, posing scalability challenges; and integration into health care should be cautious because of flaws in generating accurate responses and handling questions lacking straightforward answers.
- The paper also notes that off-the-shelf adversarial resilience becomes less effective as adversarial prompts decrease in word count, requiring careful λ tuning to balance true- and false-positive findings, and that Almanac still struggled with specific resources such as providing a clinical or physician recommendation.
- Evident from the text: complete blinding was not possible because of differing prose styles; alternate retrieval strategies were not explored; adversarial evaluation used only a subset of 25 ClinicalQA questions; and all evaluations were conducted before July 30, 2023, with specific model versions.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
0 of 25 cited sources checked · 0 supported · 0 flagged
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We note that for Almanac, the addition of the adversarial prompt lowered the average score between the query and the retrieved articles below the threshold (λ), resulting in the system abstaining from responding to a given prompt.”
§ Almanac -Retrieval-Augmented Language Models for Clinical Medicine > Results
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesA result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We evaluated our approach on a novel data set of clinical questions and showed that our framework achieves significant improvements in factuality, completeness, preference, and adversarial safety compared with baselines as assessed by a panel of board-certified physicians and health care workers.”
§ Almanac -Retrieval-Augmented Language Models for Clinical Medicine > Discussion
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesA result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Regarding adversarial safety, Almanac's performance greatly superseded that of Bard, Bing, and ChatGPT-4 in adversarial prompting (100% vs. 77, 70, and 7%, respectively).”
§ Almanac -Retrieval-Augmented Language Models for Clinical Medicine > Results
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesA result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“These results were echoed by the Nemenyi P values (P<0.01) in Figure 2 showing significant differences between Almanac and its counterparts.”
§ Almanac -Retrieval-Augmented Language Models for Clinical Medicine > Results
Load-bearingStrength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesThe paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“We note that although Almanac was able to provide more up-to-date answers than the ones provided at times, it still struggled with specific resources, such as providing a clinical or physician recommendation.”
§ Almanac -Retrieval-Augmented Language Models for Clinical Medicine > Results
Strength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passesA result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For citations, Almanac was able to provide correct and trustworthy citations for 91% of the ClinicalQA questions, with missed marks because of an inability to provide reliable sources when relying on its intrinsic knowledge.”
§ Almanac -Retrieval-Augmented Language Models for Clinical Medicine > Results
Load-bearingStrength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passes- Rests on an assumption: The predefined sources used by Almanac are trustworthy.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We evaluated our approach on a novel data set of clinical questions and showed that our framework achieves significant improvements in factuality, completeness, preference, and adversarial safety compared with baselines as assessed by a panel of board-certified physicians and health care workers.”
§ Almanac -Retrieval-Augmented Language Models for Clinical Medicine > Discussion
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesA result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For errors of input, although Bard and Bing refused to perform any clinical calculations, both ChatGPT-4 and Almanac were able to catch missing inputs (Supplementary Appendix).”
§ Almanac -Retrieval-Augmented Language Models for Clinical Medicine > Results
Strength:plain· not yet calibratedCausal:direct· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passesA result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Almanac Outperformed Counterparts across the Scope of ClinicalQA.”
§ Almanac -Retrieval-Augmented Language Models for Clinical Medicine > Discussion
Load-bearingStrength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesA result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For citations, Almanac was able to provide correct and trustworthy citations for 91% of the ClinicalQA questions, with missed marks because of an inability to provide reliable sources when relying on its intrinsic knowledge.”
§ Almanac -Retrieval-Augmented Language Models for Clinical Medicine > Results
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passes
- Probably not a substantial claim. 15
- It may state two separate claims as one. 24
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“They are not without flaws, however, particularly in generating accurate responses and handling questions that lack straightforward answers in their data sources.”
§ Almanac -Retrieval-Augmented Language Models for Clinical Medicine > Discussion
Load-bearingStrength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesProbably not a substantial claim. It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“On LiveQA, Almanac obtained an average score of 2.85 across the 102 questions, well above the best-performing model reported in the article with an average score of 0.637 (a 347% improvement).”
§ Almanac -Retrieval-Augmented Language Models for Clinical Medicine > Results
Load-bearingStrength:boosted· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Similar trends were observed for the weighted mean (with lower ranks penalized more heavily) and inverse-weighted mean (with higher ranks weighed more heavily), with a noted slightly better weighted mean performance by ChatGPT-4 in anesthesia (factuality, 5.23±0.15 vs. 5.60±0.22)and matched performances in cardiology (factuality, 6.20±0.11vs. 6.20±0.14).”
§ Almanac -Retrieval-Augmented Language Models for Clinical Medicine > Results
Strength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passesIt may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“we fed each question into a series of models: Almanac, ChatGPT-4 (May 24, 2023 version), Bing (June 28, 2023), Bard (June 28, 2023), Galactica 120B, RAG, and BioMedLM (formerly known as PubMed GPT).We then retrieved their answers.Because of the very poor performance by the last three models (Supplementary Appendix), only Almanac, ChatGPT-4, Bing, and Bard were used for further evaluation.”
§ Almanac -Retrieval-Augmented Language Models for Clinical Medicine > CLINICALQA EVALUATION
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Although ChatGPT-4 citations were mostly plagued by nonexistent or unrelated web pages, Bard either relied on its intrinsic knowledge or refused to cite sources, despite being prompted to do so.”
§ Almanac -Retrieval-Augmented Language Models for Clinical Medicine > Results
Strength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passesIt may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“We note that although Almanac was able to provide more up-to-date answers than the ones provided at times, it still struggled with specific resources, such as providing a clinical or physician recommendation.”
§ Almanac -Retrieval-Augmented Language Models for Clinical Medicine > Results
Strength:hedged· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Despite slightly worse results in the performance of Bing compared with ChatGPT-4 on the ClinicalQA data set, the Nemenyi heat maps showed no significant differences in their performances.”
§ Almanac -Retrieval-Augmented Language Models for Clinical Medicine > Results
Strength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passesIt may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For errors of input, although Bard and Bing refused to perform any clinical calculations, both ChatGPT-4 and Almanac were able to catch missing inputs (Supplementary Appendix).”
§ Almanac -Retrieval-Augmented Language Models for Clinical Medicine > Results
Strength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passesIt may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“However, despite providing sources for every question on the data set, Bing achieved a performance of 82% because of unreliable sources, including personal blogs and online forums.”
§ Almanac -Retrieval-Augmented Language Models for Clinical Medicine > Results
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Interestingly, Bard was also able to output the correct response along with the adversarial output despite being prompted not to, whereas Bing complied or refused to respond altogether.”
§ Almanac -Retrieval-Augmented Language Models for Clinical Medicine > Results
Strength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passesIt may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Across the full scope of ClinicalQA, Almanac outperformed its counterparts, with a mean rank of 1.96±0.06 in factuality, 1.85±0.06 in completeness, and 1.87±0.06 in preference, with similar overall performances observed within each specialty (Table 3 and Supplementary Appendix).”
§ Almanac -Retrieval-Augmented Language Models for Clinical Medicine > Results
Load-bearingStrength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“As such, using data sets with public-facing evaluations hinders objective experimentation, thereby undermining any subsequently drawn conclusions.”
§ Almanac -Retrieval-Augmented Language Models for Clinical Medicine > DATA SET
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
- Rests on an assumption: The training data for many proprietary models are often kept confidential, introducing an extra layer of uncertainty in estimating data contamination.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“Graders were made aware of the study's goals, but they were not cognizant of the models being tested or familiar with the prose styles adopted by each model.”
§ Almanac -Retrieval-Augmented Language Models for Clinical Medicine > CLINICALQA EVALUATION
Load-bearingStrength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesProbably not a substantial claim. It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Moreover, because these systems leverage the knowledge encoded within their weights to answer clinical queries, their outputs become contingent on the assumption that correct information outweighs misinformation within their training data set.”
§ Almanac -Retrieval-Augmented Language Models for Clinical Medicine > Discussion
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✗ Based: failsProbably not a substantial claim. The paper may give no reason why it holds.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“It is also worth noting that although our ClinicalQA benchmark shows promise, the evaluation metrics are subjective and rely on human graders, posing challenges for scalability.”
§ Almanac -Retrieval-Augmented Language Models for Clinical Medicine > Discussion
Strength:plain· not yet calibratedCausal:direct· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passesIt may state two separate claims as one.
In the paper
“The latter models were prompted with the standard”
§ Almanac -Retrieval-Augmented Language Models for Clinical Medicine > CLINICALQA EVALUATION
What the sources say
- Not matchedcites [56]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
Strength:plain· not yet calibratedCausal:direct· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
- Not matched
None of the sources cited for this claim could be matched to a paper in Crossref, so there is nothing to check it against.
In the paper
“However, despite many promising applications of LLMs in clinical medicine, adoption of these models has been limited by their tendency to generate incorrect and sometimes even harmful statements.”
- Also stated in: § Background · plain
“However, because of the nature of the training objective of predicting the next token in a sentence, large language models (LLMs) can be prone to generating incorrect statements, a phenomenon known as hallucinations.8,9”
§ Almanac -Retrieval-Augmented Language Models for Clinical Medicine > Abstract
What the sources say
- Not matchedcites [8]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not matchedcites [9]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
- Also stated in: § Background · plain
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“The evaluators were explicitly instructed to distinguish between the performance of different models rather than assigning them equivalent scores.”
§ Almanac -Retrieval-Augmented Language Models for Clinical Medicine > CLINICALQA EVALUATION
Strength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“To stay abreast of the shifting landscape of evidence-based practices, physicians often make use of point-of-care tools to improve outcomes.20”
§ Almanac -Retrieval-Augmented Language Models for Clinical Medicine > Background
What the sources say
- Not matchedcites [20]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
Strength:plain· not yet calibratedCausal:direct· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
- Not matched
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“These websites are carefully curated to ensure high-quality content in response to queries.”
§ Almanac -Retrieval-Augmented Language Models for Clinical Medicine > ARCHITECTURE
Strength:plain· not yet calibratedCausal:direct· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✗ Based: failsProbably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
None of the sources cited for this claim could be matched to a paper in Crossref, so there is nothing to check it against.
In the paper
“Moreover, studies have shown that these models may reproduce social biases and make statements that reinforce gender, racial, and religious stereotypes.10,11”
§ Almanac -Retrieval-Augmented Language Models for Clinical Medicine > Background
What the sources say
- Not matchedcites [10]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not matchedcites [11]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
Strength:hedged· not yet calibratedCausal:conditional· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
- Not matched
None of the sources cited for this claim could be matched to a paper in Crossref, so there is nothing to check it against.
In the paper
“In addition to showing poor to weak positive clinical correlation, [43][44][45][46][47][48] U.S. Medical Licensing Examination-style questions fail to encapsulate the full scope of actual clinical scenarios encountered by medical professionals.They often portray patient scenarios as neat clinical vignettes, bypassing the intricate series of microdecisions that constitute real patient care.”
§ Almanac -Retrieval-Augmented Language Models for Clinical Medicine > DATA SET
What the sources say
- Not matchedcites [43]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not matchedcites [44]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not matchedcites [45]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not matchedcites [46]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not matchedcites [47]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not matchedcites [48]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
- Not matched
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“With this framework, large language model (LLM) outputs remain grounded in truth while providing a reliable way of fact-checking.”
§ Almanac -Retrieval-Augmented Language Models for Clinical Medicine > Discussion
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✗ Based: failsProbably not a substantial claim. The paper may give no reason why it holds. It may state two separate claims as one.
- Rests on an assumption: The predefined sources used by Almanac are trustworthy.
None of the sources cited for this claim could be matched to a paper in Crossref, so there is nothing to check it against.
In the paper
“Although existing data sets, such as MultiMedQA, 30 MedMCQA, 40 and PubMedQA, 41 serve as valid benchmarks for assessing reading comprehension and knowledge recall of biomedical language models, their use as benchmarks in open-ended clinical QA tasks poses two clear problems: data contamination and poor clinical proxies.”
§ Almanac -Retrieval-Augmented Language Models for Clinical Medicine > DATA SET
What the sources say
- Not matchedcites [30]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not matchedcites [40]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not matchedcites [41]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
- Not matched
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“By supplementing responses with passages from predefined sources, our grounded system is able to dampen explainability concerns by enabling clinicians to independently verify outputs from trustworthy sources, leading to significant improvements over general retrieval systems.”
§ Almanac -Retrieval-Augmented Language Models for Clinical Medicine > Discussion
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
- Rests on an assumption: The predefined sources used by Almanac are trustworthy.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“Although efforts were made to ensure unbiased grading (e.g., arbitrary answer formatting, answer order shuffling) to blind physicians to the answer's provenance, complete blinding was not possible because of the prose styles adopted by each system.”
§ Almanac -Retrieval-Augmented Language Models for Clinical Medicine > CLINICALQA EVALUATION
Strength:plain· not yet calibratedCausal:direct· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“This threshold was on the basis of an analysis of retriever scores for a sample of 25 clinical questions from an external data set.”
§ Almanac -Retrieval-Augmented Language Models for Clinical Medicine > ARCHITECTURE
Strength:plain· not yet calibratedCausal:direct· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“Despite strong performances on medical question-answer data sets, such as MedQA, 59 these models failed to translate to real-world clinical scenarios because of the potential for harm from unmitigated hallucinations, benchmarks that do not accurately reflect clinically relevant tasks, and concerns of data contamination between train-test splits.”
§ Almanac -Retrieval-Augmented Language Models for Clinical Medicine > Discussion
What the sources say
- Not matchedcites [59]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
- Not matched
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“There was no overlap between graders, as all were asked to evaluate their own submitted questions in accordance with their specialties and fellowships.”
§ Almanac -Retrieval-Augmented Language Models for Clinical Medicine > CLINICALQA EVALUATION
Strength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“Because of the considerable burden and information overload experienced by health care workers in their day-to-day responsibilities, it is not uncommon for medical errors to occur because of human error and poor documentation.43”
§ Almanac -Retrieval-Augmented Language Models for Clinical Medicine > •
What the sources say
- Not matchedcites [43]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
Strength:plain· not yet calibratedCausal:direct· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
- Not matched
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“We noted that each retrieved passage is processed independently of other chunks, and alternate retrieval strategies were not explored.”
§ Almanac -Retrieval-Augmented Language Models for Clinical Medicine > ARCHITECTURE
Strength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
In the paper
“Current LLM evaluation metrics rely on automated methods, such as bilingual evaluation understudy, 55 but they fail to fully capture the complexity and nuances of medical retrieval tasks.”
§ Almanac -Retrieval-Augmented Language Models for Clinical Medicine > CLINICALQA EVALUATION
What the sources say
- Not matchedcites [55]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
Load-bearingStrength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
- Not matched
The paper relies on this without support.
As clinical evidence continues to grow, curated content becomes less accessible and more confined to error-prone search tools and time-consuming appraisal techniques that fail to address the unique needs of individual patients.
“However, as clinical evidence continues to grow, curated content becomes less accessible and more confined to error-prone search tools and time-consuming appraisal techniques that fail to address the unique needs of individual patients.”
The paper relies on this without support.
The training data for many proprietary models are often kept confidential, introducing an extra layer of uncertainty in estimating data contamination.
“More concerning, the training data for many proprietary models are often kept confidential, introducing an extra layer of uncertainty in estimating data contamination.”
The paper relies on this without support.
The predefined sources used by Almanac are trustworthy.
“By supplementing responses with passages from predefined sources, our grounded system is able to dampen explainability concerns by enabling clinicians to independently verify outputs from trustworthy sources”
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Implementations of RAG in the medical field include Zakka et al. (2024), who use a web browser along with an unspecified corpus of medical text and guidelines to pre-retrieve relevant information, which is then passed to a LLM. The system demonstrated an 18% increase in factuality compared to ChatGPT.”
From Grounding Large Language Models in Clinical Evidence: A Retrieval-Augmented Generation System for Querying UK NICE Clinical Guidelines (2025) · cites this paper as Zakka et al. (2024)
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“One study used adversarial prompting, defined as intentionally adding harmful directives to a prompt, to evaluate the safety of the RAG system.28”
The cited source says what the claim attributes to it.
In the citing paper
“Almanac (Zakka et al., 2024) uses RAG frameworks to convert clinical QA tasks into search and retrieval processes, which use LLMs for knowledge distillation from authoritative medical sources to minimize hallucination risks.”
From Trustworthy Medical Question Answering: An Evaluation-Centric Survey (2025) · cites this paper as Zakka et al., 2024
In the citing paper
“retrieval-augmented generation (RAG)15, a process that temporarily enhances an LLM’s knowledge by incorporating relevant text excerpts from authoritative sources into the model, such as medical guidelines16 or textbooks.”
From Development and validation of an autonomous artificial intelligence agent for clinical decision-making in oncology (2025) · cites this paper as [16]
In the citing paper
“Recent advancements have been made with the introduction of new datasets by Zakka et al.16, targeting the enhancement of open-ended responses, and LongHealth27, focusing on patient-related content.”
From Development and validation of an autonomous artificial intelligence agent for clinical decision-making in oncology (2025) · cites this paper as [16]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Below-threshold retrieval leads to both an answer and abstention
MajorThe methods say Almanac answers from intrinsic knowledge when no retrieved articles exceed the threshold. The results say an adversarial prompt lowered retrieval scores below that threshold and caused Almanac to abstain instead.
“If no articles from the database exceeded the match threshold, the language model defaulted to answering using its intrinsic knowledge.”
“the addition of the adversarial prompt lowered the average score between the query and the retrieved articles below the threshold (λ), resulting in the system abstaining from responding to a given prompt.”
LiveQA test-set citation points to a medical-exams dataset
MinorThe methods cite reference 57 for LiveQA, but reference 57 describes a question-answering dataset from medical exams. The LiveQA reference is numbered 55.
“we further evaluated it on the LiveQA 57 test set composed of 102 questions.”
“57. Jin D, Pan E, Oufattole N, Weng W-H, Fang H, Szolovits P. What disease does this patient have? A large-scale open domain question answering dataset from medical exams.”
Step-by-step prompting citation points to a different topic
MinorThe text cites reference 56 for the claim about step-by-step prompting, but that reference concerns language-knowledge graph pretraining.
“The latter models were prompted with the standard "think carefully and step by step" before each question, which has been shown to generally lead to more accurate outputs.56”
“56. Yasunaga M, Bosselut A, Ren H, et al. Deep bidirectional languageknowledge graph pretraining.”
Browser-domain citations point to different sources
MinorThe browser section cites references 52-54 for PubMed, UpToDate, and BMJ Best Practices, respectively. Reference 52 is BMJ Best Practice, while references 53 and 54 are unrelated papers.
“For our experiments, we limited our model's access to three standard domains: PubMed, 52 UpToDate, 53 and BMJ Best Practices.54”
“52. BMJ Publishing Group. BMJ Best Practice. 2023 (https://bestpractice.bmj.com).”
BLEU citation points to a LiveQA paper
MinorThe evaluation section cites reference 55 for bilingual evaluation understudy, but reference 55 is about the LiveQA task. The BLEU paper appears as reference 53.
“Current LLM evaluation metrics rely on automated methods, such as bilingual evaluation understudy, 55”
“55. Ben Abacha A, Agichtein E, Pinter Y, Demner-Fushman D. Overview of the Medical Question Answering Task at TREC 2017 LiveQA.”
Citations
Cites
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
BLEU: a method for automatic evaluation of machine translation
Papineni, K., Roukos, S., Ward, T., & Zhu, W.-J. (2001) · Proceedings of the 40th Annual Meeting on Association for Computational Linguistics - ACL '02 · doi:10.3115/1073083.1073135 · Cited by 21,344
BioBERT: a pre-trained biomedical language representation model for biomedical text mining
Lee, J., Yoon, W., Kim, S., Kim, D., Kim, S., So, C. H., & Kang, J. (2020) · Bioinformatics · doi:10.1093/bioinformatics/btz682 · Cited by 7,218
The National Center for Biotechnology Information
Benson, D. A., Boguski, M. S., Lipman, D. J., & Ostell, J. (1990) · Genomics · doi:10.1016/0888-7543(90)90583-g · Cited by 5,109
Survey of Hallucination in Natural Language Generation
Ji, Z., Lee, N., Frieske, R., Yu, T., Su, D., Xu, Y., Ishii, E., Bang, Y. J., Madotto, A., & Fung, P. (2023) · ACM Computing Surveys · doi:10.1145/3571730 · Cited by 5,025
Training language models to follow instructions with human feedback
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. (2022) · arXiv · doi:10.48550/arxiv.2203.02155 · Cited by 4,208
Large Language Models Encode Clinical Knowledge
Singhal, K., Azizi, S., Tu, T., Mahdavi, S. S., Wei, J., Chung, H. W., Scales, N., Tanwani, A., Cole-Lewis, H., Pfohl, S., et al. (2022) · arXiv · doi:10.48550/arxiv.2212.13138 · Cited by 4,083
SciBERT: A Pretrained Language Model for Scientific Text
Beltagy, I., Lo, K., & Cohan, A. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1371 · Cited by 3,218
Jin, D., Pan, E., Oufattole, N., Weng, W.-H., Fang, H., & Szolovits, P. (2020) · Applied Sciences · doi:10.48550/arxiv.2009.13081 · Cited by 2,339
Flagged first, then by citations
Research works printed but not linked: 3
- Lewis P Perez E Piktus A et al. Retrieval-augmented generation for knowledge-intensive NLP tasks. NIPS’20: Proceedings of the 34th International Conference on Neural Information Processing Systems December 2020 (https://proceedings.neurips.cc/paper/2020/file/6b493230205f780e1bc26945df7481e5-Paper.pdf).
- Wolters Kluwer. UpToDate. 2023 (https://www.uptodate.com).
- BMJ Publishing Group. BMJ Best Practice. 2023 (http://bestpractice.bmj.com).
Cited by
Sorted by citations
Items 101–200 of 430
A perspective for adapting generalist AI to specialized medical AI applications and their challenges
Wang, Z., Wang, H., Danek, B. P., Li, Y., Mack, C. D., Arbuckle, L., Biswal, D., Poon, H., Wang, Y., Rajpurkar, P., et al. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-01789-7 · Cited by 35
Fabijan, A., Zawadzka-Fabijan, A., Fabijan, R., Zakrzewski, K. M., Nowosławska, E., & Polis, B. (2024) · Journal of Clinical Medicine · doi:10.3390/jcm13144013 · Cited by 17
Accuracy of large language models in answering ophthalmology board-style questions: A meta-analysis
Wu, J.-H., Nishida, T., & Liu, T. Y. A. (2024) · Asia-Pacific Journal of Ophthalmology · doi:10.1016/j.apjo.2024.100106 · Cited by 16
REIS: A High-Performance and Energy-Efficient Retrieval System with In-Storage Processing
Chen, K., Nadig, R., Frouzakis, M., Ghiasi, N. M., Liang, Y., Mao, H., Park, J., Sadrosadati, M., & Mutlu, O. (2025) · Proceedings - International Symposium on Computer Architecture/Proceedings · doi:10.1145/3695053.3731116 · Cited by 20
Sun, Z., Yim, W.-W., Uzuner, Ö., Xia, F., & Yetişgen, M. (2025) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2025.104866 · Cited by 20
Wang, D., Ye, J., Li, J., Liang, J., Zhang, Q., Hu, Q., Pan, C., Wang, D., Liu, Z. Y., Shi, W. H., et al. (2025) · JMIR Medical Education · doi:10.2196/70190 · Cited by 15
Using Large Language Models to Retrieve Critical Data from Clinical Processes and Business Rules
Yu, Y., Gomez-Cabello, C. A., Makarova, S., Parte, Y., Borna, S., Haider, S. A., Genovese, A., Prabha, S., & Forte, A. J. (2024) · Bioengineering · doi:10.3390/bioengineering12010017 · Cited by 18
Retrieval Augmented Generation for Large Language Models in Healthcare: A Systematic ReviewPreprint
Amugongo, L. M., Mascheroni, P., Brooks, S. G., Doering, S., & Seidel, J. (2024) · Preprints.org · doi:10.20944/preprints202407.0876.v1 · Cited by 15
Klang, E., Tessler, I., Apakama, D. U., Abbott, E. E., Glicksberg, B. S., Arnold, M., Moses, A., Sakhuja, A., Soroush, A., Charney, A. W., et al. (2024) · medRxiv · doi:10.1101/2024.10.15.24315526 · Cited by 15
A Retrieval-Augmented Generation Framework for Electric Power Industry Question Answering
Lu, Y., Peng, J., Xu, X., He, Y., Li, T., Wei, J., Jing, H., Wang, H., Xu, B., & Song, H. (2024) · International Conference on Electronics, Computers and Communication Technology (CECCT) · doi:10.1145/3705754.3705771 · Cited by 15
Coen, E., Fiol, G. D., Kaphingst, K. A., Borsato, E. P., Shannon, J., Smith, H. S., Masino, A. J., & Allen, C. G. (2025) · JMIR Cancer · doi:10.2196/65848 · Cited by 17
Li, M., Zhan, Z., Yang, H., Xiao, Y., Zhou, H., Huang, J., & Zhang, R. (2025) · Science Advances · doi:10.1126/sciadv.adr1443 · Cited by 28
Shin, M., Song, J., Kim, M.-G., Yu, H. W., Choe, E. K., & Chai, Y. J. (2025) · Journal of Clinical Medicine · doi:10.3390/jcm14072450 · Cited by 15
Wang, S., Zhao, F., Bu, D., Lu, Y., Gong, M., Liu, H., Yang, Z., Zeng, X., Yuan, Z., Wan, B., et al. (2025) · Nature Communications · doi:10.1038/s41467-025-64142-2 · Cited by 13
Gomez-Cabello, C. A., Prabha, S., Haider, S. A., Genovese, A., Collaço, B. G., Wood, N. G., Bagaria, S. P., & Forte, A. J. (2025) · Bioengineering · doi:10.3390/bioengineering12111194 · Cited by 17
Large language model answers medical questions about standard pathology reports
Wang, A., Zhou, J., Zhang, P., Cao, H., Xin, H., Xu, X., & Zhou, H. (2024) · Frontiers in Medicine · doi:10.3389/fmed.2024.1402457 · Cited by 13
McCaffrey, P., Jackups, R., Seheult, J. N., Zaydman, M. A., Balis, U. J., Thaker, H. M., Rashidi, H. H., & Gullapalli, R. R. (2024) · Archives of Pathology & Laboratory Medicine · doi:10.5858/arpa.2024-0208-ra · Cited by 14
Ramadan, S., Mutsaers, A., Chen, P.-H. C., Bauman, G. S., Velker, V. M., Ahmad, B., Arifin, A. J., Nguyen, T. K., Palma, D., & Goodman, C. D. (2024) · Radiotherapy and Oncology · doi:10.1016/j.radonc.2024.110645 · Cited by 13
Zhang, Y., Ren, S., Wang, J., Lu, J., Wu, C., He, M., Liu, X., Wu, R., Zhao, J., Zhan, C., et al. (2024) · Drugs · doi:10.1007/s40265-024-02124-2 · Cited by 15
Jabal, M. S., Warman, P. I., Zhang, J., Gupta, K., Jain, A., Mazurowski, M. A., Wiggins, W. F., Magudia, K. A., & Calabrese, E. (2025) · Radiology Artificial Intelligence · doi:10.1148/ryai.240551 · Cited by 18
A generalizable deep learning system for cardiac MRI
Shad, R., Zakka, C., Kaur, D., Mathur, M., Fong, R., Cho, J., Filice, R. W., Mongan, J., Kallianos, K., Khandwala, N., et al. (2026) · Nature Biomedical Engineering · doi:10.1038/s41551-026-01637-3 · Cited by 16
Self-citationLarge Language Model Architectures in Health Care: Scoping Review of Research Perspectives
Leiser, F., Guse, R., & Sunyaev, A. (2025) · Journal of Medical Internet Research · doi:10.2196/70315 · Cited by 16
Development and evaluation of a lightweight large language model chatbot for medication enquiry
Elangovan, K., Ong, J. C. L., Jin, L., Seng, J. J. B., Kwan, Y. H., Ng, L. S., Zhong, R. J., Li, J., Ke, Y. H., Liu, N., et al. (2025) · PLOS Digital Health · doi:10.1371/journal.pdig.0000961 · Cited by 18
Aykut, A., & Sezenöz, A. S. (2024) · Ophthalmology and Therapy · doi:10.1007/s40123-024-01014-w · Cited by 12
Liu, J., Koopman, B., Brown, N. J., Chu, K., & Nguyen, A. N. (2024) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2024.103027 · Cited by 19
Pasaribu, J., Yudistira, N., & Mahmudy, W. F. (2024) · IEEE Access · doi:10.1109/access.2024.3518205 · Cited by 19
LLM-Mediated Data Extraction from Patient Records after Radical Prostatectomy
Azar, W. S., Junkin, D., Hesswani, C., Koller, C. R., Parikh, S. H., Schuppe, K., Williams, N., Nethala, D., Mendhiratta, N., Kenigsberg, A. P., et al. (2025) · NEJM AI · doi:10.1056/aics2400943 · Cited by 11
Zhang, C., Yang, H., Liu, X., Wu, R., Zong, H., Wu, E., Zhou, Y. K., Li, J., & Shen, B. (2025) · Journal of Medical Internet Research · doi:10.2196/67201 · Cited by 16
Lechner, F., Kuhn, S., & Knitza, J. (2025) · International Journal of Rheumatic Diseases · doi:10.1111/1756-185x.70124 · Cited by 11
Idan, D., Shitrit, I. B., Volevich, M., Binyamin, Y., Nassar, R., Nassar, M., Abelson, N., Zlotnik, A. A., & Einav, S. (2025) · Scientific Reports · doi:10.1038/s41598-025-21689-w · Cited by 14
Carl, N., Hetz, M. J., Wies, C., Haggenmüller, S., Winterstein, J. T., Mangold, M. H., Maywald, L., Worst, T. S., Westhoff, N., Michel, M. S., et al. (2025) · European Journal of Cancer · doi:10.1016/j.ejca.2025.116168 · Cited by 15
Perron, B. E., Hiltz, B., Khang, E. M., & Savas, S. A. (2025) · Journal of Social Work Education · doi:10.1080/10437797.2024.2411172 · Cited by 16
Kurland, D. B., Alber, D. A., Palla, A., de Souza, D. N., Lau, D., Laufer, I., Frempong-Boadu, A. K., Kondziolka, D. S., & Oermann, E. K. (2025) · Neurosurgery · doi:10.1227/neu.0000000000003354 · Cited by 10
Transforming Physiology and Healthcare through Foundation Models
Godwin, R. C., Tung, A., Berkowitz, D. E., & Melvin, R. L. (2025) · Physiology · doi:10.1152/physiol.00048.2024 · Cited by 11
ENTAgents: AI Agents for Complex Knowledge Otolaryngology
Chan, T. K., & Dinh, N.-D. (2025) · doi:10.1101/2025.01.01.25319863 · Cited by 9
Generative Artificial Intelligence and Musculoskeletal Health Care
Kunze, K. N. (2025) · HSS Journal® The Musculoskeletal Journal of Hospital for Special Surgery · doi:10.1177/15563316251335334 · Cited by 10
Califano, G., Zhang, T., & Spence, C. J. (2025) · International Journal of Human-Computer Interaction · doi:10.1080/10447318.2025.2505154 · Cited by 9
Liu, J., Nguyen, A. N., Daniel Capurro, N., & Verspoor, K. M. (2025) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2025.3574254 · Cited by 11
Chen, Y.-H., Ruan, S.-J., & Chen, P. (2025) · Journal of Medical Internet Research · doi:10.2196/75052 · Cited by 15
A case study of forensic psychiatry experts' reports analysis through large language models
Petroni, G., Alaimo, S., Mandarelli, G., Catanesi, R., Niolu, C., Siracusano, A., & Pulvirenti, A. (2025) · International Journal of Law and Psychiatry · doi:10.1016/j.ijlp.2025.102122 · Cited by 9
CG-RAG: Research Question Answering by Citation Graph Retrieval-Augmented LLMs
Hu, Y., Lei, Z., Dai, Z., Zhang, A., Angirekula, A., Zhang, Z., & Zhao, L. (2025) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3726302.3729920 · Cited by 16
Prabha, S., Gomez-Cabello, C. A., Haider, S. A., Genovese, A., Trabilsy, M., Wood, N. G., Bagaria, S. P., Tao, C., & Forte, A. J. (2025) · Bioengineering · doi:10.3390/bioengineering12080895 · Cited by 13
Vach, M., Gliem, M., Weiss, D., Ivan, V. L., Hauke, F., Boschenriedter, C., Rubbert, C., & Caspers, J. (2025) · Clinical Neuroradiology · doi:10.1007/s00062-025-01562-z · Cited by 9
Masanneck, L., Epping, P. Z., Meuth, S. G., & Pawlitzki, M. (2025) · Journal of Medical Internet Research · doi:10.2196/79379 · Cited by 9
Enhanced LLM-supported instructions for medication use through retrieval-augmented generation
de Jesus, D., Júnior, A. P. D. S., de Albergaria, E. T., Pagano, A. S., de Oliveira, I. J. R., Dias, C. D. S., Lage, E. M., de Oliveira, F. R., Oliveira, J. A., Gomes, I. D. C., et al. (2025) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2025.111135 · Cited by 10
Yang, H., Li, C., & Wang, Q. (2025) · Journal of Loss Prevention in the Process Industries · doi:10.1016/j.jlp.2025.105845 · Cited by 10
Political-RAG: using generative AI to extract political information from media content
Arslan, M., Munawar, S., & Cruz, C. (2024) · Journal of Information Technology & Politics · doi:10.1080/19331681.2024.2417263 · Cited by 9
Knowledge-Enhanced Conversational Agents
Caffaro, F., & Rizzo, G. F. (2024) · Journal of Computer Science and Technology · doi:10.1007/s11390-024-2883-4 · Cited by 10
Nomura, A., Takeji, Y., Shimojima, M., & Takamura, M. (2025) · Circulation Journal · doi:10.1253/circj.cj-24-0865 · Cited by 9
Employing large language models safely and effectively as a practicing neurosurgeon
Patil, A., Serrato, P., Cleaver, G., Limbania, D., See, A. P., & Huang, K. T. (2025) · Acta Neurochirurgica · doi:10.1007/s00701-025-06515-6 · Cited by 8
Thesen, T., Lalchandani, R. R., Blumer, J., & Lee, M. W. (2025) · British Journal of Clinical Pharmacology · doi:10.1002/bcp.70119 · Cited by 8
A Retrieval-Augmented Generation Method for Question Answering on Airworthiness Regulations
Zheng, T., Shen, S., & Zeng, C. (2025) · Electronics · doi:10.3390/electronics14163314 · Cited by 10
RAGMed: A RAG-Based Medical AI Assistant for Improving Healthcare Delivery
Patil, R., Abbidi, M., & Fannon, S. (2025) · AI · doi:10.3390/ai6100240 · Cited by 12
Li, X., He, H., Lu, G., Yue, P., Chen, J., Yang, Z., & Hon, C. (2025) · Chinese Medicine · doi:10.1186/s13020-025-01249-0 · Cited by 10
Giuffrè, M., You, K., Pang, Z., Kresevic, S., Chung, S., Chen, R., Ko, Y., Chan, C., Saarinen, T., Ajčević, M., et al. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-01589-z · Cited by 9
LLM-augmented multi-agent cooperative framework for medical case retrieval in cardiology
Deng, L., Hu, H., Lu, K., & He, P. (2025) · Journal of King Saud University Computer and Information Sciences · doi:10.1007/s44443-025-00311-z · Cited by 8
Li, C., Jia, W.-W., Chu, Y., Menge, F., Speer, T., Reißfelder, C., Hohenberger, P., Jakob, J., & Yang, C. (2025) · Oncology Research and Treatment · doi:10.1159/000544978 · Cited by 7
Kunze, K. N., Gerhold, C., Dave, U., Abunnur, N., Mamonov, A., Nwachukwu, B. U., Verma, N., & Chahla, J. (2025) · Arthroscopy: The Journal of Arthroscopy And Related · doi:10.1016/j.arthro.2025.03.066 · Cited by 6
Stellungnahme „Künstliche Intelligenz in der Medizin“
MWR (2025) · Deutsches Ärzteblatt · doi:10.3238/arztebl.2025.stellungnahme_ki_medizin · Cited by 8
Flathers, M., Dwyer, B., Rozenblit, E., & Torous, J. B. (2025) · Journal of Psychiatric Practice · doi:10.1097/pra.0000000000000892 · Cited by 7
Mathes, S., Ferber, D., Dreyer, T. F., Borm, K. J., Modersohn, L., Willem, T., Dirven, R. J., Vibert, J., Kreutzfeldt, S., Pérez-López, R., et al. (2025) · npj Precision Oncology · doi:10.1038/s41698-025-01180-5 · Cited by 7
Rewthamrongsris, P., Thongchotchat, V., Burapacheep, J., Trachoo, V., Khurshid, Z., & Porntaveetus, T. (2025) · International Dental Journal · doi:10.1016/j.identj.2025.109344 · Cited by 8
Large Language Models in Cardiovascular Prevention: A Narrative Review and Governance Framework
Santos, J. F., & Dores, H. (2026) · Diagnostics · doi:10.3390/diagnostics16030390 · Cited by 8
Physician-Reported Safety Outcomes of AI-Generated Hospital Course Summaries
Grolleau, F., Liang, A. S., Keyes, T., Ma, S. P., Lew, T., Huynh, T. R., Steele, N. Z. R., Chung, P., Qin, P., Chandra, G., et al. (2026) · JAMA Network Open · doi:10.1001/jamanetworkopen.2026.16556 · Cited by 18
Xu, Q., Wu, T., Wang, Y., Li, X., Yu, H., Cen, S., & Li, Z. (2025) · Journal of Pharmaceutical Analysis · doi:10.1016/j.jpha.2025.101406 · Cited by 7
Enhancing Dietary Supplement Question Answer via Retrieval-Augmented Generation (RAG) with LLMPreprint
Hou, Y., & Zhang, R. (2024) · medRxiv · doi:10.1101/2024.09.11.24313513 · Cited by 7
Unlocking electronic health records: a hybrid graph RAG approach to safe clinical AI for patient QA
Thio, S., Lewis, M., Denaxas, S., & Dobson, R. J. B. (2026) · Frontiers in Digital Health · doi:10.3389/fdgth.2026.1780700 · Cited by 6
Assessing retrieval-augmented large language models for medical coding
Klang, E., Tessler, I., Apakama, D. U., Abbott, E., Glicksberg, B. S., Arnold, M., Moses, A., Sakhuja, A., Soroush, A., Charney, A. W., et al. (2025) · NEJM AI · doi:10.1056/aics2401161 · Cited by 6
Comparative Analysis of Performance of Large Language Models in Urogynecology
Yadav, G. S., Pandit, K. M., Connell, P. T., Erfani, H., & Nager, C. W. (2024) · Urogynecology · doi:10.1097/spv.0000000000001545 · Cited by 6
LmRaC: a functionally extensible tool for LLM interrogation of user experimental results
Craig, D. B., & Drăghici, S. (2024) · Bioinformatics · doi:10.1093/bioinformatics/btae679 · Cited by 8
Kainer, D. (2025) · Biology Methods and Protocols · doi:10.1093/biomethods/bpaf016 · Cited by 8
Artificial Intelligence in Medical Education
Proumen, L. A., Uribe‐Marquez, S., Booth, L. G. J., & Mitchell, J. (2025) · Anesthesiology Clinics · doi:10.1016/j.anclin.2025.05.008 · Cited by 6
Clinical pathway-aware large language models for reliable and transparent medical dialogue
Wu, J., Wu, X., Zheng, Y., & Yang, J. (2025) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2025.104942 · Cited by 10
Navarese, E. P., Gorog, D. A., Markides, R. I. L., Koolaji, S., Kereiakes, D. J., Kubica, J., İsgender, M., Lüscher, T. F., & Gorog, D. A. (2026) · JACC Advances · doi:10.1016/j.jacadv.2026.102588 · Cited by 7
BRIDGE: benchmarking large language models for understanding real-world clinical practice texts
Wu, J., Gu, B., Zhou, R., Xie, K., Snyder, D., Jiang, Y., Carducci, V., Wyss, R., Desai, R. J., Alsentzer, E., et al. (2026) · Nature Biomedical Engineering · doi:10.1038/s41551-026-01719-2 · Cited by 28
Generative Artificial Intelligence Tools in Gastroenterology Training
Ahmed, T., Rabinowitz, L. G., Rodman, A. M., & Berzin, T. M. (2024) · Clinical Gastroenterology and Hepatology · doi:10.1016/j.cgh.2024.05.050 · Cited by 6
Malik, S., Frey, L. J., Gutman, J., Mushtaq, A., Warraich, F. Z., & Qureshi, K. (2024) · The American Journal of Gastroenterology · doi:10.14309/ajg.0000000000003255 · Cited by 7
Stenzl, A., Armstrong, A. J., Sboner, A., Ghith, J., Serfass, L., Bland, C. S., Schijvenaars, B. J. A., & Sternberg, C. N. (2024) · European Urology Focus · doi:10.1016/j.euf.2024.05.022 · Cited by 7
Designing Retrieval-Augmented Language Models for Clinical Decision Support
Quigley, K., Koker, T., Taylor, J., Mancuso, V., & Brattain, L. J. (2024) · Studies in computational intelligence · doi:10.1007/978-3-031-63592-2_13 · Cited by 10
Coen, E., Fiol, G. D., Kaphingst, K. A., Borsato, E. P., Shannon, J., Smith, H. S., Masino, A. J., & Allen, C. G. (2024) · Research Square · doi:10.21203/rs.3.rs-4986527/v1 · Cited by 7
Saleh, A. O. M., Tür, G., & Saygın, Y. (2025) · Machine Learning and Knowledge Extraction · doi:10.3390/make7030074 · Cited by 8
Critical Appraisal Tools for Evaluating Artificial Intelligence in Clinical Studies: Scoping Review
Cabello, J. B., García, V. R., Torralba, M., Fernandez, M. M., Ubeda, M., Ansuategi, E., Ramos-Ruperto, L., Emparanza, J. I., Urreta, I., Iglesias, M. T., et al. (2025) · Journal of Medical Internet Research · doi:10.2196/77110 · Cited by 9
A multi-agent approach to neurological clinical reasoning
Sorka, M., Gorenshtein, A., Aran, D., & Shelly, S. (2025) · PLOS Digital Health · doi:10.1371/journal.pdig.0001106 · Cited by 8
Park, J.-H., Kang, H., Jeon, J. H., Kang, S. W., Kim, J.-W., Kim, J.-M., & Lee�, H. (2026) · JMIR Mental Health · doi:10.2196/83352 · Cited by 4
Self-citationPhysicians as context engineers in the era of generative AI
Nenadic, I., Fudim, M., Loring, Z., et al. (2026) · Nature Medicine · doi:10.1038/s41591-026-04215-x · Cited by 5
Prescott, E., Aktaa, S., Abdin, A. M., Asselbergs, F. W., Bruining, N., Chevrier, A., Gale, C. P., Gonçalves, A., Hindricks, G., Irs, A., et al. (2026) · European Heart Journal - Quality of Care and Clinical Outcomes · doi:10.1093/ehjqcco/qcaf161 · Cited by 5
Bailicai: A Domain-Optimized Retrieval-Augmented Generation Framework for Medical Applications
Cui, L., Liu, Y., Ouyang, C., Yu, Y., Zhang, G., Wan, Y., & Yang, F. (2026) · Big Data Mining and Analytics · doi:10.26599/bdma.2024.9020097 · Cited by 22
Li, H., Lin, Y., & Lv, L. (2026) · Frontiers in Public Health · doi:10.3389/fpubh.2026.1805848 · Cited by 5
Redefining Bias Audits for Generative AI in Health Care
Chen, I. Y., & Alsentzer, E. (2025) · NEJM AI · doi:10.1056/aip2500015 · Cited by 5
Sawamura, S., Kohiyama, K., Takenaka, T., Sera, T., Inoue, T., & Nagai, T. (2025) · Cureus · doi:10.7759/cureus.76989 · Cited by 5
MediRAG: Secure Question Answering for Healthcare Data
Jiang, E., Chen, A. P., Tenison, I., & Kagal, L. (2024) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata62323.2024.10825604 · Cited by 6
Bai, B., Meng, X., & Zhao, C. (2025) · Applied Sciences · doi:10.3390/app151910850 · Cited by 6
Patient Version of Guideline for Fibromyalgia (2025 Edition)
Zhang, Z., Zhou, P., He, N., Wu, Z., Bo, S., Chen, C., Chi, S., Hou, N., Hu, J., Jing, F., et al. (2025) · Journal of Evidence-Based Medicine · doi:10.1111/jebm.70094 · Cited by 4
A context-augmented large language model for accurate precision oncology medicine recommendations
Jun, H., Tanaka, Y., Johri, S., Camp, S. Y., Bao, E. L., Carvalho, F. L. F., Gui, D. Y., Jordan, A. C., Labaki, C., Martin, S. D., et al. (2026) · Cancer Cell · doi:10.1016/j.ccell.2025.12.017 · Cited by 8
Hack, S., Craig, J. R., Lin, C.-F., Fu, C., Kwiatkowska, M. A., Kocum, P., Allevi, F., & Saibene, A. M. (2026) · European Archives of Oto-Rhino-Laryngology · doi:10.1007/s00405-026-10040-2 · Cited by 4
Li, P., Patel, A., Vallamchetla, S. K., Heninger, H., Contractor, H., Tao, C., & Cheung, J. (2026) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocag056 · Cited by 4
Wang, X., Ouyang, H., Bhasuran, B., et al. (2025) · IEEE International Conference on Healthcare Informatics (ICHI) · doi:10.1109/ichi64645.2025.00025 · Cited by 4
Park, S., Wee, C. W., Choi, S. H., Kim, K. H., Chang, J. S., Yoon, H. I., Lee, I. J., Kim, Y. B., Cho, J., Keum, K. C., et al. (2025) · Radiotherapy and Oncology · doi:10.1016/j.radonc.2025.111052 · Cited by 5
AutoRAG: Grounding text and symbols
Schulz, T., Luttermann, M., & Möller, R. (2024) · KI - Künstliche Intelligenz · doi:10.1007/s13218-024-00850-z · Cited by 4
da Costa, L. Y., & Filho, J. B. D. O. E. S. (2024) · Simpósio Brasileiro de Tecnologia da Informação e da Linguagem Humana (STIL) · doi:10.5753/stil.2024.245443 · Cited by 4
Metadata
- Authors
- Zakka, C., Shad, R., Chaurasia, A., Dalal, A. R., Kim, J. L., Moor, M., Fong, R., Phillips, C., Alexander, K., Ashley, E., Boyd, J., Boyd, K., Hirsch, K., Langlotz, C., Lee, R., Melia, J., Nelson, J., Sallam, K., Tullis, S., Vogelsong, M. A., Cunningham, J. P., & Hiesinger, W.
- Affiliations
- Stanford Medicine (US)
- University of Pennsylvania Health System (US)
- Stanford University (US)
- Department of Radiology and Biomedical Informatics, Stanford Medicine, Stanford, CA
- Johns Hopkins University (US)
- Columbia University (US)
- Corresponding author
- Zakka, Cyril
- ORCID
- Zakka, Cyril
- Shad, Rohan
- Chaurasia, Akash
- Dalal, Alex R.
- Kim, Jennifer L.
- Moor, Michael
- Fong, Robyn
- Phillips, Curran
- Alexander, Kevin
- Ashley, Euan
- Boyd, Jack
- Boyd, Kathleen
- Hirsch, Karen
- Langlotz, Curt
- Lee, Rita
- Melia, Joanna
- Nelson, Joanna
- Sallam, Karim
- Tullis, Stacey
- Vogelsong, Melissa Ann
- Cunningham, John Patrick
- Hiesinger, William
- Venue
- NEJM AI
- Publisher
- Massachusetts Medical Society
- Published
- 2024-01-25
- Volume
- 1
- Issue
- 2
- ISSN
- 2836-9386 (ISSN-L)
- PMC
- PMC10857783
- OpenAlex
- W4391221150
- Cited by
- 424
- References
- 26
- Citations by year
- 2026: 163 · 2025: 159 · 2024: 99
- FWCI
- 39.53
- Type
- journal article
- Language
- en
- Open access
- green
- www.ncbi.nlm.nih.gov
- Licence
- other-oa
- Locations
- NEJM AI (publishedVersion)
- PubMed (publishedVersion)
- PubMed Central (acceptedVersion, other-oa)
- Funders
- National Institutes of Health
- National Heart, Lung, and Blood Institute (R01HL157235)
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Keywords
- retrieval-augmented language models, large language models, medical question answering, clinical guidelines, treatment guidelines, factuality, clinical decision support, almanacs
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-10-29, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-29 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
