Improving large language model applications in biomedicine with retrieval-augmented generation: a systematic review, meta-analysis, and clinical development guidelines
Open accessThe DOI matches its Crossref record; 60 of the 60 references it cites were found in a registry or an index; 19 of the 27 sources its claims cite were checked for support.
Abstract
OBJECTIVE: The objectives of this study are to synthesize findings from recent research of retrieval-augmented generation (RAG) and large language models (LLMs) in biomedicine and provide clinical development guidelines to improve effectiveness. MATERIALS AND METHODS: We conducted a systematic literature review and a meta-analysis. The report was created in adherence to the Preferred Reporting Items for Systematic Reviews and Meta-Analyses 2020 analysis. Searches were performed in 3 databases (PubMed, Embase, PsycINFO) using terms related to "retrieval augmented generation" and "large language model," for articles published in 2023 and 2024. We selected studies that compared baseline LLM performance with RAG performance. We developed a random-effect meta-analysis model, using odds ratio as the effect size. RESULTS: Among 335 studies, 20 were included in this literature review. The pooled effect size was 1.35, with a 95% confidence interval of 1.19-1.53, indicating a statistically significant effect (P = .001). We reported clinical tasks, baseline LLMs, retrieval sources and strategies, as well as evaluation methods. DISCUSSION: Building on our literature review, we developed Guidelines for Unified Implementation and Development of Enhanced LLM Applications with RAG in Clinical Settings to inform clinical applications using RAG. CONCLUSION: Overall, RAG implementation showed a 1.35 odds ratio increase in performance compared to baseline LLMs. Future research should focus on (1) system-level enhancement: the combination of RAG and agent, (2) knowledge-level enhancement: deep integration of knowledge into LLM, and (3) integration-level enhancement: integrating RAG systems within electronic health records.
Overview
CoChat Summary
TL;DR
This systematic review and random-effects meta-analysis found that retrieval-augmented generation (RAG) improved biomedical LLM performance versus baseline LLMs, with a pooled effect size of 1.35 (95% CI 1.19-1.53). The authors also proposed GUIDE-RAG clinical development guidelines to improve performance, transparency, and integration of RAG systems in clinical settings.
Key findings
- Among 335 studies identified, 20 were included; all 20 were published in 2024, and 1 was a conference paper.
- RAG improved performance compared with baseline LLMs with a pooled effect size of 1.35 (95% CI 1.19-1.53; P = .001); heterogeneity was low to moderate (I2 = 37%), and the prediction interval was 1.01 to 1.8.
- Subgroup analysis found similar benefit for clinical decision-making (13 studies; OR 1.46, 95% CI 1.16-1.71) and medical question-answering (7 studies; OR 1.32, 95% CI 1.08-1.63), with no statistically significant difference between groups.
- Human evaluation showed a larger effect than automatic evaluation: OR 1.65 (95% CI 1.36-2.03) vs OR 1.20 (95% CI 1.1-1.41), with the difference statistically significant (P < .01).
Methods
- Systematic literature review and meta-analysis conducted according to PRISMA 2020 and Cochrane Handbook guidance.
- Searches were run on December 12, 2024 in PubMed, Embase, and PsycINFO for peer-reviewed English-language papers published in 2023-2024 that compared baseline LLM performance with RAG performance on biomedical questions.
- Meta-analysis used a random-effects model with odds ratio as the effect size; subgroup analyses examined baseline LLM, retrieval strategy, evaluation method, and task type.
Limitations
- The paper states that it was limited to peer-reviewed biomedical database publications, excluded preprints, included only English-language studies, and did not search sources such as IEEE Xplore or Google Scholar.
- It also notes that free-text searching with automatic MeSH/Emtree mapping may introduce variability if underlying algorithms change.
- Evident from the text: the included studies were heterogeneous in RAG architecture, datasets, retrieval sources, and evaluation methods, which is why a random-effects model was used.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures




Verification
Claims the paper makes
11 Valid · 4 Not valid · 6 Not checked · 42 Own results and interpretations
The cited sources that could be checked couldn’t confirm or dispute this claim.
In the paper
“Accuracy was the most frequently reported metric, covering Likert scale ratings, match rates, correct treatment percentages,9 AUC, AUPRC, and F1 scores, as well as text similarity metrics like ROUGE (ROUGE-1, ROUGE-2, ROUGE-L), BLEU, METEOR, and BERTScore,21 which compared LLM-generated responses to expert-provided answers.”
§ Improving large language model applications in biomedicine with retrieval-augmented generation: a systematic review, meta-analysis, and clinical development guidelines > Results > Evaluation
What the sources say
- Could not verifycites [9] Ayala O, Bechard P. Reducing hallucination in structured outputs via Retrieval-Augmented Generation. Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Association for Computational Linguistics; 2024; 228- 38. 10.18653/v1/2024.naacl-industry.19 sourceExperimental — accuracy not yet measured
Section 4.2 (Metrics; page unavailable) describes the paper’s own measures—trigger exact match, bag of steps, hallucination rates, and retrieval recall—but does not compare how frequently evaluation metrics are reported. Searches for the claimed frequency and the named text-similarity metrics found no passage addressing that point. (LLM, conf=0.76)
- Not checked — no full text of the sourcecites [21] Peters JL, Sutton AJ, Jones DR, et al Contour-enhanced meta-analysis funnel plots help distinguish publication bias from other causes of asymmetry. J Clin Epidemiol. 2008; 61: 991- 996. 10.1016/j.jclinepi.2007.11.010 18538991 source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Could not verify
The cited sources that could be checked couldn’t confirm or dispute this claim.
In the paper
“However, these guidelines do not cover RAG applications in clinical settings, emphasizing the need for a more specific guideline.”
§ Improving large language model applications in biomedicine with retrieval-augmented generation: a systematic review, meta-analysis, and clinical development guidelines > Introduction
What the sources say
- Could not verifycites [11] Vasey B, Nagendran M, Campbell B, et al Reporting guideline for the early stage clinical evaluation of decision support systems driven by artificial intelligence: DECIDE-AI. BMJ. 2022; 377: e070904. 10.1136/bmj-2022-070904 35584845 PMC9116198 sourceExperimental — accuracy not yet measured
The Abstract describes a gap in guidance for early clinical evaluation of AI systems, and Table 1 describes DECIDE-AI as a guideline for evaluating AI interventions in live clinical settings. Neither passage establishes whether existing guidelines cover clinical RAG applications. Searches for “RAG” and “retrieval augmented generation” found no passage addressing that question. (LLM, conf=0.75)
- Could not verifycites [12] Mongan J, Moy L, Kahn CE. Checklist for Artificial Intelligence in Medical Imaging (CLAIM): a guide for authors and reviewers. Radiol Artif Intell. 2020; 2: e200029. 10.1148/ryai.2020200029 33937821 PMC8017414 sourceExperimental — accuracy not yet measured
The Abstract (page unavailable) describes CLAIM as a guideline for AI research in medical imaging, including text analysis. Neither that passage nor the searched passages on RAG, large language models, and clinical applications assess whether existing guidelines cover clinical RAG applications; the absence of RAG from this paper does not establish the broader claim. (LLM, conf=0.74)
- Could not verifycites [13] Martindale APL, Ng B, Ngai V, et al Concordance of randomised controlled trials for artificial intelligence interventions with the CONSORT-AI reporting guidelines. Nat Commun. 2024; 15: 6376- 6311. 10.1038/s41467-024-45355-3 39075055 PMC11286847 sourceExperimental — accuracy not yet measured
The Discussion says that medical large language models could be a focus of future CONSORT-AI iterations, but it does not say that existing guidelines exclude RAG applications in clinical settings. Searches for “RAG” and “retrieval augmented generation” found no passage addressing that specific gap or the need for a RAG-specific guideline. (LLM, conf=0.72)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Could not verify
Every source the paper cites for this claim was checked and supports it.
In the paper
“a preliminary study found that GPT-4 Turbo had only a 16% accuracy rate in extracting table data.27”
§ Improving large language model applications in biomedicine with retrieval-augmented generation: a systematic review, meta-analysis, and clinical development guidelines > Discussion
What the sources say
- Supportedcites [27] Kresevic S, Giuffrè M, Ajcevic M, et al Optimization of hepatological clinical guidelines interpretation by large language models: a retrieval augmented generation-based framework. NPJ Digit Med. 2024; 7: 102- 109. 10.1038/s41746-024-01091-y 38654102 PMC11039454 source
The Methods section (Ablation study: customized LLM framework; page not provided) reports preliminary experiments in which the GPT image-conversion process achieved 16.0% accuracy in extracting pertinent table information. The claim paraphrases that finding; the separate 28.0% result concerns accuracy on table-based questions, not extraction of table information. (LLM, conf=0.88)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The sources with a verdict support this claim; the rest weren’t checked yet, have no full text we could read, or their verdicts aren’t rated yet.
In the paper
“Only 3 studies in our review explicitly mentioned extracting text from tables or figures during the pre-retrieval process.25,27,44”
§ Improving large language model applications in biomedicine with retrieval-augmented generation: a systematic review, meta-analysis, and clinical development guidelines > Discussion
What the sources say
- Not checked — no full text of the sourcecites [25] Yazaki M, Maki S, Furuya T, et al Emergency patient triage improvement through a retrieval-augmented generation enhanced large-scale language model. Prehosp Emerg Care. 2024; 1- 7. 10.1080/10903127.2024.2374400 38950135 source
- Supportedcites [27] Kresevic S, Giuffrè M, Ajcevic M, et al Optimization of hepatological clinical guidelines interpretation by large language models: a retrieval augmented generation-based framework. NPJ Digit Med. 2024; 7: 102- 109. 10.1038/s41746-024-01091-y 38654102 PMC11039454 source
The Methods section (Ablation study: customized LLM framework) explicitly says that tables presented as images were manually converted to CSV files, then converted into text-based lists for use as context. The citation identifies this paper as one of the citing review’s three studies; it does not ask this individual paper to establish the review’s count. (LLM, conf=0.91)
- Supportedcites [44] Hewitt KJ, Wiest IC, Carrero ZI, et al Large language models as a diagnostic support tool in neuropathology. J Pathol Clin Res. 2024; 10: e70009. 10.1002/2056-4538.70009 39505569 PMC11540532 source
The Materials and methods section (‘RAG approach’; page unavailable) says the study converted relevant tables to plain text for use by the LLMs. That supports citing it as an example of table-text preparation before retrieval; the count of three studies is the citing review’s own finding, not a claim this paper makes. (LLM, conf=0.88)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Every source the paper cites for this claim was checked and supports it.
In the paper
“Only 1 study in our review, which focused on gastrointestinal radiology diagnosis based on imaging descriptions, compared the cost and response time between LLMs and LLMs with RAG.31”
§ Improving large language model applications in biomedicine with retrieval-augmented generation: a systematic review, meta-analysis, and clinical development guidelines > Discussion
What the sources say
- Supportedcites [31] Rau S, Rau A, Nattenmüller J, et al A retrieval-augmented chatbot based on GPT-4 provides appropriate differential diagnosis in gastrointestinal radiology: a proof of concept study. Eur Radiol Exp. 2024; 8: 60. 10.1186/s41747-024-00457-x 38755410 PMC11098977 source
The Methods describe a gastrointestinal imaging-aware GPT-4 chatbot with knowledge retrieval and a generic GPT-4 comparator, tested on descriptions of radiological findings. The Results > Statistical evaluation reports response time and cost per case for both. The statement that this was the only such study in the citing authors’ review is their own characterization, not a finding attributed to this source. (LLM, conf=0.93)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
19 more claims · 1 with their cited sources checked
Create a free account to see all 63 claims and every source verdict.
- It may state two separate claims as one. 20
- Probably not a substantial claim. 24
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Rated major: it would touch a result — a value a result rests on, reported and recomputed p-values on opposite sides of the significance level, two passages that conflict with no sign which is right, or a cited work retracted before this paper was published.
In the paper
“In the subgroup analysis, 12 studies used simple data retrieval strategies (OR 1.30, 95% CI [1.16, 1.45]), while 5 studies used complex data retrieval strategies (OR 1.30, 95% CI [1.07, 1.24]), with no statistically significant difference observed between the 2 approaches.”
§ Improving large language model applications in biomedicine with retrieval-augmented generation: a systematic review, meta-analysis, and clinical development guidelines > Results > Retrieval strategies
Reported 1.3 · recomputed 0.06
Citations
Cites
Bias in meta-analysis detected by a simple, graphical test
Egger, M., Smith, G. D., Schneider, M., & Minder, C. (1997) · BMJ · doi:10.1136/bmj.315.7109.629 · Cited by 56,722
Quantifying heterogeneity in a meta‐analysis
Higgins, J. P. T., & Thompson, S. G. (2002) · Statistics in Medicine · doi:10.1002/sim.1186 · Cited by 36,763
Shamseer, L., Moher, D., Clarke, M., Ghersi, D., Liberati, A., Petticrew, M., Shekelle, P., Stewart, L. A., & the PRISMA-P Group (2015) · BMJ · doi:10.1136/bmj.g7647 · Cited by 13,120
Large language models in medicine
Thirunavukarasu, A. J., Ting, D. S. J., Elangovan, K., Gutierrez, L., Tan, T. F., & Ting, D. S. W. (2023) · Nature Medicine · doi:10.1038/s41591-023-02448-8 · Cited by 3,944
Ayers, J. W., Poliak, A., Dredze, M., Leas, E. C., Zhu, Z., Kelley, J. B., Faix, D. J., Goodman, A. M., Longhurst, C. A., Hogarth, M., et al. (2023) · JAMA Internal Medicine · doi:10.1001/jamainternmed.2023.1838 · Cited by 2,613
Peters, J. L., Sutton, A. J., Jones, D. R., Abrams, K. R., & Rushton, L. (2008) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2007.11.010 · Cited by 2,053 · cited 2×
“Accuracy was the most frequently reported metric, covering Likert scale ratings, match rates, correct treatment percentages, 9 AUC, AUPRC, and F1 scores, as well as text similarity metrics like ROUGE (ROUGE-1, ROUGE-2, ROUGE-L), BLEU, METEOR, and BERTScore, 21 which compared LLM-generated responses to expert-provided…”
A survey on large language model based autonomous agents
Wang, L., Ma, C., Feng, X., Zhang, Z., Yang, H., Zhang, J., Chen, Z., Tang, J., Chen, X., Lin, Y., et al. (2024) · Frontiers of Computer Science · doi:10.1007/s11704-024-40231-1 · Cited by 1,941
A Survey on Large Language Model based Autonomous Agents
Wang, L., Ma, C., Feng, X., Zhang, Z., Yang, H., Zhang, J., Chen, Z., Tang, J., Chen, X., Lin, Y., et al. (2023) · arXiv · doi:10.48550/arxiv.2308.11432 · Cited by 1,925
Checklist for Artificial Intelligence in Medical Imaging (CLAIM): a guide for authors and reviewers
Mongan, J., Moy, L., & Kahn, C. E. (2020) · Radiology: Artificial Intelligence · doi:10.1148/ryai.2020200029 · Cited by 1,443 · cited 1×
“…and LLMs in healthcare, including DECIDE-AI (Developmental and Exploratory Clinical Investigations of DEcision support systems driven by Artificial Intelligence), 11 CLAIM (Checklist for Artificial Intelligence in Medical Imaging), 12 and CONSORT-AI (Consolidated Standards of Reporting Trials-AI).”
(2010) · Paediatric and Perinatal Epidemiology · doi:10.1111/j.1365-3016.2010.01108.x · Cited by 1,443
Research works printed but not linked: 3
- Lefebvre C, Glanville J, Briscoe S, et al Chapter 4: searching for and selecting studies. Cochrane Handbook for Systematic Reviews of Interventions Version, Vol. 6. Cochrane, 2024. https://training.cochrane.org/handbook/current/chapter-04
- Chapter 3 Effect Sizes | Doing Meta-Analysis in R. Accessed October 13, 2024. https://bookdown.org/MathiasHarrer/Doing_Meta_Analysis_in_R/effects.html
- Xi Z, Chen W, Guo X, et al The rise and potential of large language model based agents: a survey. September 14, 2023, preprint: not peer reviewed.
Cited by
Items 101–179 of 179
Automated Meta-Analysis of Transcranial Magnetic Stimulation Based on GraphRAG: A Methodology StudyPreprint
Zhong, Y., Jin, J., Wang, X., He, W., & Yin, T. (2026) · Research Square · doi:10.21203/rs.3.rs-10200494/v1
Mihara, H., Matsuzaki, J., Okugawa, T., Oshima, T., Asakawa, A., Kamiya, T., Kessoku, T., Suzuki, H., Nakamura, K., Hojo, M., et al. (2026) · Journal of Gastroenterology · doi:10.1007/s00535-026-02467-5
Reich, C., Leverenz, J., Brand, C., Niemeier, L., Branzei, I., Yıldırım, M., Sedaghat‐Hamedani, F., Amr, A., Frey, N., & Meder, B. (2026) · Frontiers in Digital Health · doi:10.3389/fdgth.2026.1847603
Large language models in spine care and research
Galbusera, F., & Cina, A. (2026) · European Spine Journal · doi:10.1007/s00586-026-10204-y
Enhancing RAGs for Rheumatology Triage: Strategies for Optimized Knowledge Retrieval
Buonocore, T. M., Cardinale, E., Sakellariou, G., Bellazzi, R., & Sacchi, L. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-95841-0_13
Guideline-grounded Retrieval-augmented Generation for Ophthalmic Clinical Decision Support
Chen, S., Cui, S., & Cao, Z. (2026) · doi:10.1145/3811238.3811871 · Cited by 1
Hata, N., Oami, T., Kawakami, E., Hanai, A., & Nakada, T. (2026) · JMIR Formative Research · doi:10.2196/81606
Garcia, I., & Herrera, D. (2026) · Journal of Artificial Intelligence for Healthcare Systems · doi:10.68159/k345628701
OpenEvidence errs on the safe side in a structured test of triage recommendations
Jia, E., Omar, M., Barash, Y., Brook, O. R., Ahmed, M., Kruskal, J. B., Gorenshtein, A., & Klang, E. (2026) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2026.106687 · Cited by 3
SynBioGPT2: A dynamic reasoning framework enables high-fidelity design of microbial cell factories
Mao, Z., Du, J., Guan, J., Wang, W., Wang, R., Li, H., Shi, Z., Yuan, Q., Liao, X., & Ma, H. (2026) · BioDesign Research · doi:10.1016/j.bidere.2026.100093
Ma, R., Xu, K., 张鸣明, Li, X., Ma, X., Wang, J., & Li, J. (2026) · Medicine · doi:10.1097/md.0000000000050741
Jun, T. J. (2025) · Korean Journal of Medicine · doi:10.3904/kjm.2025.100.5.219 · Cited by 1
What clinicians should know about how large language models reason
Lyndon, S., & Benrimoh, D. A. (2026) · Nature Reviews Psychology · doi:10.1038/s44159-026-00617-3
Pergantis, P., Konstantinos, G., Bardis, N. G., Skianis, C. A., & Drigas, A. S. (2026) · Multimodal Technologies and Interaction · doi:10.3390/mti10090095
Zhang, K., Jiang, B., Guo, Y., & Liu, X. (2026) · Research Square · doi:10.21203/rs.3.rs-9463160/v1
Mohamed, A. H., Shalaby, K. S., Kaladharan, A., Sen, S., Ataş, H., & Kodamullil, A. T. (2026) · Knowledge-Based Systems · doi:10.64898/2026.02.18.706536
Escudero-Arnanz, Ó., Valero-Méndez, M. E., Sánchez-Ramos, N., & Soguero-Ruíz, C. (2026) · JMIR AI · doi:10.2196/83206
“Prior studies have demonstrated that RAG integration significantly reduces “hallucinations” and enhances reliability in medical reasoning [11].”
Mayer, Y., Demetriou, B., Rasperini, G., Gabay, E., Gašperšič, R., Božić, D., Calatrava, J., Ginesin, O., Zigdon‐Giladi, H., & Faria-Almeida, R. (2026) · Journal Of Clinical Periodontology · doi:10.1111/jcpe.70160
Bogachov, B., Mao, Q., & Zhao, Y. F. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.6878405
Perez, J., Gutierrez, A., & Lopez, C. (2024) · Journal of Health Informatics and Digital Systems · doi:10.68159/u033966980
PCOS-GPT: A RAG Enhanced LLM for Personalized Ultrasound Based PCOS Diagnosis
Venkatesh, M., Chitra, P., Dharini, A., Umayal, C. C., & Kiruthika, B. (2025) · International Conference on Computational Intelligence and Communication Networks (CICN) · doi:10.1109/cicn67655.2025.11368163
Benito, K. K. A., Corpuz, R. C. P., Gonzales, A. M., & Comia, L. V. (2026) · International Conference on Integrated Intelligence and Cognitive Engineering (ICIICE) · doi:10.1109/iciice69672.2026.11565138
Rong, J., Liang, M., Wang, Z., Ye, Z., Luo, J., & Liang, Y. I. X. I. U. (2026) · Digital Health · doi:10.1177/20552076261464997 · Cited by 1
Fraser, A. G. (2026) · European Heart Journal - Digital Health · doi:10.1093/ehjdh/ztag095
Zhu, G., Luo, Z., Xu, X., Zhang, H., Fu, Y., ALMasri, H., Beznosenko, A., Tian, S., Liu, Y., 谢杰斌, et al. (2026) · Research Square · doi:10.21203/rs.3.rs-10011746/v1
Boltaboyeva, A., Baigarayeva, Z., Иманбек, Б., Amangeldy, B., Tasmurzayev, N., Ozhikenov, K., Alimbayeva, Z. N., Alimbayev, C. A., & Karymsakova, N. (2026) · Algorithms · doi:10.3390/a19020099 · Cited by 1
Peng, L., Xie, W., Gao, D., TIan, H., Yang, Z., Xian, N., Li, K., Liu, L., Zhao, Y., & Liu, Y. (2026) · doi:10.2196/preprints.91173
Liu, W., Zhao, J., Zhang, D., Dang, C., Lu, Y., & Du, X. (2026) · Frontiers in Endocrinology · doi:10.3389/fendo.2026.1955223
Agentic Self-RAG: An Ablation Study for Zero-Shot Biomedical Question Answering
Wibawa, R. P., Putri, S. W., Wijaya, A. L., & Hidayaturrahman (2026) · IEEE International Conference on Industry 4.0, Artificial Intelligence, and Communications Technology (IAICT) · doi:10.1109/iaict71158.2026.11621038
Kocbek, P., Frkatović-Hodžić, A., Lalić, D., Hui, V. C. C., Lauc, G., & Štiglic, G. (2025) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata66926.2025.11401037 · Cited by 1
“Biomedical AI represents a key application area for Retrieval-Augmented Generation (RAG), characterized by a vast and rapidly evolving knowledge base where factual accuracy and evidence-based reasoning are paramount [20], [21].”
InfluentialSun, N. Y., Genovese, A., Prabha, S., Gomez-Cabello, C. A., Haider, S. A., Collaço, B. G., Pan, T., Wood, N. G., & Forte, A. J. (2026) · Bioengineering · doi:10.3390/bioengineering13040378
“RAG is often assumed to reduce risks associated with standalone language models by grounding responses in external reference material [46,47], yet our findings show that retrieval does not eliminate failure modes and can vary depending on configuration.”
Cardiology Knowledge Assessment of Retrieval-Augmented Open versus Proprietary Large Language ModelsPreprint
Tarabanis, C., Khurshid, S. S., Karamanou, A., et al. (2025) · medRxiv · doi:10.1101/2025.09.11.25335607
Jiang, X., Wang, C., Shan, G., Wu, Z., Li, P., Yang, Y., Zha, S., Liu, L., & Shi, T. (2026) · Digital Health · doi:10.1177/20552076261434052
“Although long context inputs may introduce the lost in the middle problem, 29 the scale of the coding table in this study remains controllable and well within the processing capacity of modern LLMs.”
Nagarajan, R., Hayakawa, J., Patterson, M., & Martel, S. (2026) · Journal of Medical Systems · doi:10.1007/s10916-026-02371-8
Mayer, Y., Томов, Г., Gabay, E., & Sahar-Helft, S. (2026) · Research Square · doi:10.21203/rs.3.rs-10232539/v1
Corral-Gudino, L., Ramos-Casals, M., Marcos, M., & Rios-Blanco, J. J. (2026) · Medicina Clínica (English Edition) · doi:10.1016/j.medcli.2026.107567
Joseph, J. T., Vishwanath, R., & Sisirkumar, J. V. (2026) · medRxiv · doi:10.64898/2026.09.22.26363693
Hasei, J., Toda, Y., Nishimura, Y., IKEZAKI, T., Matsuoka, H., Takeuchi, K., Nakahara, R., & Ozaki, T. (2026) · Frontiers in Digital Health · doi:10.2196/preprints.109786
Large language models for pneumonia detection in radiology reports via text analysis
Chen, W., Li, X., Qu, Z., Liu, T., Chen, B., Huang, L., He, N., Wang, Z., & Lyu, J. (2026) · PLOS Digital Health · doi:10.1371/journal.pdig.0001104
Mahomed, S. M., Hänscheid, T., Almeida, Y., Grobusch, M. P., & Rebelo, M. J. F. (2025) · Malaria Journal · doi:10.1186/s12936-025-05722-x · Cited by 1
The Economics of Accuracy for Medical Reasoning with Large Language ModelsPreprint
Bhattacharyya, K. (2025) · medRxiv · doi:10.64898/2025.12.22.25342804
Svensson, A., & Lindberg, E. (2025) · Journal of Artificial Intelligence for Healthcare Systems · doi:10.68159/y427182552
Williams, N., Collins, G., & Brooks, E. (2026) · International Journal of Pharmaceutical and Phytopharmacological Research · doi:10.51847/txl4jnck8i
Patil, S., Karabacak, M., Southerby, M., & Margetis, K. D. (2026) · Cureus · doi:10.7759/cureus.115367
Advancing the application and evaluation of large language models in health and biomedicine
Henry, S. B. (2025) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocaf043 · Cited by 1
Grosjean, S., Khashei, I., Presciani, D., Capitanio, L., & Martinelli, L. P. (2026) · Artificial Intelligence in Emergency Medicine · doi:10.1016/j.aiemed.2026.100036
OsteoRAG: Retrieval-augmented generation for evidence-based osteoporosis knowledge management
Xu, X., & Shen, J. (2026) · Science Progress · doi:10.1177/00368504261489884
Large reasoning models as thinking machines for medicine
Zhou, H.-Y., Rodman, A. M., Liu, P., Rajpurkar, P., Hu, Y., Wong, T. Y., & Topol, E. J. (2026) · Nature Biomedical Engineering · doi:10.1038/s41551-026-01701-y · Cited by 1
Monfared, V., & Rawassizadeh, R. (2026) · medRxiv · doi:10.64898/2026.07.22.26358688
Integrating Large Language Models and Knowledge Graphs: A Systematic Review
Essalmani, R., Youb, I., Hamlich, M., & Oussalah, M. (2026) · International Conference on Intelligent Systems and Digital Applications (ISDA) · doi:10.1109/isda70544.2026.11605987
Lang, S., Motov, S., Krueckel, J., Scherer, J. N., Staartjes, V. E., & Wuertz‐Kozak, K. (2026) · JOR Spine · doi:10.1002/jsp2.70203
Sannathimmappa, M. B. (2026) · Bulletin of the National Research Centre/Bulletin of the National Research Center · doi:10.1186/s42269-026-01482-z
Retrieval-Augmented Language Models for Checking Safety Consistency in Pharmaceutical Product Labels
Dupont, C., & Martin, J. (2025) · Pharmacophore · doi:10.51847/fzeuaxeluk
Rao, A., Cholankeril, G. T., Flores, A. G., Sood, G., White, T., Kanwal, F., & El‐Serag, H. B. (2026) · Hepatology Communications · doi:10.1097/hc9.0000000000000895 · Cited by 1
The economics of accuracy for medical reasoning with large language models
Bhattacharyya, K., & Kamabattula, S. (2026) · PLOS Digital Health · doi:10.1371/journal.pdig.0001182 · Cited by 1
“As opposed to a static representation of medical knowledge in model weights that becomes obsolete requiring re-training as clinical guidelines evolve, RAG systems retrieve relevant documents and ground their responses in current or changing evidence and business logic [10–12].”
Ensuring trustworthy AI assisted guideline development for clinical practice
Zhang, Q., & Fu, Y. (2026) · npj Digital Medicine · doi:10.1038/s41746-026-03098-z
Ren, L., Fang, F., Zhang, Y., & Ma, Y. (2026) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2026.106598 · Cited by 1
Dynamic alignment of large language models for evidence-grounded heart failure decision support
Liu, L., Dong, C., Ba, Y., Yan, H., Tang, X., Sun, Y., Chen, H., Shi, B., Yu, Q., & Zhang, S. (2026) · Digital Health · doi:10.1177/20552076261487568 · Cited by 2
Adiniaev, Y., Omar, M., Daniel, O., Timor, T. M., Barash, Y., Brook, O. R., Klang, E., & Gorenshtein, A. (2026) · Neurological Sciences · doi:10.1007/s10072-026-09239-2
Souza, M. L. H., Barcelos, J., da Costa, C. A., Piovesan, L. D., Abe, J. M., & Leite, M. V. (2026) · Advances in sustainability science and technology · doi:10.1007/978-3-032-25267-8_4
Development and performance of a specialized large language model for restorative treatment planning
Shirani, M. (2026) · Journal of Prosthetic Dentistry · doi:10.1016/j.prosdent.2026.08.012
Özmen, B. B., Almeida, V. F. A., Berber, I., Ha, J. Y., Duraes, E. F. R., Isakov, R., Bernard, S. L., Djohan, R. S., & Schwarz, G. S. (2026) · Plastic Surgery · doi:10.1177/22925503261446323
Generation and Automatic Evaluation of Soap Notes from Medical Dialogue Using Large Language Models
Saito, T., Yamanaka, R., Wakabayashi, Y., & Kitaoka, N. (2025) · International Conference on Speech Database and Assessments · doi:10.1109/o-cocosda68185.2025.11385094
“First, regarding the effectiveness of RAG, which Asgari et al. did not validate, one meta-analysis suggests that its use can significantly improve LLM performance in clinical settings [9], making it a viable approach for our research.”
Alkhouli, M. A., & Ellis, S. G. (2025) · JACC: Cardiovascular Interventions · doi:10.1016/j.jcin.2025.08.023
Recent trends of retrieval-augmented generation in healthcare: A systematic review
Landolsi, M. Y., Fadhli, I., Boufaied, A., & Romdhane, L. B. (2026) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2026.115912
Yuliana, M., Aryoyudanta, B., Soekinto, T. W., Setiawan, I. M. A., Choi, Y. K., Dicianno, B. E., Saptono, A., & Parmanto, B. (2026) · doi:10.2196/preprints.104756
A Comparative Performance Study of Retrieval-Augmented Generation Systems in Gynecologic OncologyPreprint
Peng, S., Kuerbanjiang, W., Xiao, H., He, X., & Yi, Y. (2025) · Research Square · doi:10.21203/rs.3.rs-8221196/v1 · Cited by 1
Garcia, I., & Herrera, D. (2026) · Journal of Health Informatics and Digital Systems · doi:10.68159/q638866091
An AI-powered trisomy 21 Research Assistant
Nandi, S., Sundararajan, Z., Subirana-Granés, M., Espinosa, J. M., Pividori, M., Sullivan, K. D., Galbraith, M. D., & Costello, J. C. (2026) · Scientific Reports · doi:10.64898/2026.06.08.730893
On-Premises AI-Tool for Generating Nursing Care Summaries: A Phased-Implementation Study in Japan
Hirata, R., Oda, Y., Morikawa, S., Shigematsu, K., Yamamoto, D., Ito, S., & Tago, M. (2025) · Nursing Research and Reviews · doi:10.2147/nrr.s551576 · Cited by 4
“When validating the nursing summaries created by AI, the total scores for the five evaluation items in Ver.”
Mohammadi, H., Almodaresi, F., Hogue, G. F. J., Wright, A., Orlic-Milacic, M., Li, N. T., Mawani, A., & Stein, L. D. (2025) · Research Square · doi:10.64898/2025.12.12.693752
“Recent advances in artificial intelligence, particularly large language models (LLMs) and retrieval-augmented generation (RAG), offer a promising avenue for improving the accessibility of biological knowledgebases like Reactome (10 – 18).”
Self-citationFarha, R., Rahman, M. M., & Khalifa, F. (2026) · IEEE International Conference on Computing and Machine Intelligence (ICMI) · doi:10.1109/icmi68585.2026.11539807
Lee, E. J., Jung, J. Y., Kim, D. K., Park, J. W., & Kwak, Y. H. (2026) · PLoS ONE · doi:10.1371/journal.pone.0350770
ENAYATI, M., Priya, V., Prochaska, E., Sobe, K., & Wolfien, M. (2026) · Sci · doi:10.3390/sci8090266
Vollmer, A., Schorn, L., Schrader, F., Kübler, N., Sproll, C. K., Vollmer, M., Singh, D. D., & Saravi, B. (2026) · Diagnostics · doi:10.3390/diagnostics16152456 · Cited by 2
Metadata
- Authors
- Liu, S., McCoy, A. B., & Wright, A.
- Affiliations
- Vanderbilt University (US)
- Vanderbilt University Medical Center (US)
- Corresponding author
- Liu, Siru
- Venue
- Journal of the American Medical Informatics Association
- Publisher
- Oxford University Press (OUP)
- Published
- 2025-01-15
- Volume
- 32
- Issue
- 4
- Pages
- 605-615
- ISSN
- 1067-5027 (ISSN-L)
- 1527-974X
- PMC
- PMC12005634
- OpenAlex
- W4406421570
- Cited by
- 200
- References
- 49
- Citations by year
- 2026: 138 · 2025: 60
- FWCI
- 11.70
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Journal of the American Medical Informatics Association (publishedVersion, cc-by-nc)
- PubMed Central (publishedVersion, cc-by-nc)
- PubMed (publishedVersion)
- Funders
- National Institutes of Health (R00LM014097-02, R01LM013995-01)
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
- Keywords
- retrieval-augmented generation, large language models, biomedical applications, medical question answering, electronic health record integration, LLM agents
- MeSH terms
- Large Language Models, Humans, Natural Language Processing, Programming Languages, Information Storage and Retrieval
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.