Improving large language model applications in biomedicine with retrieval-augmented generation: a systematic review, meta-analysis, and clinical development guidelines
Open accessThe DOI matches its Crossref record; 60 of the 60 references it cites were found in a registry or an index; 19 of the 27 sources its claims cite were checked for support.
Abstract
OBJECTIVE: The objectives of this study are to synthesize findings from recent research of retrieval-augmented generation (RAG) and large language models (LLMs) in biomedicine and provide clinical development guidelines to improve effectiveness. MATERIALS AND METHODS: We conducted a systematic literature review and a meta-analysis. The report was created in adherence to the Preferred Reporting Items for Systematic Reviews and Meta-Analyses 2020 analysis. Searches were performed in 3 databases (PubMed, Embase, PsycINFO) using terms related to "retrieval augmented generation" and "large language model," for articles published in 2023 and 2024. We selected studies that compared baseline LLM performance with RAG performance. We developed a random-effect meta-analysis model, using odds ratio as the effect size. RESULTS: Among 335 studies, 20 were included in this literature review. The pooled effect size was 1.35, with a 95% confidence interval of 1.19-1.53, indicating a statistically significant effect (P = .001). We reported clinical tasks, baseline LLMs, retrieval sources and strategies, as well as evaluation methods. DISCUSSION: Building on our literature review, we developed Guidelines for Unified Implementation and Development of Enhanced LLM Applications with RAG in Clinical Settings to inform clinical applications using RAG. CONCLUSION: Overall, RAG implementation showed a 1.35 odds ratio increase in performance compared to baseline LLMs. Future research should focus on (1) system-level enhancement: the combination of RAG and agent, (2) knowledge-level enhancement: deep integration of knowledge into LLM, and (3) integration-level enhancement: integrating RAG systems within electronic health records.
Overview
CoChat Summary
TL;DR
This systematic review and random-effects meta-analysis found that retrieval-augmented generation (RAG) improved biomedical LLM performance versus baseline LLMs, with a pooled effect size of 1.35 (95% CI 1.19-1.53). The authors also proposed GUIDE-RAG clinical development guidelines to improve performance, transparency, and integration of RAG systems in clinical settings.
Key findings
- Among 335 studies identified, 20 were included; all 20 were published in 2024, and 1 was a conference paper.
- RAG improved performance compared with baseline LLMs with a pooled effect size of 1.35 (95% CI 1.19-1.53; P = .001); heterogeneity was low to moderate (I2 = 37%), and the prediction interval was 1.01 to 1.8.
- Subgroup analysis found similar benefit for clinical decision-making (13 studies; OR 1.46, 95% CI 1.16-1.71) and medical question-answering (7 studies; OR 1.32, 95% CI 1.08-1.63), with no statistically significant difference between groups.
- Human evaluation showed a larger effect than automatic evaluation: OR 1.65 (95% CI 1.36-2.03) vs OR 1.20 (95% CI 1.1-1.41), with the difference statistically significant (P < .01).
Methods
- Systematic literature review and meta-analysis conducted according to PRISMA 2020 and Cochrane Handbook guidance.
- Searches were run on December 12, 2024 in PubMed, Embase, and PsycINFO for peer-reviewed English-language papers published in 2023-2024 that compared baseline LLM performance with RAG performance on biomedical questions.
- Meta-analysis used a random-effects model with odds ratio as the effect size; subgroup analyses examined baseline LLM, retrieval strategy, evaluation method, and task type.
Limitations
- The paper states that it was limited to peer-reviewed biomedical database publications, excluded preprints, included only English-language studies, and did not search sources such as IEEE Xplore or Google Scholar.
- It also notes that free-text searching with automatic MeSH/Emtree mapping may introduce variability if underlying algorithms change.
- Evident from the text: the included studies were heterogeneous in RAG architecture, datasets, retrieval sources, and evaluation methods, which is why a random-effects model was used.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures




Verification
Claims the paper makes
11 Valid · 4 Not valid · 6 Not checked · 42 Own results and interpretations
The cited sources that could be checked couldn’t confirm or dispute this claim.
In the paper
“Accuracy was the most frequently reported metric, covering Likert scale ratings, match rates, correct treatment percentages,9 AUC, AUPRC, and F1 scores, as well as text similarity metrics like ROUGE (ROUGE-1, ROUGE-2, ROUGE-L), BLEU, METEOR, and BERTScore,21 which compared LLM-generated responses to expert-provided answers.”
§ Improving large language model applications in biomedicine with retrieval-augmented generation: a systematic review, meta-analysis, and clinical development guidelines > Results > Evaluation
What the sources say
- Could not verifycites [9] Ayala O, Bechard P. Reducing hallucination in structured outputs via Retrieval-Augmented Generation. Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Association for Computational Linguistics; 2024; 228- 38. 10.18653/v1/2024.naacl-industry.19 sourceExperimental — accuracy not yet measured
Section 4.2 (Metrics; page unavailable) describes the paper’s own measures—trigger exact match, bag of steps, hallucination rates, and retrieval recall—but does not compare how frequently evaluation metrics are reported. Searches for the claimed frequency and the named text-similarity metrics found no passage addressing that point. (LLM, conf=0.76)
- Not checked — no full text of the sourcecites [21] Peters JL, Sutton AJ, Jones DR, et al Contour-enhanced meta-analysis funnel plots help distinguish publication bias from other causes of asymmetry. J Clin Epidemiol. 2008; 61: 991- 996. 10.1016/j.jclinepi.2007.11.010 18538991 source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Could not verify
The cited sources that could be checked couldn’t confirm or dispute this claim.
In the paper
“However, these guidelines do not cover RAG applications in clinical settings, emphasizing the need for a more specific guideline.”
§ Improving large language model applications in biomedicine with retrieval-augmented generation: a systematic review, meta-analysis, and clinical development guidelines > Introduction
What the sources say
- Could not verifycites [11] Vasey B, Nagendran M, Campbell B, et al Reporting guideline for the early stage clinical evaluation of decision support systems driven by artificial intelligence: DECIDE-AI. BMJ. 2022; 377: e070904. 10.1136/bmj-2022-070904 35584845 PMC9116198 sourceExperimental — accuracy not yet measured
The Abstract describes a gap in guidance for early clinical evaluation of AI systems, and Table 1 describes DECIDE-AI as a guideline for evaluating AI interventions in live clinical settings. Neither passage establishes whether existing guidelines cover clinical RAG applications. Searches for “RAG” and “retrieval augmented generation” found no passage addressing that question. (LLM, conf=0.75)
- Could not verifycites [12] Mongan J, Moy L, Kahn CE. Checklist for Artificial Intelligence in Medical Imaging (CLAIM): a guide for authors and reviewers. Radiol Artif Intell. 2020; 2: e200029. 10.1148/ryai.2020200029 33937821 PMC8017414 sourceExperimental — accuracy not yet measured
The Abstract (page unavailable) describes CLAIM as a guideline for AI research in medical imaging, including text analysis. Neither that passage nor the searched passages on RAG, large language models, and clinical applications assess whether existing guidelines cover clinical RAG applications; the absence of RAG from this paper does not establish the broader claim. (LLM, conf=0.74)
- Could not verifycites [13] Martindale APL, Ng B, Ngai V, et al Concordance of randomised controlled trials for artificial intelligence interventions with the CONSORT-AI reporting guidelines. Nat Commun. 2024; 15: 6376- 6311. 10.1038/s41467-024-45355-3 39075055 PMC11286847 sourceExperimental — accuracy not yet measured
The Discussion says that medical large language models could be a focus of future CONSORT-AI iterations, but it does not say that existing guidelines exclude RAG applications in clinical settings. Searches for “RAG” and “retrieval augmented generation” found no passage addressing that specific gap or the need for a RAG-specific guideline. (LLM, conf=0.72)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Could not verify
Every source the paper cites for this claim was checked and supports it.
In the paper
“a preliminary study found that GPT-4 Turbo had only a 16% accuracy rate in extracting table data.27”
§ Improving large language model applications in biomedicine with retrieval-augmented generation: a systematic review, meta-analysis, and clinical development guidelines > Discussion
What the sources say
- Supportedcites [27] Kresevic S, Giuffrè M, Ajcevic M, et al Optimization of hepatological clinical guidelines interpretation by large language models: a retrieval augmented generation-based framework. NPJ Digit Med. 2024; 7: 102- 109. 10.1038/s41746-024-01091-y 38654102 PMC11039454 source
The Methods section (Ablation study: customized LLM framework; page not provided) reports preliminary experiments in which the GPT image-conversion process achieved 16.0% accuracy in extracting pertinent table information. The claim paraphrases that finding; the separate 28.0% result concerns accuracy on table-based questions, not extraction of table information. (LLM, conf=0.88)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The sources with a verdict support this claim; the rest weren’t checked yet, have no full text we could read, or their verdicts aren’t rated yet.
In the paper
“Only 3 studies in our review explicitly mentioned extracting text from tables or figures during the pre-retrieval process.25,27,44”
§ Improving large language model applications in biomedicine with retrieval-augmented generation: a systematic review, meta-analysis, and clinical development guidelines > Discussion
What the sources say
- Not checked — no full text of the sourcecites [25] Yazaki M, Maki S, Furuya T, et al Emergency patient triage improvement through a retrieval-augmented generation enhanced large-scale language model. Prehosp Emerg Care. 2024; 1- 7. 10.1080/10903127.2024.2374400 38950135 source
- Supportedcites [27] Kresevic S, Giuffrè M, Ajcevic M, et al Optimization of hepatological clinical guidelines interpretation by large language models: a retrieval augmented generation-based framework. NPJ Digit Med. 2024; 7: 102- 109. 10.1038/s41746-024-01091-y 38654102 PMC11039454 source
The Methods section (Ablation study: customized LLM framework) explicitly says that tables presented as images were manually converted to CSV files, then converted into text-based lists for use as context. The citation identifies this paper as one of the citing review’s three studies; it does not ask this individual paper to establish the review’s count. (LLM, conf=0.91)
- Supportedcites [44] Hewitt KJ, Wiest IC, Carrero ZI, et al Large language models as a diagnostic support tool in neuropathology. J Pathol Clin Res. 2024; 10: e70009. 10.1002/2056-4538.70009 39505569 PMC11540532 source
The Materials and methods section (‘RAG approach’; page unavailable) says the study converted relevant tables to plain text for use by the LLMs. That supports citing it as an example of table-text preparation before retrieval; the count of three studies is the citing review’s own finding, not a claim this paper makes. (LLM, conf=0.88)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Every source the paper cites for this claim was checked and supports it.
In the paper
“Only 1 study in our review, which focused on gastrointestinal radiology diagnosis based on imaging descriptions, compared the cost and response time between LLMs and LLMs with RAG.31”
§ Improving large language model applications in biomedicine with retrieval-augmented generation: a systematic review, meta-analysis, and clinical development guidelines > Discussion
What the sources say
- Supportedcites [31] Rau S, Rau A, Nattenmüller J, et al A retrieval-augmented chatbot based on GPT-4 provides appropriate differential diagnosis in gastrointestinal radiology: a proof of concept study. Eur Radiol Exp. 2024; 8: 60. 10.1186/s41747-024-00457-x 38755410 PMC11098977 source
The Methods describe a gastrointestinal imaging-aware GPT-4 chatbot with knowledge retrieval and a generic GPT-4 comparator, tested on descriptions of radiological findings. The Results > Statistical evaluation reports response time and cost per case for both. The statement that this was the only such study in the citing authors’ review is their own characterization, not a finding attributed to this source. (LLM, conf=0.93)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
19 more claims · 1 with their cited sources checked
Create a free account to see all 63 claims and every source verdict.
- It may state two separate claims as one. 20
- Probably not a substantial claim. 24
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Rated major: it would touch a result — a value a result rests on, reported and recomputed p-values on opposite sides of the significance level, two passages that conflict with no sign which is right, or a cited work retracted before this paper was published.
In the paper
“In the subgroup analysis, 12 studies used simple data retrieval strategies (OR 1.30, 95% CI [1.16, 1.45]), while 5 studies used complex data retrieval strategies (OR 1.30, 95% CI [1.07, 1.24]), with no statistically significant difference observed between the 2 approaches.”
§ Improving large language model applications in biomedicine with retrieval-augmented generation: a systematic review, meta-analysis, and clinical development guidelines > Results > Retrieval strategies
Reported 1.3 · recomputed 0.06
Citations
Cites
Bias in meta-analysis detected by a simple, graphical test
Egger, M., Smith, G. D., Schneider, M., & Minder, C. (1997) · BMJ · doi:10.1136/bmj.315.7109.629 · Cited by 56,722
Quantifying heterogeneity in a meta‐analysis
Higgins, J. P. T., & Thompson, S. G. (2002) · Statistics in Medicine · doi:10.1002/sim.1186 · Cited by 36,763
Shamseer, L., Moher, D., Clarke, M., Ghersi, D., Liberati, A., Petticrew, M., Shekelle, P., Stewart, L. A., & the PRISMA-P Group (2015) · BMJ · doi:10.1136/bmj.g7647 · Cited by 13,120
Large language models in medicine
Thirunavukarasu, A. J., Ting, D. S. J., Elangovan, K., Gutierrez, L., Tan, T. F., & Ting, D. S. W. (2023) · Nature Medicine · doi:10.1038/s41591-023-02448-8 · Cited by 3,944
Ayers, J. W., Poliak, A., Dredze, M., Leas, E. C., Zhu, Z., Kelley, J. B., Faix, D. J., Goodman, A. M., Longhurst, C. A., Hogarth, M., et al. (2023) · JAMA Internal Medicine · doi:10.1001/jamainternmed.2023.1838 · Cited by 2,613
Peters, J. L., Sutton, A. J., Jones, D. R., Abrams, K. R., & Rushton, L. (2008) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2007.11.010 · Cited by 2,053 · cited 2×
“Accuracy was the most frequently reported metric, covering Likert scale ratings, match rates, correct treatment percentages, 9 AUC, AUPRC, and F1 scores, as well as text similarity metrics like ROUGE (ROUGE-1, ROUGE-2, ROUGE-L), BLEU, METEOR, and BERTScore, 21 which compared LLM-generated responses to expert-provided…”
A survey on large language model based autonomous agents
Wang, L., Ma, C., Feng, X., Zhang, Z., Yang, H., Zhang, J., Chen, Z., Tang, J., Chen, X., Lin, Y., et al. (2024) · Frontiers of Computer Science · doi:10.1007/s11704-024-40231-1 · Cited by 1,941
A Survey on Large Language Model based Autonomous Agents
Wang, L., Ma, C., Feng, X., Zhang, Z., Yang, H., Zhang, J., Chen, Z., Tang, J., Chen, X., Lin, Y., et al. (2023) · arXiv · doi:10.48550/arxiv.2308.11432 · Cited by 1,925
Checklist for Artificial Intelligence in Medical Imaging (CLAIM): a guide for authors and reviewers
Mongan, J., Moy, L., & Kahn, C. E. (2020) · Radiology: Artificial Intelligence · doi:10.1148/ryai.2020200029 · Cited by 1,443 · cited 1×
“…and LLMs in healthcare, including DECIDE-AI (Developmental and Exploratory Clinical Investigations of DEcision support systems driven by Artificial Intelligence), 11 CLAIM (Checklist for Artificial Intelligence in Medical Imaging), 12 and CONSORT-AI (Consolidated Standards of Reporting Trials-AI).”
(2010) · Paediatric and Perinatal Epidemiology · doi:10.1111/j.1365-3016.2010.01108.x · Cited by 1,443
Research works printed but not linked: 3
- Lefebvre C, Glanville J, Briscoe S, et al Chapter 4: searching for and selecting studies. Cochrane Handbook for Systematic Reviews of Interventions Version, Vol. 6. Cochrane, 2024. https://training.cochrane.org/handbook/current/chapter-04
- Chapter 3 Effect Sizes | Doing Meta-Analysis in R. Accessed October 13, 2024. https://bookdown.org/MathiasHarrer/Doing_Meta_Analysis_in_R/effects.html
- Xi Z, Chen W, Guo X, et al The rise and potential of large language model based agents: a survey. September 14, 2023, preprint: not peer reviewed.
Cited by
Liu, S., Wright, A. P., McCoy, A. B., Huang, S. S., Steitz, B., & Wright, A. (2025) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocaf059 · Cited by 44
“20,21 24 Integrating external knowledge, for example, a nurse triage book with over 200 protocols for common concerns and symptoms, might further support the emergency tri-age task.”
Self-citationThe perils of politeness: how large language models may amplify medical misinformation
Rosen, K. L., Sui, M., Heydari, K., Enichen, E. J., & Kvedar, J. C. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-02135-7 · Cited by 41
Xu, X., & Sankar, R. (2025) · Information · doi:10.3390/info16100894 · Cited by 20
“Prompt engineering and structured integration of retrieved content are also critical to ensure coherence and factual alignment during response synthesis [42].”
InfluentialNanua, S., Steward, R., Neely, B., Datto, M., & Youens, K. E. (2025) · Journal of Pathology Informatics · doi:10.1016/j.jpi.2025.100520 · Cited by 25
Reason, T., Klijn, S. L., Rawlinson, W. D., Benbow, E., Langham, J., Teitsson, S., Johannesen, K. M., & Malcolm, B. (2025) · PharmacoEconomics - Open · doi:10.1007/s41669-025-00580-4 · Cited by 27
Streamlining evidence based clinical recommendations with large language models
Li, D., Jiang, N., Huang, K., Tu, R., Ouyang, S., Yu, H., Lin, Q., Chen, Y., Zhou, T., Tong, D., et al. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-02273-y · Cited by 28
“Forinstance,inQuestion1,Quickerrecommendedcombinationtherapy basedsolelyonclinicalevidence,whereastheguideline — while acknowl-edging the superior ef fi cacy of combination therapy — conditionally preferred methotrexate (MTX) monotherapy due to cost and practical considerations (see Supplementary Note 4 for more detail…”
InfluentialLiu, E., Yang, H., Sharma, S., van Leerdam, M., Niu, P., VandeHaar, M. J., & Hostens, M. (2025) · Journal of Dairy Science · doi:10.3168/jds.2025-26775 · Cited by 18
Rafiq, S., Macha, M. A., & Assad, A. (2025) · International Journal of Data Science and Analytics · doi:10.1007/s41060-025-00773-4 · Cited by 14
“Moreover, the integration of LLMs with knowledge graphs has improved the understanding of biomedical literature and genomic data, so enabling more accurate predictions of gene-editing results [49].”
Madrid-García, A., Benavent, D., Plasencia‐Rodríguez, C., Rosales, Z., Barbancho, B. M., & Freites, D. (2025) · EULAR Rheumatology Open · doi:10.1016/j.ero.2025.08.001 · Cited by 19
“RAG is gaining traction due to its capacity to promote equity, reliability, and personalisation, and it has been used in some medical disciplines [11 − 13].”
Kuo, S.-M., Tai, S.-K., Lin, H., & Chen, R.-C. (2025) · AI · doi:10.3390/ai6080188 · Cited by 24
van Kessel, R., Anderson, M., McMillan, B., Matthews, M. R., Rust, P., Pearcy, P., Nasir, K., & Mossialos, E. A. (2026) · BMJ Health & Care Informatics · doi:10.1136/bmjhci-2025-101959 · Cited by 20
“39 40 It should also expand the scope of this analysis to inpatient care settings.”
Hamamoto, R., Koyama, T., Takahashi, S., Yasuda, T., Kobayashi, K., Akagi, Y., Kouno, N., Sudo, K., Hirata, M., Sunami, K., et al. (2026) · Journal of Hematology & Oncology · doi:10.1186/s13045-026-01781-y · Cited by 8
“Fourth, combining RAG-based evidence presentation (to expose the provenance of recommendations) with HITL supervision can reduce ungrounded assertions and enable clinicians to detect biased reasoning in high-stakes steps such as treatment selection and trial referral [163].”
Optimizing Order Sets With a Large Language Model–Powered Multiagent System
Liu, S., Huang, S. S., McCoy, A. B., Wright, A. P., Horst, S., & Wright, A. T. (2025) · JAMA Network Open · doi:10.1001/jamanetworkopen.2025.33277 · Cited by 22
Self-citationYoung, R., Matthews, A. M., & Poston, B. (2025) · Algorithms · doi:10.3390/a18050296 · Cited by 11
Rewthamrongsris, P., Thongchotchat, V., Burapacheep, J., Trachoo, V., Khurshid, Z., & Porntaveetus, T. (2025) · International Dental Journal · doi:10.1016/j.identj.2025.109344 · Cited by 9
“20 However, RAG does not invariably enhance LLM performance and may introduce novel challenges.”
He, J., Zhang, B., Rouhizadeh, H., Chen, Y., Yang, R., Lu, J. S., Chen, X., Liu, N., & Teodoro, D. (2026) · Artificial Intelligence Review · doi:10.21203/rs.3.rs-8330917/v1 · Cited by 33
Foundation models: the next level of AI in ART
Hammer, H. L., Thambawita, V. L., & Riegler, M. A. (2025) · Human Reproduction · doi:10.1093/humrep/deaf136 · Cited by 8
Chen, H.-K., & Wang, H.-H. (2025) · Biomimetics · doi:10.3390/biomimetics10090626 · Cited by 8
“Amugongo et al. [22] surveyed the integration of RAG with large language models for clinical knowledge translation, Liu et al. [23] provided systematic guidelines for biomedical applications through meta-analysis, and Gargari and Habibi [24] highlighted opportunities for enhancing medical AI with…”
Blašković, L., Tanković, N., Lorencin, I., & Šegota, S. B. (2025) · Big Data and Cognitive Computing · doi:10.3390/bdcc9100256 · Cited by 8
Retrieval augmented generation for building datasets from scientific literature
Maharana, P. R., Verma, A., & Joshi, K. (2025) · Journal of Physics Materials · doi:10.1088/2515-7639/ade1fa · Cited by 16
Imran, M., & Lee, Y. (2026) · IEEE Access · doi:10.1109/access.2026.3656368 · Cited by 15
“3) Hybrid / ensemble retrieval: combining dense and sparse (or multiple) retrievers [83], [110], [111] improves robustness and recall but adds orchestration complexity and operational burden in clinical settings.”
Hack, S., Craig, J. R., Lin, C.-F., Fu, C., Kwiatkowska, M. A., Kocum, P., Allevi, F., & Saibene, A. M. (2026) · European Archives of Oto-Rhino-Laryngology · doi:10.1007/s00405-026-10040-2 · Cited by 4
Madrid-García, A., Benavent, D., Plasencia-Rodríguez, C., Rosales, Z., Barbancho, B. M., & Freites, D. (2025) · medRxiv · doi:10.1101/2025.04.10.25325588 · Cited by 5
“In healthcare, RAG has gained considerable traction because it promotes equity, reliability, and personalization [14 – 16] Recently, a comprehensive introduction to RAG architecture in healthcare along with a detailed explanation of its underlying mechanisms was published by Ng et al [14] The…”
OmniCellAgent: An AI Scientist for Omic-Driven Scientific DiscoveryPreprint
Huang, D., Li, H., Li, W., Zhang, H., Xu, T., Lu, Y., Fang, K., Xu, Z., Chen, J., Dickson, P., et al. (2025) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2025.07.31.667797 · Cited by 9
LLM Selection and Vector Database Tuning: A Methodology for Enhancing RAG Systems
Pawlik, Ł. (2025) · Applied Sciences · doi:10.3390/app152010886 · Cited by 11
Kartal, A., Manalil, N. F., Cheng, C. D., Chung, L. K., Gebhard, H., Greenberg, M. D., Härtl, R., & Elsayed, G. A. (2025) · Global Spine Journal · doi:10.1177/21925682251411225 · Cited by 6
An Introduction to Pathology Foundation Models
Veremis, B. M., Chen, S., & Campanella, G. S. V. (2026) · Head and Neck Pathology · doi:10.1007/s12105-025-01878-9 · Cited by 3
AI in Pediatric Spine Care: Clinical, Research, and Ethical Considerations
Nugraha, H. K., Rasmussen, A. P., Mulford, K. L., Yang, L., Wyles, C. C., & Larson, A. N. (2025) · Journal of Clinical Medicine · doi:10.3390/jcm14228115 · Cited by 4
Wang, M., Shen, Y., Zhao, B., Zhou, X., Sun, L., & Liu, X. (2025) · Health Information Science and Systems · doi:10.1007/s13755-025-00379-x · Cited by 3
“Recent studies demonstrated that RAG implementation increased outcomes by 1.35 times compared to baseline LLMs in medical scenarios [10], suggesting the potential of RAG in anesthesiology.”
InfluentialLai, C.-H., Chen, Y.-J., & Chen, Z.-P. (2025) · Applied Sciences · doi:10.3390/app152413050 · Cited by 4
Rivera, L. R. P., Gursky, A. K., Elmer, N. A., Boyd, C. J., & Karp, N. S. (2025) · Aesthetic Surgery Journal · doi:10.1093/asj/sjaf249 · Cited by 9
Wang, J.-F., Chang, C.-C., Chiang, T.-M., Yeh, T., Cheng, E., Lee, Y., & Chen, H. (2025) · Health Informatics Journal · doi:10.1177/14604582251381233 · Cited by 4
“3,15,16 Retrieval-augmented generation (RAG) The core concept of RAG 17 is to combine external retrieval mechanisms with generative models to provide more accurate content.”
From Data‐Driven to Intelligence‐Driven: Innovating Research Paradigms of Perovskite Solar Cells
Li, A., & Yi, C. (2025) · Solar RRL · doi:10.1002/solr.202500681 · Cited by 7
Optimizing RAG-based LLMs for healthcare question answering tasks
Bedwa, M., Hooda, N., & Kumar, V. (2025) · Knowledge and Information Systems · doi:10.1007/s10115-025-02638-5 · Cited by 6
Diagnostic performance of four AI tools in pharmacology MCQs: Accuracy, sensitivity, and specificity
Al-Rahahleh, A. J., Rizik, M., Al-Ashwal, F. Y., & Abu-Farha, R. K. (2025) · PLoS ONE · doi:10.1371/journal.pone.0337688 · Cited by 2
“This performance may reflect not only improved access to external databases but also the cognitive scaffolding advantage of retrieval-augmented models, which allows them to overcome some inherent reasoning limitations seen in stand-alone LLMs [15].”
Frisone, F., Pupillo, C., Rossi, C., & Riva, G. (2026) · Frontiers in Public Health · doi:10.3389/fpubh.2026.1603238 · Cited by 4
“Recent health-information evaluations show that RAG reduces hallucinations and increases appropriate non-response when the reference corpus is reliable (25, 26).”
Large language model enhanced framework for systematic reviews and meta-analyses
Shen, J., Luo, Z.-Y., Jia, D., Wang, S.-R., Sun, F., & Wu, J.-G. (2025) · BMJ digital health & AI · doi:10.1136/bmjdhai-2025-000017 · Cited by 6
Xing, X., Lin, L., Murad, M. H., & Tong, J. (2025) · Cochrane Evidence Synthesis and Methods · doi:10.1002/cesm.70047 · Cited by 4
Cardiology knowledge assessment of retrieval-augmented open versus proprietary large language models
Tarabanis, C., Khurshid, S. S., Karamanou, A., Piperaki, R., Mavromatis, L. A., Hatzimemos, A., Tachmatzidis, D., Bakogiannis, C., Vassilikos, V., Ellinor, P. T., et al. (2026) · PLOS Digital Health · doi:10.1371/journal.pdig.0001029 · Cited by 6
“We submit that RAG does introduce inference-time (during each user query) computational overhead, but these costs are largely front-loaded, scale linearly with corpus size, and are typically far lower than those associated with large-scale model training or fine-tuning [20].”
UQA-RAG: Enhancing Urdu Question-Answer Retrieval With Semantic Similarity and LLMs
Khan, T. F., Arslan, M., Alturise, F., Almutairi, A. H., & Raza, M. T. (2026) · IEEE Access · doi:10.1109/access.2025.3649743 · Cited by 2
“On the same note, the medical and biomedical disciplines have adopted RAG to improve question-answering systems and clinical decision support [24].”
Romanyukha, A. A., Mazloumi, M., Waelheyns, T. D., Mishra, N., Jacobs, J., & Fitousi, N. T. (2025) · Physica Medica · doi:10.1016/j.ejmp.2025.105090 · Cited by 2
Evaluating a Large Reasoning Model’s Performance on Open-Ended Medical ScenariosPreprint
Hoyt, R. E., Knight, D. R. T., Haider, M., & Bajwa, M. (2025) · medRxiv · doi:10.1101/2025.04.29.25326666 · Cited by 4
“[22]”
A survey on medical multimodal retrieval-augmented generation (RAG)
Wang, R.-P., Wang, J., Huang, Y., Guo, H.-Y., & Pang, J.-J. (2026) · High-Confidence Computing · doi:10.1016/j.hcc.2026.100415 · Cited by 3
Mali, Y., Zeng, Z., Heo, K., Zhang, G., Kashyap, A., Chen, J., Keramatian, K., Saraf, G., Solmi, M., Tam, E. M., et al. (2025) · medRxiv · doi:10.64898/2025.11.30.25341311 · Cited by 1
Su, L., Xu, L., Tang, J., & Zuo, M. (2026) · BMC Nephrology · doi:10.1186/s12882-026-05048-z · Cited by 2
“They accurately specify complex clinical protocols—such as intradialytic hypotension management and vascular access surveillance—and are superior in clarifying core clinical thresholds [58].”
Tumor Board–Inspired Multiagent Artificial Intelligence System for Interpreting Oncology Guidelines
Wang, J., Arora, K., Swoboda, D. M., & Nazha, A. (2026) · JCO Clinical Cancer Informatics · doi:10.1200/cci-25-00286 · Cited by 4
Shinde, A., Mhaismale, H., Pawar, P., Mane, S., & Bembde, A. (2026) · Drug Development and Industrial Pharmacy · doi:10.1080/03639045.2026.2649846 · Cited by 3
“…in structured assessments, these models achieved accuracy rates exceeding 92% and reduced the need for manual review by approximately 89% [46,47]. generates diverse candidates; cuts Doe runs by 50% ema + 1 [24] 3: Digital twin control predictive simulation real-time data → scale-up models…”
Li, M., Chen, D., Xiao, Q., He, Z., Zhang, Y., Zhong, J., Luo, Y., & Ma, H. (2026) · Journal of Medical Internet Research · doi:10.2196/89850 · Cited by 3
“This design is consistent with growing evidence that retrieval augmentation improves reliability in medical question answering and EHR summarization [40].”
Chen, R. K., Warburton, A., Do, R., Chen, D. D., Katz, D., & Burnett, G. W. (2026) · Anesthesia & Analgesia · doi:10.1213/ane.0000000000008148 · Cited by 3
Li, J., Chang, Y., Liu, Y., Wang, Y., Ou, B., Lin, S., Tang, S.-K., & Zheng, L. (2026) · Frontiers in Medicine · doi:10.3389/fmed.2026.1804478 · Cited by 5
Ren, S., Zheng, X., Zhao, J., Du, J., Zhang, Y., Bi, C., Song, J., Zhang, J., Lang, H., Zhang, F., et al. (2026) · JMIR Medical Informatics · doi:10.2196/78519 · Cited by 1
“Retrieval-augmented generation (RAG) has been proposed as a mechanism to mitigate these shortcomings by grounding model outputs in external knowledge sources [29,30].”
Toward integrated sleep health: multimodal AI in Hang Hao Meng agent
Cai, M., Liang, S., Zhen, S., Wei, S., Sun, T., Liu, J., Mao, H., & Zhang, J. (2026) · npj Digital Medicine · doi:10.1038/s41746-026-02432-9 · Cited by 2
“By anchoring LLM responses to a veri fi ed knowledge base of clinical guidelines and medical literature, RAG signi fi cantly enhances the accuracy and safety of AI-driven advice 18,19 .”
Medical Literature in the Age of Artificial Intelligence
Scott, H. (2026) · iNew Medicine · doi:10.1002/inm3.70046 · Cited by 1
Baskaran, R., Sirikonda, S., Singh, A., Srinivasan, S., Leveridge, B., Casals-Farre, O., Kaur, H., Manivannan, S., Elangovan, K., Quek, C. W. N., et al. (2026) · Frontiers in Medicine · doi:10.3389/fmed.2026.1888765 · Cited by 1
Curated retrieval-augmented generation for dental traumatology
Espona, J., Roig, E., Garcia, M., Durán‐Sindreu, F., Abella, F., Dummer, P. M. H., González, J. A., Elmsmari, F., Pineda, K., Figueras-Álvarez, Ò., et al. (2026) · Journal of Dentistry · doi:10.1016/j.jdent.2026.106947 · Cited by 3
Mali, Y., Zeng, Z., Heo, K., Zhang, G., Kashyap, A., Chen, J., Keramatian, K., Saraf, G., Solmi, M., Tam, E., et al. (2026) · The Canadian Journal of Psychiatry · doi:10.1177/07067437261457232 · Cited by 3
A roadmap for medical large language models: a review of foundations, applications, and challenges
Sha, Y.-Y., Yu, L., Lin, Z.-H., Kaur, A., Lou, Y., Gornale, S. S., Zhang, T., Wong, L. S., Wang, Z., Yan, Y., et al. (2026) · Military Medical Research · doi:10.1016/j.mmr.2026.100050 · Cited by 4
“Because medical knowledge evolves and clinical recommendations are periodically updated, many Med-LLMs incorporate RAG and related knowledge integration mechanisms [91,95].”
InfluentialKang, X., Ding, W., Matsumoto, K., Wang, L., Yu, H.-T., & Shi, X. (2026) · Information Fusion · doi:10.1016/j.inffus.2026.104702 · Cited by 1
Performance of Large Language Models on the Radiation and Cancer Biology Practice Exam
Bertschmann, J., Xu, Y., Bayley, C., Abdellatif, A., & Lee, S. L. (2024) · Frontiers in Oncology · doi:10.1016/j.ijrobp.2024.07.1347 · Cited by 1
Salahi‐Niri, A., Safavi‐Naini, S. A. A., Devi, J., Naderi, N., Sebastian, S., Adamina, M., Nadkarni, G. N., Soroush, A., & El‐Hussuna, A. (2026) · Colorectal Disease · doi:10.1111/codi.70436 · Cited by 2
“To our knowledge, this represents the first application of RAG-based multi-guideline comparative analysis in the IBD domain [ 9–13 ].”
Zhang, Y., Li, L., Tan, C., Ji, M., Tian, X., Mu, X., Li, J., Gu, Y., & Liu, H. (2026) · JMIR Medical Informatics · doi:10.2196/98207 · Cited by 4
“With retrieval-augmented generation, external sources such as guideline statements, drug labels, and local clinical pathways can be incorporated into the reasoning process, thereby improving knowledge coverage and output consistency [27-29].”
da Silva, R. C., Romano, M., Mendes, T., Isidoro, M., Ravichandran, S., Kumar, S., van der Heijden, M., Fail, O., & Gnanapragasam, V. J. (2026) · Cureus · doi:10.7759/cureus.107449 · Cited by 1
“The volume of biomedical literature and clinical data generated daily far exceeds what any individual physician can assimilate, widening the gap between available evidence and bedside practice [1].”
Mijares, J., Gan, E., Jairath, N. K., Hamad, J., Alomari, A. K., Ramachandran, V., & Que, S. K. T. (2026) · Cancers · doi:10.3390/cancers18152524
Application of Retrieval-Augmented Generation Large Language Models in Stomatology: A Scoping Review
junshengLao, Liang, Y., & Xu, L. (2026) · International Dental Journal · doi:10.1016/j.identj.2026.111195
Lee, H., Choi, S., Kim, D., Hong, K., Kim, H., Jeong, C. W., & Lee, H.-C. (2026) · Journal of Medical Systems · doi:10.1007/s10916-026-02465-3
黄, 俊. (2026) · Advances in Clinical Medicine · doi:10.12677/acm.2026.1662488
From Automation to Autonomy: A Survey of Agentic Workflows in CI/CD Orchestration
Amar, A., Qaiser, I., & Kanwal, A. (2026) · International Journal of Innovations in Science and Technology · doi:10.33411/ijist/1813
Li, X., Chen, F., Wu, W., Zeng, L., Gu, Z., & Zhang, Y. (2026) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2026.114841
Lai, C.-H., Hsieh, C.-H., Wu, S.-W., & Chan, Y.-N. (2026) · International Conference on e-Education, e-Business, e-Management and e-Learning · doi:10.1109/ic4e69618.2026.11631527
Prabha, S., Collaço, B. G., Gomez-Cabello, C. A., Haider, S. A., Genovese, A., Fang, Z., Wood, N. G., Bagaria, S. P., Tao, C., & Forte, A. J. (2026) · Journal of Medical Internet Research · doi:10.2196/90692 · Cited by 1
Interpreting Omics Data Analysis with Large Language Models for Disease Target and Drug DiscoveryPreprint
XU, Z., Chen, W., Ren, W., Xu, T., Amaechin, S., Khan, R., Chen, Y., Province, M. A., Payne, P., & Li, F. (2026) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.64898/2026.04.30.721768 · Cited by 1
Zhou, Z. (2026) · Research Square · doi:10.21203/rs.3.rs-9171815/v1
A Multi-Sector Approach to Retrieval Augmented Generation in Agriculture
Benfenati, D., Rinaldi, A. M., Russo, C., & Tommasino, C. (2025) · Conference proceedings/Conference proceedings - IEEE International Conference on Systems, Man, and Cybernetics · doi:10.1109/smc58881.2025.11343229 · Cited by 1
“RAG has been particularly effective in complex but structured domains like biology, chemistry, and others [21]–[23], though it remains underutilized in biological and agricultural sciences.”
Stifini, D., Penna, A. D., Mihaljevic, A., Missios, P., Bitzer, M., Eickhoff, C., & Capobianco, I. (2026) · Research Square · doi:10.21203/rs.3.rs-8849187/v1
Beyond text: quantifying the impact of figure-guided retrieval in biomedical RAG systems
Venkatesh, R. J., Gajendran, S., & Kumar, C. U. O. (2026) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2026.1861611 · Cited by 1
Guan, J., Shang, J., Peng, C., & Sun, Y. (2026) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.64898/2026.07.03.736267
Beltramin, D., & Cédric Bousquet, B. (2025) · Pharmacoepidemiology and Drug Safety · doi:10.1002/pds.70137
“This approach is closer to a well-established method frequently used to improve performances of LLMs: Retrieval augmented generation (RAG) [2, 3].”
Branda, F., & Ciccozzi, M. (2026) · The Lancet Digital Health · doi:10.1016/j.landig.2026.101048
Al-Farsi, M., Al-Harthy, S., & Al-Rawahi, N. (2025) · Journal of Artificial Intelligence for Healthcare Systems · doi:10.68159/d964147354
Mingole, B., Majumdar, A., Choudhury, F. A., Kraschnewski, J. L., Sundar, S. S., & Yadav, A. (2026) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3805689.3812315 · Cited by 2
Zhu, W., Jiang, X., Zhang, L., Zhou, P., Xie, X., & Wang, H. (2025) · Journal of Translational Medicine · doi:10.1186/s12967-025-06795-7
“Future work will focus on introducing advanced large language models [29] and integrating additional modalities such as genomic or proteomic data to further improve performance.”
Lorensia, A., Vincent, D., Febrianto, D. R., Jabar, B. A., & Halim, R. (2026) · International Conference on ICT for Smart Society (ICISS) · doi:10.1109/iciss71466.2026.11715110
A Multi-Agent RAG Framework for Biomedical Literature AnalysisPreprint
Palem, R. R., Chen, H., & Yue, Z. (2026) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.64898/2026.05.26.727050
Samaan, S., Devi, J., Vincent, M. L., Coombs, S. S., Sehgal, P., Mohamed, M. F. H., Rai, V., Johnson, A. M., Yarur, A. J., Barnes, E. L., et al. (2026) · doi:10.2196/preprints.107732 · Cited by 1
AI-Empowered Handover Tool Design in Medical Practice Towards Patient Safety
Salim, K., Dao, D. K. T., & Nguyen, H. D. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-31407-9_21
AI Prompt Engineering for Neurologists and Trainees
Junior, V. F. M., Hadar, P., Murphy, S. N., & Moura, L. M. V. R. (2025) · Seminars in Neurology · doi:10.1055/a-2742-2349 · Cited by 2
“30 Before responding, the model retrieves this content and generates outputs grounded in authoritative sources, which signi fi cantly reduces hallucinations and improves clinical accuracy.”
Shahri, N. S. H., & Nasharuddin, N. A. (2026) · Communications in computer and information science · doi:10.1007/978-981-92-4302-0_18
Jeong, E., Jeong, W., Song, E., Park, E. H., Oh, K. H., & Jun, J. K. (2026) · Digital Health · doi:10.1177/20552076261450314
Levy, R., Phillips, R. L., O’Sullivan, G. M., Capra, S., Penm, J. H.-D., Lee, A., Sagnella, S., Segura, J. M., Forrey, R. A., Patanwala, A. E., et al. (2026) · Journal of the American Pharmacists Association · doi:10.1016/j.japh.2026.103063
Popescu, A., Ionescu, M., Stan, E., Dumitrescu, S., & Pavel, I. (2026) · Journal of Health Informatics and Digital Systems · doi:10.68159/j827489866
Automated Review of Patient Records
Nguyen, T. T., Li, K. Z., Chaitanuwong, P., & Moss, H. E. (2026) · Ophthalmology Science · doi:10.1016/j.xops.2026.101265
Med.ai ASK: an agentic system for biomedical question answering
Nguyen, N. T. H., Lituiev, D., Liu, Z., Kashyap, A., Jenkinson, G., Kuhl, K., Corrado, C. T., Patel, N. M., Snigdha, K., Saeedi, S., et al. (2026) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocag038 · Cited by 2
(2025) · doi:10.1109/ieem63636.2025.11357684
“Secondly, RAG significantly improves the factual grounding of LLM responses [12].”
InfluentialTruong, V. Q., Yang, S., Shen, L., & Ritchie, M. D. (2025) · IEEE International Conference on Bioinformatics and Biomedicine (BIBM) · doi:10.1109/bibm66473.2025.11356640
Krump, P. A., Blasingame, M. N., Koonce, T. Y., Williams, A. M., Su, J., & Giuse, N. B. (2026) · medRxiv · doi:10.64898/2026.08.10.26360108
ReviewGuard: Enhancing Deficient Peer Review Detection via LLM-Driven Data Augmentation
Zhang, H., Li, R., Shrestha, S., Mamidala, S. H., Putta, R., Aggarwal, A., Xiao, T., Ding, J., & Chen, H. (2025) · ACM/IEEE Joint Conference on Digital Libraries · doi:10.1109/jcdl67857.2025.00032 · Cited by 6
“RAG integrates targeted literature retrieval into the generation process to ground assessments in verifiable evidence [27], [28], while fine-tuning adapts LLMs to the stylistic and evaluative norms of peer review, improving factual accuracy and alignment with domain-specific criteria [3], [29].”
Sánchez, I. S. T., Manrique, R., Ortega, N., et al. (2026) · IEEE Colombian Conference on Applications of Computational Intelligence (ColCACI) · doi:10.1109/colcaci71483.2026.11691480
Exploring RAGs Applications in Healthcare
Almohsen, K., & Hewahi, N. (2025) · International Conference on Foundation and Large Language Models (FLLM) · doi:10.1109/fllm67465.2025.11391150
“These limitations include [2], [5], [7], [13]: • Knowledge limitation as these models are pre-trained on fixed datasets, resulting in having a static knowledge base that is restricted to the information present at the time of training, which might become outdated.”
InfluentialAnghelescu, A., Munteanu, C., Anghelescu, L. A.-M., & Onose, G. (2025) · Balneo and PRM Research Journal · doi:10.12680/balneo.2025.847 · Cited by 1
Metadata
- Authors
- Liu, S., McCoy, A. B., & Wright, A.
- Affiliations
- Vanderbilt University (US)
- Vanderbilt University Medical Center (US)
- Corresponding author
- Liu, Siru
- Venue
- Journal of the American Medical Informatics Association
- Publisher
- Oxford University Press (OUP)
- Published
- 2025-01-15
- Volume
- 32
- Issue
- 4
- Pages
- 605-615
- ISSN
- 1067-5027 (ISSN-L)
- 1527-974X
- PMC
- PMC12005634
- OpenAlex
- W4406421570
- Cited by
- 200
- References
- 49
- Citations by year
- 2026: 138 · 2025: 60
- FWCI
- 11.70
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Journal of the American Medical Informatics Association (publishedVersion, cc-by-nc)
- PubMed Central (publishedVersion, cc-by-nc)
- PubMed (publishedVersion)
- Funders
- National Institutes of Health (R00LM014097-02, R01LM013995-01)
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
- Keywords
- retrieval-augmented generation, large language models, biomedical applications, medical question answering, electronic health record integration, LLM agents
- MeSH terms
- Large Language Models, Humans, Natural Language Processing, Programming Languages, Information Storage and Retrieval
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.