CoChat

The DOI matches its Crossref record; 60 of the 60 references it cites were found in a registry or an index; 19 of the 27 sources its claims cite were checked for support.

Improving large language model applications in biomedicine with retrieval-augmented generation: a systematic review, meta-analysis, and clinical development guidelines

Open access
Liu, Siru, McCoy, Allison B, Wright, Adam · 2025 · Journal of the American Medical Informatics Association

Abstract

OBJECTIVE: The objectives of this study are to synthesize findings from recent research of retrieval-augmented generation (RAG) and large language models (LLMs) in biomedicine and provide clinical development guidelines to improve effectiveness. MATERIALS AND METHODS: We conducted a systematic literature review and a meta-analysis. The report was created in adherence to the Preferred Reporting Items for Systematic Reviews and Meta-Analyses 2020 analysis. Searches were performed in 3 databases (PubMed, Embase, PsycINFO) using terms related to "retrieval augmented generation" and "large language model," for articles published in 2023 and 2024. We selected studies that compared baseline LLM performance with RAG performance. We developed a random-effect meta-analysis model, using odds ratio as the effect size. RESULTS: Among 335 studies, 20 were included in this literature review. The pooled effect size was 1.35, with a 95% confidence interval of 1.19-1.53, indicating a statistically significant effect (P = .001). We reported clinical tasks, baseline LLMs, retrieval sources and strategies, as well as evaluation methods. DISCUSSION: Building on our literature review, we developed Guidelines for Unified Implementation and Development of Enhanced LLM Applications with RAG in Clinical Settings to inform clinical applications using RAG. CONCLUSION: Overall, RAG implementation showed a 1.35 odds ratio increase in performance compared to baseline LLMs. Future research should focus on (1) system-level enhancement: the combination of RAG and agent, (2) knowledge-level enhancement: deep integration of knowledge into LLM, and (3) integration-level enhancement: integrating RAG systems within electronic health records.

Overview

CoChat Summary

TL;DR

This systematic review and random-effects meta-analysis found that retrieval-augmented generation (RAG) improved biomedical LLM performance versus baseline LLMs, with a pooled effect size of 1.35 (95% CI 1.19-1.53). The authors also proposed GUIDE-RAG clinical development guidelines to improve performance, transparency, and integration of RAG systems in clinical settings.

Key findings

  • Among 335 studies identified, 20 were included; all 20 were published in 2024, and 1 was a conference paper.
  • RAG improved performance compared with baseline LLMs with a pooled effect size of 1.35 (95% CI 1.19-1.53; P = .001); heterogeneity was low to moderate (I2 = 37%), and the prediction interval was 1.01 to 1.8.
  • Subgroup analysis found similar benefit for clinical decision-making (13 studies; OR 1.46, 95% CI 1.16-1.71) and medical question-answering (7 studies; OR 1.32, 95% CI 1.08-1.63), with no statistically significant difference between groups.
  • Human evaluation showed a larger effect than automatic evaluation: OR 1.65 (95% CI 1.36-2.03) vs OR 1.20 (95% CI 1.1-1.41), with the difference statistically significant (P < .01).

Methods

  • Systematic literature review and meta-analysis conducted according to PRISMA 2020 and Cochrane Handbook guidance.
  • Searches were run on December 12, 2024 in PubMed, Embase, and PsycINFO for peer-reviewed English-language papers published in 2023-2024 that compared baseline LLM performance with RAG performance on biomedical questions.
  • Meta-analysis used a random-effects model with odds ratio as the effect size; subgroup analyses examined baseline LLM, retrieval strategy, evaluation method, and task type.

Limitations

  • The paper states that it was limited to peer-reviewed biomedical database publications, excluded preprints, included only English-language studies, and did not search sources such as IEEE Xplore or Google Scholar.
  • It also notes that free-text searching with automatic MeSH/Emtree mapping may introduce variability if underlying algorithms change.
  • Evident from the text: the included studies were heterogeneous in RAG architecture, datasets, retrieval sources, and evaluation methods, which is why a random-effects model was used.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1093/jamia/ocaf008

Cites

60 of 60 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Figures

Figure 1. Flow diagram of included studies illustrating the systematic review process. Out of 335 records identified from PubMed, Embase, and PsycINFO, 84 duplicates were removed, leaving 251 records screened. After excluding 200 records, 51 reports were assessed for eligibility. 31 reports were excluded for reasons such as lack of RAG-LLM comparison, non-clinical focus, or commentary. Ultimately, 20 studies were included in the final review. LLM, large language model; RAG, retrieval-augmented generation.
Figure 1. Flow diagram of included studies illustrating the systematic review process. Out of 335 records identified from PubMed, Embase, and PsycINFO, 84 duplicates were removed, leaving 251 records screened. After excluding 200 records, 51 reports were assessed for eligibility. 31 reports were excluded for reasons such as lack of RAG-LLM comparison, non-clinical focus, or commentary. Ultimately, 20 studies were included in the final review. LLM, large language model; RAG, retrieval-augmented generation.CC BY-NC · PubMed Central
Figure 2. A forest plot showing the odds ratio (OR) of total impacts of the RAG-enhanced system compared with the baseline LLM system in clinical tasks. The left panel lists individual studies (e.g., Zakka et al., Murugan et al., Long et al.) along with their corresponding ORs and 95% confidence intervals (CIs). The right panel visualizes these ORs, with square markers proportional to study weights and horizontal lines representing CIs. The overall pooled OR is 1.35 (95% CI: 1.19–1.53), represented by a diamond at the bottom.
Figure 2. A forest plot showing the odds ratio (OR) of total impacts of the RAG-enhanced system compared with the baseline LLM system in clinical tasks. The left panel lists individual studies (e.g., Zakka et al., Murugan et al., Long et al.) along with their corresponding ORs and 95% confidence intervals (CIs). The right panel visualizes these ORs, with square markers proportional to study weights and horizontal lines representing CIs. The overall pooled OR is 1.35 (95% CI: 1.19–1.53), represented by a diamond at the bottom.CC BY-NC · PubMed Central
Figure 3. An example of using RAG in clinical applications, with identified strategies in 3 stages: pre-retrieval, retrieval, and post-retrieval. The user query seeks recommended medications for symptomatic peripheral artery disease without increased bleeding risk. The system retrieves evidence from clinical guidelines, processes it through embeddings and a vector database, and outputs a response, including rivaroxaban with low-dose aspirin, as recommended by the retrieved guideline information. In this example, GPT-4 suggested a dose of 75 mg Aspirin, but the common low-dose Aspirin is 81 mg.
Figure 3. An example of using RAG in clinical applications, with identified strategies in 3 stages: pre-retrieval, retrieval, and post-retrieval. The user query seeks recommended medications for symptomatic peripheral artery disease without increased bleeding risk. The system retrieves evidence from clinical guidelines, processes it through embeddings and a vector database, and outputs a response, including rivaroxaban with low-dose aspirin, as recommended by the retrieved guideline information. In this example, GPT-4 suggested a dose of 75 mg Aspirin, but the common low-dose Aspirin is 81 mg.CC BY-NC · PubMed Central
Figure 4. Overview of GUIDE-RAG: This framework streamlines RAG in clinical applications through three iterative stages. In the pre-retrieval stage, it focuses on defining tasks, identifying relevant resources, and preprocessing content. The retrieval stage enhances data retrieval with optimized chunking, indexing, and advanced strategies such as graph-based retrieval. The post-retrieval stage emphasizes system evaluation, monitoring, knowledge updates, and implementing few-shot learning for complex clinical reasoning, ensuring robust and adaptive performance. CoT, chain-of-thought.
Figure 4. Overview of GUIDE-RAG: This framework streamlines RAG in clinical applications through three iterative stages. In the pre-retrieval stage, it focuses on defining tasks, identifying relevant resources, and preprocessing content. The retrieval stage enhances data retrieval with optimized chunking, indexing, and advanced strategies such as graph-based retrieval. The post-retrieval stage emphasizes system evaluation, monitoring, knowledge updates, and implementing few-shot learning for complex clinical reasoning, ensuring robust and adaptive performance. CoT, chain-of-thought.CC BY-NC · PubMed Central

Verification

Claims the paper makes

11 Valid · 4 Not valid · 6 Not checked · 42 Own results and interpretations

19 more claims · 1 with their cited sources checked

Create a free account to see all 63 claims and every source verdict.

See all claims — free account
  • It may state two separate claims as one. 20
  • Probably not a substantial claim. 24

Create a free account to see these claims.

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

1 of 1

Citations

Cites

Show all 49 references

Research works printed but not linked: 2

  1. Lefebvre, Chapter 4: searching for and selecting studies, Cochrane Handbook for Systematic Reviews of Interventions Version, Vol., 6
  2. Chapter 3 Effect Sizes

Cited by

See all 200

Metadata

Authors
Liu, S., McCoy, A. B., & Wright, A.
Affiliations
  • Vanderbilt University (US)
  • Vanderbilt University Medical Center (US)
Corresponding author
Liu, Siru
Venue
Journal of the American Medical Informatics Association
Published
2025-01-15
Volume
32
Issue
4
Pages
605-615
ISSN
  • 1067-5027 (ISSN-L)
  • 1527-974X
OpenAlex
W4406421570
Cited by
200
References
49
Citations by year
2026: 138 · 2025: 60
FWCI
11.70
Type
journal article
Language
en
Open access
Locations
  • Journal of the American Medical Informatics Association (publishedVersion, cc-by-nc)
  • PubMed Central (publishedVersion, cc-by-nc)
  • PubMed (publishedVersion)
Funders
National Institutes of Health (R00LM014097-02, R01LM013995-01)
Topics
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
Keywords
retrieval-augmented generation, large language models, biomedical applications, medical question answering, electronic health record integration, LLM agents
MeSH terms
Large Language Models, Humans, Natural Language Processing, Programming Languages, Information Storage and Retrieval

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (gpt-6-sol).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Improving large language model applications in biomedicine with retrieval-augmented generation: a systematic review, meta-analysis, and clinical development guidelines. https://app.cochat.ai/papers/10.1093/jamia/ocaf008
The paper
Liu, S., McCoy, A. B., & Wright, A. (2025). Improving large language model applications in biomedicine with retrieval-augmented generation: a systematic review, meta-analysis, and clinical development guidelines. Journal of the American Medical Informatics Association. https://doi.org/10.1093/jamia/ocaf008

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.