The DOI matches its Crossref record; 60 of the 60 references it cites were found in a registry or an index; 19 of the 27 sources its claims cite were checked for support.
Improving large language model applications in biomedicine with retrieval-augmented generation: a systematic review, meta-analysis, and clinical development guidelines
Open accessAbstract
OBJECTIVE: The objectives of this study are to synthesize findings from recent research of retrieval-augmented generation (RAG) and large language models (LLMs) in biomedicine and provide clinical development guidelines to improve effectiveness. MATERIALS AND METHODS: We conducted a systematic literature review and a meta-analysis. The report was created in adherence to the Preferred Reporting Items for Systematic Reviews and Meta-Analyses 2020 analysis. Searches were performed in 3 databases (PubMed, Embase, PsycINFO) using terms related to "retrieval augmented generation" and "large language model," for articles published in 2023 and 2024. We selected studies that compared baseline LLM performance with RAG performance. We developed a random-effect meta-analysis model, using odds ratio as the effect size. RESULTS: Among 335 studies, 20 were included in this literature review. The pooled effect size was 1.35, with a 95% confidence interval of 1.19-1.53, indicating a statistically significant effect (P = .001). We reported clinical tasks, baseline LLMs, retrieval sources and strategies, as well as evaluation methods. DISCUSSION: Building on our literature review, we developed Guidelines for Unified Implementation and Development of Enhanced LLM Applications with RAG in Clinical Settings to inform clinical applications using RAG. CONCLUSION: Overall, RAG implementation showed a 1.35 odds ratio increase in performance compared to baseline LLMs. Future research should focus on (1) system-level enhancement: the combination of RAG and agent, (2) knowledge-level enhancement: deep integration of knowledge into LLM, and (3) integration-level enhancement: integrating RAG systems within electronic health records.
Overview
CoChat Summary
TL;DR
This systematic review and random-effects meta-analysis found that retrieval-augmented generation (RAG) improved biomedical LLM performance versus baseline LLMs, with a pooled effect size of 1.35 (95% CI 1.19-1.53). The authors also proposed GUIDE-RAG clinical development guidelines to improve performance, transparency, and integration of RAG systems in clinical settings.
Key findings
- Among 335 studies identified, 20 were included; all 20 were published in 2024, and 1 was a conference paper.
- RAG improved performance compared with baseline LLMs with a pooled effect size of 1.35 (95% CI 1.19-1.53; P = .001); heterogeneity was low to moderate (I2 = 37%), and the prediction interval was 1.01 to 1.8.
- Subgroup analysis found similar benefit for clinical decision-making (13 studies; OR 1.46, 95% CI 1.16-1.71) and medical question-answering (7 studies; OR 1.32, 95% CI 1.08-1.63), with no statistically significant difference between groups.
- Human evaluation showed a larger effect than automatic evaluation: OR 1.65 (95% CI 1.36-2.03) vs OR 1.20 (95% CI 1.1-1.41), with the difference statistically significant (P < .01).
Methods
- Systematic literature review and meta-analysis conducted according to PRISMA 2020 and Cochrane Handbook guidance.
- Searches were run on December 12, 2024 in PubMed, Embase, and PsycINFO for peer-reviewed English-language papers published in 2023-2024 that compared baseline LLM performance with RAG performance on biomedical questions.
- Meta-analysis used a random-effects model with odds ratio as the effect size; subgroup analyses examined baseline LLM, retrieval strategy, evaluation method, and task type.
Limitations
- The paper states that it was limited to peer-reviewed biomedical database publications, excluded preprints, included only English-language studies, and did not search sources such as IEEE Xplore or Google Scholar.
- It also notes that free-text searching with automatic MeSH/Emtree mapping may introduce variability if underlying algorithms change.
- Evident from the text: the included studies were heterogeneous in RAG architecture, datasets, retrieval sources, and evaluation methods, which is why a random-effects model was used.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures




Verification
Claims the paper makes
11 Valid · 4 Not valid · 6 Not checked · 42 Own results and interpretations
The cited sources that could be checked couldn’t confirm or dispute this claim.
In the paper
“Accuracy was the most frequently reported metric, covering Likert scale ratings, match rates, correct treatment percentages,9 AUC, AUPRC, and F1 scores, as well as text similarity metrics like ROUGE (ROUGE-1, ROUGE-2, ROUGE-L), BLEU, METEOR, and BERTScore,21 which compared LLM-generated responses to expert-provided answers.”
§ Improving large language model applications in biomedicine with retrieval-augmented generation: a systematic review, meta-analysis, and clinical development guidelines > Results > Evaluation
What the sources say
- Could not verifycites [9] Ayala, 2024, Reducing hallucination in structured outputs via Retrieval-Augmented Generation sourceExperimental — accuracy not yet measured
Section 4.2 (Metrics; page unavailable) describes the paper’s own measures—trigger exact match, bag of steps, hallucination rates, and retrieval recall—but does not compare how frequently evaluation metrics are reported. Searches for the claimed frequency and the named text-similarity metrics found no passage addressing that point. (LLM, conf=0.76)
- Not checked — no full text of the sourcecites [21] Peters, 2008, Contour-enhanced meta-analysis funnel plots help distinguish publication bias from other causes of asymmetry, J Clin Epidemiol source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Could not verify
The cited sources that could be checked couldn’t confirm or dispute this claim.
In the paper
“However, these guidelines do not cover RAG applications in clinical settings, emphasizing the need for a more specific guideline.”
§ Improving large language model applications in biomedicine with retrieval-augmented generation: a systematic review, meta-analysis, and clinical development guidelines > Introduction
What the sources say
- Could not verifycites [11] Vasey, 2022, Reporting guideline for the early stage clinical evaluation of decision support systems driven by artificial intelligence: DECIDE-AI, BMJ sourceExperimental — accuracy not yet measured
The Abstract describes a gap in guidance for early clinical evaluation of AI systems, and Table 1 describes DECIDE-AI as a guideline for evaluating AI interventions in live clinical settings. Neither passage establishes whether existing guidelines cover clinical RAG applications. Searches for “RAG” and “retrieval augmented generation” found no passage addressing that question. (LLM, conf=0.75)
- Could not verifycites [12] Mongan, 2020, Checklist for Artificial Intelligence in Medical Imaging (CLAIM): a guide for authors and reviewers, Radiol Artif Intell sourceExperimental — accuracy not yet measured
The Abstract (page unavailable) describes CLAIM as a guideline for AI research in medical imaging, including text analysis. Neither that passage nor the searched passages on RAG, large language models, and clinical applications assess whether existing guidelines cover clinical RAG applications; the absence of RAG from this paper does not establish the broader claim. (LLM, conf=0.74)
- Could not verifycites [13] Martindale, 2024, Concordance of randomised controlled trials for artificial intelligence interventions with the CONSORT-AI reporting guidelines, Nat Commun sourceExperimental — accuracy not yet measured
The Discussion says that medical large language models could be a focus of future CONSORT-AI iterations, but it does not say that existing guidelines exclude RAG applications in clinical settings. Searches for “RAG” and “retrieval augmented generation” found no passage addressing that specific gap or the need for a RAG-specific guideline. (LLM, conf=0.72)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Could not verify
Every source the paper cites for this claim was checked and supports it.
In the paper
“a preliminary study found that GPT-4 Turbo had only a 16% accuracy rate in extracting table data.27”
§ Improving large language model applications in biomedicine with retrieval-augmented generation: a systematic review, meta-analysis, and clinical development guidelines > Discussion
What the sources say
- Supportedcites [27] Kresevic, 2024, Optimization of hepatological clinical guidelines interpretation by large language models: a retrieval augmented generation-based framework, NPJ Digit Med source
The Methods section (Ablation study: customized LLM framework; page not provided) reports preliminary experiments in which the GPT image-conversion process achieved 16.0% accuracy in extracting pertinent table information. The claim paraphrases that finding; the separate 28.0% result concerns accuracy on table-based questions, not extraction of table information. (LLM, conf=0.88)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The sources with a verdict support this claim; the rest weren’t checked yet, have no full text we could read, or their verdicts aren’t rated yet.
In the paper
“Only 3 studies in our review explicitly mentioned extracting text from tables or figures during the pre-retrieval process.25,27,44”
§ Improving large language model applications in biomedicine with retrieval-augmented generation: a systematic review, meta-analysis, and clinical development guidelines > Discussion
What the sources say
- Not checked — no full text of the sourcecites [25] Yazaki, 2024, Emergency patient triage improvement through a retrieval-augmented generation enhanced large-scale language model, Prehosp Emerg Care source
- Supportedcites [27] Kresevic, 2024, Optimization of hepatological clinical guidelines interpretation by large language models: a retrieval augmented generation-based framework, NPJ Digit Med source
The Methods section (Ablation study: customized LLM framework) explicitly says that tables presented as images were manually converted to CSV files, then converted into text-based lists for use as context. The citation identifies this paper as one of the citing review’s three studies; it does not ask this individual paper to establish the review’s count. (LLM, conf=0.91)
- Supportedcites [44] Hewitt, 2024, Large language models as a diagnostic support tool in neuropathology, J Pathol Clin Res source
The Materials and methods section (‘RAG approach’; page unavailable) says the study converted relevant tables to plain text for use by the LLMs. That supports citing it as an example of table-text preparation before retrieval; the count of three studies is the citing review’s own finding, not a claim this paper makes. (LLM, conf=0.88)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Every source the paper cites for this claim was checked and supports it.
In the paper
“Only 1 study in our review, which focused on gastrointestinal radiology diagnosis based on imaging descriptions, compared the cost and response time between LLMs and LLMs with RAG.31”
§ Improving large language model applications in biomedicine with retrieval-augmented generation: a systematic review, meta-analysis, and clinical development guidelines > Discussion
What the sources say
- Supportedcites [31] Rau, 2024, A retrieval-augmented chatbot based on GPT-4 provides appropriate differential diagnosis in gastrointestinal radiology: a proof of concept study, Eur Radiol Exp source
The Methods describe a gastrointestinal imaging-aware GPT-4 chatbot with knowledge retrieval and a generic GPT-4 comparator, tested on descriptions of radiological findings. The Results > Statistical evaluation reports response time and cost per case for both. The statement that this was the only such study in the citing authors’ review is their own characterization, not a finding attributed to this source. (LLM, conf=0.93)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
19 more claims · 1 with their cited sources checked
Create a free account to see all 63 claims and every source verdict.
- It may state two separate claims as one. 20
- Probably not a substantial claim. 24
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Rated major: it would touch a result — a value a result rests on, reported and recomputed p-values on opposite sides of the significance level, two passages that conflict with no sign which is right, or a cited work retracted before this paper was published.
In the paper
“In the subgroup analysis, 12 studies used simple data retrieval strategies (OR 1.30, 95% CI [1.16, 1.45]), while 5 studies used complex data retrieval strategies (OR 1.30, 95% CI [1.07, 1.24]), with no statistically significant difference observed between the 2 approaches.”
§ Improving large language model applications in biomedicine with retrieval-augmented generation: a systematic review, meta-analysis, and clinical development guidelines > Results > Retrieval strategies
Reported 1.3 · recomputed 0.06
Citations
Cites
Bias in meta-analysis detected by a simple, graphical test
Egger, M., Smith, G. D., Schneider, M., & Minder, C. (1997) · BMJ · doi:10.1136/bmj.315.7109.629 · Cited by 56,722
Quantifying heterogeneity in a meta‐analysis
Higgins, J. P. T., & Thompson, S. G. (2002) · Statistics in Medicine · doi:10.1002/sim.1186 · Cited by 36,763
Shamseer, L., Moher, D., Clarke, M., Ghersi, D., Liberati, A., Petticrew, M., Shekelle, P., Stewart, L. A., & the PRISMA-P Group (2015) · BMJ · doi:10.1136/bmj.g7647 · Cited by 13,120
Large language models in medicine
Thirunavukarasu, A. J., Ting, D. S. J., Elangovan, K., Gutierrez, L., Tan, T. F., & Ting, D. S. W. (2023) · Nature Medicine · doi:10.1038/s41591-023-02448-8 · Cited by 3,944
Ayers, J. W., Poliak, A., Dredze, M., Leas, E. C., Zhu, Z., Kelley, J. B., Faix, D. J., Goodman, A. M., Longhurst, C. A., Hogarth, M., et al. (2023) · JAMA Internal Medicine · doi:10.1001/jamainternmed.2023.1838 · Cited by 2,613
Peters, J. L., Sutton, A. J., Jones, D. R., Abrams, K. R., & Rushton, L. (2008) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2007.11.010 · Cited by 2,053 · cited 2×
“Accuracy was the most frequently reported metric, covering Likert scale ratings, match rates, correct treatment percentages, 9 AUC, AUPRC, and F1 scores, as well as text similarity metrics like ROUGE (ROUGE-1, ROUGE-2, ROUGE-L), BLEU, METEOR, and BERTScore, 21 which compared LLM-generated responses to expert-provided…”
A survey on large language model based autonomous agents
Wang, L., Ma, C., Feng, X., Zhang, Z., Yang, H., Zhang, J., Chen, Z., Tang, J., Chen, X., Lin, Y., et al. (2024) · Frontiers of Computer Science · doi:10.1007/s11704-024-40231-1 · Cited by 1,941
A Survey on Large Language Model based Autonomous Agents
Wang, L., Ma, C., Feng, X., Zhang, Z., Yang, H., Zhang, J., Chen, Z., Tang, J., Chen, X., Lin, Y., et al. (2023) · arXiv · doi:10.48550/arxiv.2308.11432 · Cited by 1,925
Checklist for Artificial Intelligence in Medical Imaging (CLAIM): a guide for authors and reviewers
Mongan, J., Moy, L., & Kahn, C. E. (2020) · Radiology: Artificial Intelligence · doi:10.1148/ryai.2020200029 · Cited by 1,443 · cited 1×
“…and LLMs in healthcare, including DECIDE-AI (Developmental and Exploratory Clinical Investigations of DEcision support systems driven by Artificial Intelligence), 11 CLAIM (Checklist for Artificial Intelligence in Medical Imaging), 12 and CONSORT-AI (Consolidated Standards of Reporting Trials-AI).”
(2010) · Paediatric and Perinatal Epidemiology · doi:10.1111/j.1365-3016.2010.01108.x · Cited by 1,443
Research works printed but not linked: 2
- Lefebvre, Chapter 4: searching for and selecting studies, Cochrane Handbook for Systematic Reviews of Interventions Version, Vol., 6
- Chapter 3 Effect Sizes
Cited by
Liu, S., Wright, A. P., McCoy, A. B., Huang, S. S., Steitz, B., & Wright, A. (2025) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocaf059 · Cited by 44
“20,21 24 Integrating external knowledge, for example, a nurse triage book with over 200 protocols for common concerns and symptoms, might further support the emergency tri-age task.”
Self-citationThe perils of politeness: how large language models may amplify medical misinformation
Rosen, K. L., Sui, M., Heydari, K., Enichen, E. J., & Kvedar, J. C. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-02135-7 · Cited by 41
Xu, X., & Sankar, R. (2025) · Information · doi:10.3390/info16100894 · Cited by 20
“Prompt engineering and structured integration of retrieved content are also critical to ensure coherence and factual alignment during response synthesis [42].”
InfluentialNanua, S., Steward, R., Neely, B., Datto, M., & Youens, K. E. (2025) · Journal of Pathology Informatics · doi:10.1016/j.jpi.2025.100520 · Cited by 25
Reason, T., Klijn, S. L., Rawlinson, W. D., Benbow, E., Langham, J., Teitsson, S., Johannesen, K. M., & Malcolm, B. (2025) · PharmacoEconomics - Open · doi:10.1007/s41669-025-00580-4 · Cited by 26
Metadata
- Authors
- Liu, S., McCoy, A. B., & Wright, A.
- Affiliations
- Vanderbilt University (US)
- Vanderbilt University Medical Center (US)
- Corresponding author
- Liu, Siru
- Venue
- Journal of the American Medical Informatics Association
- Publisher
- Oxford University Press (OUP)
- Published
- 2025-01-15
- Volume
- 32
- Issue
- 4
- Pages
- 605-615
- ISSN
- 1067-5027 (ISSN-L)
- 1527-974X
- PMC
- PMC12005634
- OpenAlex
- W4406421570
- Cited by
- 200
- References
- 49
- Citations by year
- 2026: 138 · 2025: 60
- FWCI
- 11.70
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Journal of the American Medical Informatics Association (publishedVersion, cc-by-nc)
- PubMed Central (publishedVersion, cc-by-nc)
- PubMed (publishedVersion)
- Funders
- National Institutes of Health (R00LM014097-02, R01LM013995-01)
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
- Keywords
- retrieval-augmented generation, large language models, biomedical applications, medical question answering, electronic health record integration, LLM agents
- MeSH terms
- Large Language Models, Humans, Natural Language Processing, Programming Languages, Information Storage and Retrieval
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Improving large language model applications in biomedicine with retrieval-augmented generation: a systematic review, meta-analysis, and clinical development guidelines. https://app.cochat.ai/papers/10.1093/jamia/ocaf008
- The paper
- Liu, S., McCoy, A. B., & Wright, A. (2025). Improving large language model applications in biomedicine with retrieval-augmented generation: a systematic review, meta-analysis, and clinical development guidelines. Journal of the American Medical Informatics Association. https://doi.org/10.1093/jamia/ocaf008
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.