Improving large language model applications in biomedicine with retrieval-augmented generation: a systematic review, meta-analysis, and clinical development guidelines
Open accessThe DOI matches its Crossref record; 60 of the 60 references it cites were found in a registry or an index; 19 of the 27 sources its claims cite were checked for support.
Abstract
OBJECTIVE: The objectives of this study are to synthesize findings from recent research of retrieval-augmented generation (RAG) and large language models (LLMs) in biomedicine and provide clinical development guidelines to improve effectiveness. MATERIALS AND METHODS: We conducted a systematic literature review and a meta-analysis. The report was created in adherence to the Preferred Reporting Items for Systematic Reviews and Meta-Analyses 2020 analysis. Searches were performed in 3 databases (PubMed, Embase, PsycINFO) using terms related to "retrieval augmented generation" and "large language model," for articles published in 2023 and 2024. We selected studies that compared baseline LLM performance with RAG performance. We developed a random-effect meta-analysis model, using odds ratio as the effect size. RESULTS: Among 335 studies, 20 were included in this literature review. The pooled effect size was 1.35, with a 95% confidence interval of 1.19-1.53, indicating a statistically significant effect (P = .001). We reported clinical tasks, baseline LLMs, retrieval sources and strategies, as well as evaluation methods. DISCUSSION: Building on our literature review, we developed Guidelines for Unified Implementation and Development of Enhanced LLM Applications with RAG in Clinical Settings to inform clinical applications using RAG. CONCLUSION: Overall, RAG implementation showed a 1.35 odds ratio increase in performance compared to baseline LLMs. Future research should focus on (1) system-level enhancement: the combination of RAG and agent, (2) knowledge-level enhancement: deep integration of knowledge into LLM, and (3) integration-level enhancement: integrating RAG systems within electronic health records.
Overview
CoChat Summary
TL;DR
This systematic review and random-effects meta-analysis found that retrieval-augmented generation (RAG) improved biomedical LLM performance versus baseline LLMs, with a pooled effect size of 1.35 (95% CI 1.19-1.53). The authors also proposed GUIDE-RAG clinical development guidelines to improve performance, transparency, and integration of RAG systems in clinical settings.
Key findings
- Among 335 studies identified, 20 were included; all 20 were published in 2024, and 1 was a conference paper.
- RAG improved performance compared with baseline LLMs with a pooled effect size of 1.35 (95% CI 1.19-1.53; P = .001); heterogeneity was low to moderate (I2 = 37%), and the prediction interval was 1.01 to 1.8.
- Subgroup analysis found similar benefit for clinical decision-making (13 studies; OR 1.46, 95% CI 1.16-1.71) and medical question-answering (7 studies; OR 1.32, 95% CI 1.08-1.63), with no statistically significant difference between groups.
- Human evaluation showed a larger effect than automatic evaluation: OR 1.65 (95% CI 1.36-2.03) vs OR 1.20 (95% CI 1.1-1.41), with the difference statistically significant (P < .01).
Methods
- Systematic literature review and meta-analysis conducted according to PRISMA 2020 and Cochrane Handbook guidance.
- Searches were run on December 12, 2024 in PubMed, Embase, and PsycINFO for peer-reviewed English-language papers published in 2023-2024 that compared baseline LLM performance with RAG performance on biomedical questions.
- Meta-analysis used a random-effects model with odds ratio as the effect size; subgroup analyses examined baseline LLM, retrieval strategy, evaluation method, and task type.
Limitations
- The paper states that it was limited to peer-reviewed biomedical database publications, excluded preprints, included only English-language studies, and did not search sources such as IEEE Xplore or Google Scholar.
- It also notes that free-text searching with automatic MeSH/Emtree mapping may introduce variability if underlying algorithms change.
- Evident from the text: the included studies were heterogeneous in RAG architecture, datasets, retrieval sources, and evaluation methods, which is why a random-effects model was used.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures




Verification
Claims the paper makes
11 Valid · 4 Not valid · 6 Not checked · 42 Own results and interpretations
The cited sources that could be checked couldn’t confirm or dispute this claim.
In the paper
“Accuracy was the most frequently reported metric, covering Likert scale ratings, match rates, correct treatment percentages,9 AUC, AUPRC, and F1 scores, as well as text similarity metrics like ROUGE (ROUGE-1, ROUGE-2, ROUGE-L), BLEU, METEOR, and BERTScore,21 which compared LLM-generated responses to expert-provided answers.”
§ Improving large language model applications in biomedicine with retrieval-augmented generation: a systematic review, meta-analysis, and clinical development guidelines > Results > Evaluation
What the sources say
- Could not verifycites [9] Ayala, 2024, Reducing hallucination in structured outputs via Retrieval-Augmented Generation sourceExperimental — accuracy not yet measured
Section 4.2 (Metrics; page unavailable) describes the paper’s own measures—trigger exact match, bag of steps, hallucination rates, and retrieval recall—but does not compare how frequently evaluation metrics are reported. Searches for the claimed frequency and the named text-similarity metrics found no passage addressing that point. (LLM, conf=0.76)
- Not checked — no full text of the sourcecites [21] Peters, 2008, Contour-enhanced meta-analysis funnel plots help distinguish publication bias from other causes of asymmetry, J Clin Epidemiol source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Could not verify
The cited sources that could be checked couldn’t confirm or dispute this claim.
In the paper
“However, these guidelines do not cover RAG applications in clinical settings, emphasizing the need for a more specific guideline.”
§ Improving large language model applications in biomedicine with retrieval-augmented generation: a systematic review, meta-analysis, and clinical development guidelines > Introduction
What the sources say
- Could not verifycites [11] Vasey, 2022, Reporting guideline for the early stage clinical evaluation of decision support systems driven by artificial intelligence: DECIDE-AI, BMJ sourceExperimental — accuracy not yet measured
The Abstract describes a gap in guidance for early clinical evaluation of AI systems, and Table 1 describes DECIDE-AI as a guideline for evaluating AI interventions in live clinical settings. Neither passage establishes whether existing guidelines cover clinical RAG applications. Searches for “RAG” and “retrieval augmented generation” found no passage addressing that question. (LLM, conf=0.75)
- Could not verifycites [12] Mongan, 2020, Checklist for Artificial Intelligence in Medical Imaging (CLAIM): a guide for authors and reviewers, Radiol Artif Intell sourceExperimental — accuracy not yet measured
The Abstract (page unavailable) describes CLAIM as a guideline for AI research in medical imaging, including text analysis. Neither that passage nor the searched passages on RAG, large language models, and clinical applications assess whether existing guidelines cover clinical RAG applications; the absence of RAG from this paper does not establish the broader claim. (LLM, conf=0.74)
- Could not verifycites [13] Martindale, 2024, Concordance of randomised controlled trials for artificial intelligence interventions with the CONSORT-AI reporting guidelines, Nat Commun sourceExperimental — accuracy not yet measured
The Discussion says that medical large language models could be a focus of future CONSORT-AI iterations, but it does not say that existing guidelines exclude RAG applications in clinical settings. Searches for “RAG” and “retrieval augmented generation” found no passage addressing that specific gap or the need for a RAG-specific guideline. (LLM, conf=0.72)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Could not verify
Every source the paper cites for this claim was checked and supports it.
In the paper
“a preliminary study found that GPT-4 Turbo had only a 16% accuracy rate in extracting table data.27”
§ Improving large language model applications in biomedicine with retrieval-augmented generation: a systematic review, meta-analysis, and clinical development guidelines > Discussion
What the sources say
- Supportedcites [27] Kresevic, 2024, Optimization of hepatological clinical guidelines interpretation by large language models: a retrieval augmented generation-based framework, NPJ Digit Med source
The Methods section (Ablation study: customized LLM framework; page not provided) reports preliminary experiments in which the GPT image-conversion process achieved 16.0% accuracy in extracting pertinent table information. The claim paraphrases that finding; the separate 28.0% result concerns accuracy on table-based questions, not extraction of table information. (LLM, conf=0.88)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The sources with a verdict support this claim; the rest weren’t checked yet, have no full text we could read, or their verdicts aren’t rated yet.
In the paper
“Only 3 studies in our review explicitly mentioned extracting text from tables or figures during the pre-retrieval process.25,27,44”
§ Improving large language model applications in biomedicine with retrieval-augmented generation: a systematic review, meta-analysis, and clinical development guidelines > Discussion
What the sources say
- Not checked — no full text of the sourcecites [25] Yazaki, 2024, Emergency patient triage improvement through a retrieval-augmented generation enhanced large-scale language model, Prehosp Emerg Care source
- Supportedcites [27] Kresevic, 2024, Optimization of hepatological clinical guidelines interpretation by large language models: a retrieval augmented generation-based framework, NPJ Digit Med source
The Methods section (Ablation study: customized LLM framework) explicitly says that tables presented as images were manually converted to CSV files, then converted into text-based lists for use as context. The citation identifies this paper as one of the citing review’s three studies; it does not ask this individual paper to establish the review’s count. (LLM, conf=0.91)
- Supportedcites [44] Hewitt, 2024, Large language models as a diagnostic support tool in neuropathology, J Pathol Clin Res source
The Materials and methods section (‘RAG approach’; page unavailable) says the study converted relevant tables to plain text for use by the LLMs. That supports citing it as an example of table-text preparation before retrieval; the count of three studies is the citing review’s own finding, not a claim this paper makes. (LLM, conf=0.88)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Every source the paper cites for this claim was checked and supports it.
In the paper
“Only 1 study in our review, which focused on gastrointestinal radiology diagnosis based on imaging descriptions, compared the cost and response time between LLMs and LLMs with RAG.31”
§ Improving large language model applications in biomedicine with retrieval-augmented generation: a systematic review, meta-analysis, and clinical development guidelines > Discussion
What the sources say
- Supportedcites [31] Rau, 2024, A retrieval-augmented chatbot based on GPT-4 provides appropriate differential diagnosis in gastrointestinal radiology: a proof of concept study, Eur Radiol Exp source
The Methods describe a gastrointestinal imaging-aware GPT-4 chatbot with knowledge retrieval and a generic GPT-4 comparator, tested on descriptions of radiological findings. The Results > Statistical evaluation reports response time and cost per case for both. The statement that this was the only such study in the citing authors’ review is their own characterization, not a finding attributed to this source. (LLM, conf=0.93)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
19 more claims · 1 with their cited sources checked
Create a free account to see all 63 claims and every source verdict.
- It may state two separate claims as one. 20
- Probably not a substantial claim. 24
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Rated major: it would touch a result — a value a result rests on, reported and recomputed p-values on opposite sides of the significance level, two passages that conflict with no sign which is right, or a cited work retracted before this paper was published.
In the paper
“In the subgroup analysis, 12 studies used simple data retrieval strategies (OR 1.30, 95% CI [1.16, 1.45]), while 5 studies used complex data retrieval strategies (OR 1.30, 95% CI [1.07, 1.24]), with no statistically significant difference observed between the 2 approaches.”
§ Improving large language model applications in biomedicine with retrieval-augmented generation: a systematic review, meta-analysis, and clinical development guidelines > Results > Retrieval strategies
Reported 1.3 · recomputed 0.06
Citations
Cites
Bias in meta-analysis detected by a simple, graphical test
Egger, M., Smith, G. D., Schneider, M., & Minder, C. (1997) · BMJ · doi:10.1136/bmj.315.7109.629 · Cited by 56,722
Quantifying heterogeneity in a meta‐analysis
Higgins, J. P. T., & Thompson, S. G. (2002) · Statistics in Medicine · doi:10.1002/sim.1186 · Cited by 36,763
Shamseer, L., Moher, D., Clarke, M., Ghersi, D., Liberati, A., Petticrew, M., Shekelle, P., Stewart, L. A., & the PRISMA-P Group (2015) · BMJ · doi:10.1136/bmj.g7647 · Cited by 13,120
Large language models in medicine
Thirunavukarasu, A. J., Ting, D. S. J., Elangovan, K., Gutierrez, L., Tan, T. F., & Ting, D. S. W. (2023) · Nature Medicine · doi:10.1038/s41591-023-02448-8 · Cited by 3,944
Ayers, J. W., Poliak, A., Dredze, M., Leas, E. C., Zhu, Z., Kelley, J. B., Faix, D. J., Goodman, A. M., Longhurst, C. A., Hogarth, M., et al. (2023) · JAMA Internal Medicine · doi:10.1001/jamainternmed.2023.1838 · Cited by 2,613
Peters, J. L., Sutton, A. J., Jones, D. R., Abrams, K. R., & Rushton, L. (2008) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2007.11.010 · Cited by 2,053 · cited 2×
“Accuracy was the most frequently reported metric, covering Likert scale ratings, match rates, correct treatment percentages, 9 AUC, AUPRC, and F1 scores, as well as text similarity metrics like ROUGE (ROUGE-1, ROUGE-2, ROUGE-L), BLEU, METEOR, and BERTScore, 21 which compared LLM-generated responses to expert-provided…”
A survey on large language model based autonomous agents
Wang, L., Ma, C., Feng, X., Zhang, Z., Yang, H., Zhang, J., Chen, Z., Tang, J., Chen, X., Lin, Y., et al. (2024) · Frontiers of Computer Science · doi:10.1007/s11704-024-40231-1 · Cited by 1,941
A Survey on Large Language Model based Autonomous Agents
Wang, L., Ma, C., Feng, X., Zhang, Z., Yang, H., Zhang, J., Chen, Z., Tang, J., Chen, X., Lin, Y., et al. (2023) · arXiv · doi:10.48550/arxiv.2308.11432 · Cited by 1,925
Checklist for Artificial Intelligence in Medical Imaging (CLAIM): a guide for authors and reviewers
Mongan, J., Moy, L., & Kahn, C. E. (2020) · Radiology: Artificial Intelligence · doi:10.1148/ryai.2020200029 · Cited by 1,443 · cited 1×
“…and LLMs in healthcare, including DECIDE-AI (Developmental and Exploratory Clinical Investigations of DEcision support systems driven by Artificial Intelligence), 11 CLAIM (Checklist for Artificial Intelligence in Medical Imaging), 12 and CONSORT-AI (Consolidated Standards of Reporting Trials-AI).”
(2010) · Paediatric and Perinatal Epidemiology · doi:10.1111/j.1365-3016.2010.01108.x · Cited by 1,443
Medical Records That Guide and Teach
Weed, L. L. (1968) · New England Journal of Medicine · doi:10.1056/nejm196803142781105 · Cited by 996
(2024) · doi:10.1109/access.2024.3365742 · Cited by 934
Retrieval-Augmented Generation for Large Language Models: A Survey
Gao, Y., Xiong, Y., Gao, X., Jia, K., Pan, J., Bi, Y., Dai, Y., Sun, J., Wang, M., & Wang, H. (2023) · arXiv · doi:10.48550/arxiv.2312.10997 · Cited by 876
SMART on FHIR: a standards-based, interoperable apps platform for electronic health records
Mandel, J. C., Kreda, D. A., Mandl, K. D., Kohane, I. S., & Ramoni, R. B. (2016) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocv189 · Cited by 829
Gornik, H. L., Aronow, H. D., Goodney, P. P., Arya, S., Brewster, L. P., Byrd, L., Chandra, V., Drachman, D. E., Eaves, J. M., Ehrman, J. K., et al. (2024) · Circulation · doi:10.1161/cir.0000000000001251 · Cited by 771 · cited 1×
“After integrating retrieved text from the 2024 American College of Cardiology / American Heart Association Guideline for the Management of Lower Extremity Peripheral Artery Disease, 10 the model correctly recommended rivaroxaban.”
Chen, J., Xiao, S., Zhang, P., Luo, K., Lian, D., & Liu, Z. (2024) · Findings of the Association for Computational Linguistics ACL 2024 · doi:10.18653/v1/2024.findings-acl.137 · Cited by 757 · cited 1×
“48,49 5) Explore advanced retrieval strategies, including iterative and graph-based retrieval.”
Vasey, B., Nagendran, M., Campbell, B., Clifton, D. A., Collins, G. S., Denaxas, S., Denniston, A. K., Faes, L., Geerts, B., Ibrahim, M., et al. (2022) · BMJ · doi:10.1136/bmj-2022-070904 · Cited by 665
Retrieval Augmentation Reduces Hallucination in Conversation
Shuster, K., Poff, S., Chen, M., Kiela, D., & Weston, J. (2021) · Findings of the Association for Computational Linguistics: EMNLP 2021 · doi:10.18653/v1/2021.findings-emnlp.320 · Cited by 575
Siren's Song in the AI Ocean: A Survey on Hallucination in Large Language Models
Zhang, Y., Li, Y., Cui, L., Cai, D., Liu, L., Fu, T., Huang, X., Zhao, E., Zhang, Y., Xu, C., et al. (2023) · arXiv · doi:10.48550/arxiv.2309.01219 · Cited by 544
Siren's Song in the AI Ocean: A Survey on Hallucination in Large Language Models
Zhang, Y., Li, Y., Cui, L., Cai, D., Liu, L., Fu, T., Huang, X., Zhao, E., Zhang, Y., Chen, Y., et al. (2025) · Computational Linguistics · doi:10.1162/coli.a.16 · Cited by 534 · cited 1×
“27,41 Citation metrics measured the accuracy of provided references, with valid references considered those that pointed to established publications, guidelines, or research.”
Almanac — Retrieval-Augmented Language Models for Clinical Medicine
Zakka, C., Shad, R., Chaurasia, A., Dalal, A. R., Kim, J. L., Moor, M., Fong, R., Phillips, C., Alexander, K., Ashley, E., et al. (2024) · NEJM AI · doi:10.1056/aioa2300068 · Cited by 424
Tam, T. Y. C., Sivarajkumar, S., Kapoor, S., Stolyar, A. V., Polanska, K., McCarthy, K. R., Osterhoudt, H., Wu, X., Visweswaran, S., Fu, S., et al. (2024) · npj Digital Medicine · doi:10.1038/s41746-024-01258-7 · Cited by 424 · cited 1×
“14 However, these guidelines do not cover RAG applications in clinical settings, emphasizing the need for a more specific guideline.”
A survey on LLM-based multi-agent systems: Workflow, infrastructure, and challenges
Li, X., Wang, S., Zeng, S., Wu, Y., & Yang, Y. (2024) · Vicinagearth · doi:10.1007/s44336-024-00009-2 · Cited by 404 · cited 1×
“57 For example, clinicians and patients have diverse information access needs, some needing to analyze text from a knowledge base, others needing to incorporate structured data from an EHR.”
Using AI-generated suggestions from ChatGPT to optimize clinical decision support
Liu, S., Wright, A. P., Patterson, B. L., Wanderer, J. P., Turer, R. W., Nelson, S. D., McCoy, A. B., Sittig, D. F., & Wright, A. (2023) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocad072 · Cited by 397
Self-citationBenchmarking Retrieval-Augmented Generation for Medicine
Xiong, G., Jin, Q., Lu, Z., & Zhang, A. (2024) · Findings of the Association for Computational Linguistics ACL 2024 · doi:10.18653/v1/2024.findings-acl.372 · Cited by 310
Zaretsky, J., Kim, J. M., Baskharoun, S., Zhao, Y., Austrian, J., Aphinyanaphongs, Y., Gupta, R., Blecker, S. B., & Feldman, J. (2024) · JAMA Network Open · doi:10.1001/jamanetworkopen.2024.0357 · Cited by 303
Kresevic, S., Giuffrè, M., Ajcevic, M., Accardo, A., Crocè, L. S., & Shung, D. L. (2024) · npj Digital Medicine · doi:10.1038/s41746-024-01091-y · Cited by 207
Hallucination is Inevitable: An Innate Limitation of Large Language Models
Xu, Z., Jain, S., & Kankanhalli, M. (2024) · arXiv · doi:10.48550/arxiv.2401.11817 · Cited by 180
From local to global: a graph rag approach to query-focused summarization
Edge, D., Trinh, H., Cheng, N., Bradley, J., Chao, A., Mody, A., Truitt, S., Metropolitansky, D., Ness, R. O., & Larson, J. (2024) · arXiv · doi:10.48550/arxiv.2404.16130 · Cited by 159
Alkhalaf, M., Yu, P., Yin, M., & Deng, C. (2024) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2024.104662 · Cited by 156
Sawarkar, K., Mangal, A., & Solanki, S. R. (2024) · 2024 IEEE 7th International Conference on Multimedia Information Processing and Retrieval (MIPR) · doi:10.1109/mipr62202.2024.00031 · Cited by 145
Jeong, M., Sohn, J., Sung, M., & Kang, J. (2024) · Bioinformatics · doi:10.1093/bioinformatics/btae238 · Cited by 132 · cited 6×
“34 The self-curated datasets averaged 76 questions, ranging from 7 to 314.”
InfluentialEmbi, P. J. (2021) · JAMA Network Open · doi:10.1001/jamanetworkopen.2021.4622 · Cited by 99
Morris, J. H., Soman, K., Akbas, R. E., Zhou, X., Smith, B., Meng, E. C., Huang, C. C., Cerono, G., Schenk, G., Rizk-Jackson, A., et al. (2023) · Bioinformatics · doi:10.1093/bioinformatics/btad080 · Cited by 99 · cited 1×
“32 Two studies used textbooks, such as Harrison's Principles of Internal Medicine , while 3 others utilized electronic health record (EHR) data.”
Reducing hallucination in structured outputs via Retrieval-Augmented Generation
Ayala, O., & Bechard, P. (2024) · Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 6: Industry Track) · doi:10.18653/v1/2024.naacl-industry.19 · Cited by 97
Assessing the Value of ChatGPT for Clinical Decision Support Optimization
Liu, S., Wright, A. P., Patterson, B. L., Wanderer, J. P., Turer, R. W., Nelson, S. D., McCoy, A. B., Sittig, D. F., & Wright, A. (2023) · doi:10.1101/2023.02.21.23286254 · Cited by 90
Self-citationGlicksberg, B. S., Timsina, P., Patel, D., Sawant, A., Vaid, A., Raut, G., Charney, A. W., Apakama, D., Carr, B. G., Freeman, R., et al. (2024) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocae103 · Cited by 82
ChunkRAG: Novel LLM-Chunk Filtering Method for RAG SystemsPreprint
Singh, I. S., Aggarwal, R., Allahverdiyev, I., Taha, M., Akalin, A., Zhu, K., & O'Brien, S. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2410.19572 · Cited by 78
Wang, D., Liang, J., Ye, J., Li, J., Li, J., Zhang, Q., Hu, Q., Pan, C., Wang, D., Liu, Z., et al. (2024) · Journal of Medical Internet Research · doi:10.2196/58041 · Cited by 76
Du, X., Novoa-Laurentiev, J., Plasek, J. M., Chuang, Y.-W., Wang, L., Marshall, G. A., Mueller, S. K., Chang, F., Datta, S., Paek, H., et al. (2024) · eBioMedicine · doi:10.1016/j.ebiom.2024.105401 · Cited by 69
Empowering personalized pharmacogenomics with generative AI solutions
(2024) · doi:10.1093/jamia/ocae039 · Cited by 65
Large Language Models and the Degradation of the Medical Record
McCoy, L. G., Manrai, A. K., & Rodman, A. M. (2024) · New England Journal of Medicine · doi:10.1056/nejmp2405999 · Cited by 61
Martindale, A. P. L., Llewellyn, C. D., de Visser, R. O., Ng, B., Ngai, V., Kale, A. U., di Ruffano, L. F., Golub, R. M., Collins, G. S., Moher, D., et al. (2024) · Nature Communications · doi:10.1038/s41467-024-45355-3 · Cited by 58
Chen, X., Wang, L., You, M., Liu, W., Fu, Y., Xu, J., Zhang, S., Chen, G., Li, K., & Li, J. (2024) · Journal of Medical Internet Research · doi:10.2196/58158 · Cited by 48
Rare disease diagnosis using knowledge guided retrieval augmentation for ChatGPT
Zelin, C., Chung, W. K., Jeanne, M., Zhang, G., & Weng, C. (2024) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2024.104702 · Cited by 47
RDguru: A Conversational Intelligent Agent for Rare Diseases
Yang, J., Shu, L., Duan, H., & Li, H. (2024) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2024.3464555 · Cited by 47
Kwon, T., Ong, K. T.-I., Kang, D., Moon, S., Lee, J. R., Hwang, D., Sohn, B., Sim, Y., Lee, D., & Yeo, J. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i16.29802 · Cited by 47
Long, C., Subburam, D., Lowe, K., dos Santos, A., Zhang, J., Hwang, S., Saduka, N., Horev, Y., Su, T., Côté, D. W. J., et al. (2024) · Otolaryngology–Head and Neck Surgery · doi:10.1002/ohn.864 · Cited by 46
Rau, S., Rau, A., Nattenmüller, J., Fink, A., Bamberg, F., Reisert, M., & Russe, M. F. (2024) · European Radiology Experimental · doi:10.1186/s41747-024-00457-x · Cited by 33
Cerf, V. G. (2023) · Communications of the ACM · doi:10.1145/3606337 · Cited by 31
Performance of Publicly Available Large Language Models on Internal Medicine Board-style Questions
Tarabanis, C., Zahid, S., Mamalis, M. E., Zhang, K., Kalampokis, E., & Jankelson, L. (2024) · PLOS Digital Health · doi:10.1371/journal.pdig.0000604 · Cited by 30
Wu, Z., Hasan, A., Wu, J., Kim, Y., Cheung, J. P. Y., Zhang, T. G., & Wu, H. (2024) · Clinical Natural Language Processing Workshop (ClinicalNLP) · doi:10.18653/v1/2024.clinicalnlp-1.33 · Cited by 29
Yazaki, M., Maki, S., Furuya, T., Inoue, K., Nagai, K., Nagashima, Y., Maruyama, J., Toki, Y., Kitagawa, K., Iwata, S., et al. (2025) · Prehospital Emergency Care · doi:10.1080/10903127.2024.2374400 · Cited by 29
Large language models as a diagnostic support tool in neuropathology
Hewitt, K. J., Wiest, I. C., Carrero, Z. I., Bejan, L., Millner, T. O., Brandner, S., & Kather, J. N. (2024) · The Journal of Pathology: Clinical Research · doi:10.1002/2056-4538.70009 · Cited by 27
Investigating the Factual Knowledge Boundary of Large Language Models with Retrieval Augmentation
Ren, R., Wang, Y., Qu, Y., Zhao, W. X., Liu, J., Tian, H., Wu, H., Wen, J.-R., & Wang, H. (2023) · arXiv · doi:10.48550/arxiv.2307.11019 · Cited by 21 · cited 1×
“59 However, the integration of external knowledge into LLM reasoning is typically limited to providing the retrieved data as additional context for the LLM’s query during generation.”
Malik, S., Kharel, H., Dahiya, D. S., Ali, H., Blaney, H. L., Singh, A. D., Dhar, J., Perisetti, A., Facciorusso, A., Chandan, S., et al. (2024) · Annals of Gastroenterology · doi:10.20524/aog.2024.0907 · Cited by 18
Research on Chunking Algorithms of Data De-duplication
(2012) · doi:10.1007/978-3-642-31698-2_144 · Cited by 14
Kreimeyer, K., Canzoniero, J. V., Fatteh, M., Anagnostou, V., & Botsis, T. (2024) · Studies in Health Technology and Informatics · doi:10.3233/shti240575 · Cited by 8 · cited 1×
“In clinical decision making, RAG has supported personalized treatment, 23,24 emergency triage, 25 and disease management.”
How Does Knowledge Selection Help Retrieval Augmented Generation?
Li, X., & Ouyang, J. (2024) · arXiv · doi:10.48550/arxiv.2410.13258 · Cited by 6
Research works printed but not linked: 2
- Lefebvre, Chapter 4: searching for and selecting studies, Cochrane Handbook for Systematic Reviews of Interventions Version, Vol., 6
- Chapter 3 Effect Sizes
Cited by
Liu, S., Wright, A. P., McCoy, A. B., Huang, S. S., Steitz, B., & Wright, A. (2025) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocaf059 · Cited by 44
“20,21 24 Integrating external knowledge, for example, a nurse triage book with over 200 protocols for common concerns and symptoms, might further support the emergency tri-age task.”
Self-citationThe perils of politeness: how large language models may amplify medical misinformation
Rosen, K. L., Sui, M., Heydari, K., Enichen, E. J., & Kvedar, J. C. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-02135-7 · Cited by 41
Xu, X., & Sankar, R. (2025) · Information · doi:10.3390/info16100894 · Cited by 20
“Prompt engineering and structured integration of retrieved content are also critical to ensure coherence and factual alignment during response synthesis [42].”
InfluentialNanua, S., Steward, R., Neely, B., Datto, M., & Youens, K. E. (2025) · Journal of Pathology Informatics · doi:10.1016/j.jpi.2025.100520 · Cited by 25
Reason, T., Klijn, S. L., Rawlinson, W. D., Benbow, E., Langham, J., Teitsson, S., Johannesen, K. M., & Malcolm, B. (2025) · PharmacoEconomics - Open · doi:10.1007/s41669-025-00580-4 · Cited by 27
Metadata
- Authors
- Liu, S., McCoy, A. B., & Wright, A.
- Affiliations
- Vanderbilt University (US)
- Vanderbilt University Medical Center (US)
- Corresponding author
- Liu, Siru
- Venue
- Journal of the American Medical Informatics Association
- Publisher
- Oxford University Press (OUP)
- Published
- 2025-01-15
- Volume
- 32
- Issue
- 4
- Pages
- 605-615
- ISSN
- 1067-5027 (ISSN-L)
- 1527-974X
- PMC
- PMC12005634
- OpenAlex
- W4406421570
- Cited by
- 200
- References
- 49
- Citations by year
- 2026: 138 · 2025: 60
- FWCI
- 11.70
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Journal of the American Medical Informatics Association (publishedVersion, cc-by-nc)
- PubMed Central (publishedVersion, cc-by-nc)
- PubMed (publishedVersion)
- Funders
- National Institutes of Health (R00LM014097-02, R01LM013995-01)
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
- Keywords
- retrieval-augmented generation, large language models, biomedical applications, medical question answering, electronic health record integration, LLM agents
- MeSH terms
- Large Language Models, Humans, Natural Language Processing, Programming Languages, Information Storage and Retrieval
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.