Retrieval augmented generation for 10 large language models and its generalizability in assessing medical fitness
Open accessPublished version of a preprint
Abstract
Large Language Models (LLMs) hold promise for medical applications but often lack domain-specific expertise. Retrieval Augmented Generation (RAG) enables customization by integrating specialized knowledge. This study assessed the accuracy, consistency, and safety of LLM-RAG models in determining surgical fitness and delivering preoperative instructions using 35 local and 23 international guidelines. Ten LLMs (e.g., GPT3.5, GPT4, GPT4o, Gemini, Llama2, and Llama3, Claude) were tested across 14 clinical scenarios. A total of 3234 responses were generated and compared to 448 human-generated answers. The GPT4 LLM-RAG model with international guidelines generated answers within 20 s and achieved the highest accuracy, which was significantly better than human-generated responses (96.4% vs. 86.6%, p = 0.016). Additionally, the model exhibited an absence of hallucinations and produced more consistent output than humans. This study underscores the potential of GPT-4-based LLM-RAG models to deliver highly accurate, efficient, and consistent preoperative assessments.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Retrieval-augmented generation reduces hallucination rates in medical question-answering tasks.
“reducing hallucination rates in medical question-answering (QA) tasks [7–9].”
From The economics of accuracy for medical reasoning with large language models (2026) · cites this paper as [7]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Large language models in medicine
Thirunavukarasu, A. J., Ting, D. S. J., Elangovan, K., Gutierrez, L., Tan, T. F., & Ting, D. S. W. (2023) · Nature Medicine · doi:10.1038/s41591-023-02448-8 · Cited by 3,944 · see citing works
Self-citationThe role of artificial intelligence in achieving the Sustainable Development Goals
Vinuesa, R., Azizpour, H., Leite, I., Balaam, M., Dignum, V., Domisch, S., Felländer, A., Langhans, S. D., Tegmark, M., & Fuso Nerini, F. (2020) · Nature Communications · doi:10.1038/s41467-019-14108-y · Cited by 3,294 · see citing works
Artificial intelligence for sustainability: Challenges, opportunities, and a research agenda
Nishant, R., Kennedy, M., & Corbett, J. (2020) · International Journal of Information Management · doi:10.1016/j.ijinfomgt.2020.102104 · Cited by 1,203 · see citing works
Prompt Engineering as an Important Emerging Skill for Medical Professionals: Tutorial
Meskó, B. (2023) · Journal of Medical Internet Research · doi:10.2196/50638 · Cited by 709 · see citing works
Case cancellations on the day of surgery: an investigation in an Australian paediatric hospital
Haana, V., Sethuraman, K., Stephens, L., et al. (2009) · ANZ Journal of Surgery · doi:10.1111/j.1445-2197.2009.05019.x · Cited by 88 · see citing works
Pal, S., Bhattacharya, M., Lee, S., & Chakraborty, C. (2023) · Annals of Biomedical Engineering · doi:10.1007/s10439-023-03306-x · Cited by 87 · see citing works
Koushan, M., Wood, L. C., & Greatbanks, R. (2021) · International Journal for Quality in Health Care · doi:10.1093/intqhc/mzab092 · Cited by 82 · see citing works
The Role of the Nurse and the Preoperative Assessment in Patient Transitions
Malley, A., Kenner, C. A., Kim, T., & Blakeney, B. (2015) · AORN Journal · doi:10.1016/j.aorn.2015.06.004 · Cited by 73 · see citing works
Reasons for cancellation of cases on the day of surgery-a prospective study.
Garg, R. K., Bhalotra, A. R., Bhadoria, P., et al. (2009) · PubMed · Cited by 70 · see citing works
Nicholson, A., Coldwell, C. H., Lewis, S. R., & Smith, A. F. (2013) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd010160.pub2 · Cited by 42 · see citing works
Almanac: Retrieval-Augmented Language Models for Clinical MedicinePreprint
Hiesinger, W., Zakka, C., Chaurasia, A. R., Shad, R., Dalal, A. R., Kim, J. L., Moor, M., Alexander, K. M., Ashley, E. A., Boyd, J. H., et al. (2023) · Research Square · doi:10.21203/rs.3.rs-2883198/v1 · Cited by 37 · see citing works
Lee, S.-W., & Choi, W.-J. (2023) · Anesthesia and Pain Medicine · doi:10.17085/apm.23056 · Cited by 36 · see citing works
Modifiable risk factors for patients undergoing lung cancer surgery and their optimization: a review
Gagné, S., & McIsaac, D. I. (2018) · Journal of Thoracic Disease · doi:10.21037/jtd.2018.10.04 · Cited by 28 · see citing works
Lim, D. Y. Z., Ke, Y. H., Sng, G. G. R., Tung, J. Y. M., Chai, J. X., & Abdullah, H. R. (2023) · British Journal of Anaesthesia · doi:10.1016/j.bja.2023.06.052 · Cited by 24 · see citing works
Self-citationLiu, S., Lu, X., Jiang, M., et al. (2021) · Annals of Translational Medicine · doi:10.21037/atm-21-4665 · Cited by 20 · see citing works
Naderi-Boldaji, V., Banifatemi, M., Zand, F., et al. (2023) · Patient Safety in Surgery · doi:10.1186/s13037-023-00377-6 · Cited by 16 · see citing works
Improving preoperative medication compliance with standardized instructions.
Pfeifer, K. J., Slawski, B. A., Am, M., et al. (2016) · PubMed · Cited by 14 · see citing works
Tan, T. F., Elangovan, K., Ong, J. C. L., Lee, A., Shah, N. H., Sung, J. J. �., Wong, T. Y., Lan, X., Liu, N., Wang, H., et al. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.5029562 · Cited by 8 · see citing works
Self-citation2019 Update in perioperative cardiovascular medicine
Cohn, S. L., Grant, P., & Slawski, B. A. (2019) · Cleveland Clinic Journal of Medicine · doi:10.3949/ccjm.86a.19077 · Cited by 4 · see citing works
Flagged first, then by citations
Research works printed but not linked: 3
- [14]Yang, R. et al. Integrating UMLS Knowledge into Large Language Models for Medical Question Answering. arXiv [cs.CL] (2023).
- [19]Hinton, G., Vinyals, O. & Dean, J. Distilling the knowledge in a neural network. arXiv [stat.ML] (2015).
- [28]Gemini Team et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv [cs.CL] (2024).
Cited by
Sorted by citations
Retrieval augmented generation for large language models in healthcare: A systematic review
Amugongo, L. M., Mascheroni, P., Brooks, S., Doering, S., & Seidel, J. (2025) · PLOS Digital Health · doi:10.1371/journal.pdig.0000877 · Cited by 267 · see citing works
Innovating global regulatory frameworks for generative AI in medical devices is an urgent priority
Ong, J. C. L., Ning, Y., Liu, M., Ma, Y., Zhao, L., Singh, K., Chang, R. T., Vogel, S., Lim, J. C. W., Tan, I. S. K., et al. (2026) · npj Digital Medicine · doi:10.1038/s41746-026-02552-2 · Cited by 7 · see citing works
Self-citationThe economics of accuracy for medical reasoning with large language models
Bhattacharyya, K., & Kamabattula, S. (2026) · PLOS Digital Health · doi:10.1371/journal.pdig.0001182
“First, retrieval-augmented generation (RAG) enables models to access external knowledge at inference time, sepa-rating factual knowledge from parametric memory and reducing hallucination rates in medical question-answering (QA) tasks [7–9].”
- Authors
- Ke, Y. H., Jin, L., Elangovan, K., Abdullah, H. R., Liu, N., Sia, A. T. H., Soh, C. R., Tung, J. Y. M., Ong, J. C. L., Kuo, C.-F., Wu, S.-C., Kovacheva, V. P., & Ting, D. S. W.
- Affiliations
- Singapore General Hospital (SG)
- SingHealth (SG)
- Duke-NUS Medical School (SG)
- KK Women's and Children's Hospital (SG)
- Chang Gung University (TW)
- Brigham and Women's Hospital (US)
- Corresponding author
- Ting, Daniel Shu Wei
- ORCID
- Venue
- npj Digital Medicine
- Published
- 2025-04-05
- Volume
- 8
- Issue
- 1
- Pages
- 187-187
- ISSN
- 2398-6352 (ISSN-L)
- PMC
- PMC11971376
- OpenAlex
- W4409157497
- Cited by
- 151
- References
- 19
- Citations by year
- 2026: 103 · 2025: 48
- FWCI
- 36.35
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- npj Digital Medicine (publishedVersion, cc-by-nc-nd)
- Research Square (acceptedVersion, cc-by)
- SSRN Electronic Journal (acceptedVersion)
- Europe PMC (PubMed Central) (publishedVersion, cc-by-nc-nd)
- PubMed Central (publishedVersion, cc-by-nc-nd)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Funders
- Singapore General Hospital
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Keywords
- retrieval-augmented generation, large language models, preoperative assessment, preoperative instructions, GPT-4, clinical guidelines, hallucinations, model accuracy, consistency, generalizability, clinical decision support
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
