The DOI matches its Crossref record; 16 of the 16 references it cites were found in a registry or an index.
Applying generative AI with retrieval augmented generation to summarize and extract key clinical information from electronic health records
Open accessAbstract
BACKGROUND: Malnutrition is a prevalent issue in aged care facilities (RACFs), leading to adverse health outcomes. The ability to efficiently extract key clinical information from a large volume of data in electronic health records (EHR) can improve understanding about the extent of the problem and developing effective interventions. This research aimed to test the efficacy of zero-shot prompt engineering applied to generative artificial intelligence (AI) models on their own and in combination with retrieval augmented generation (RAG), for the automating tasks of summarizing both structured and unstructured data in EHR and extracting important malnutrition information. METHODOLOGY: We utilized Llama 2 13B model with zero-shot prompting. The dataset comprises unstructured and structured EHRs related to malnutrition management in 40 Australian RACFs. We employed zero-shot learning to the model alone first, then combined it with RAG to accomplish two tasks: generate structured summaries about the nutritional status of a client and extract key information about malnutrition risk factors. We utilized 25 notes in the first task and 1,399 in the second task. We evaluated the model's output of each task manually against a gold standard dataset. RESULT: The evaluation outcomes indicated that zero-shot learning applied to generative AI model is highly effective in summarizing and extracting information about nutritional status of RACFs' clients. The generated summaries provided concise and accurate representation of the original data with an overall accuracy of 93.25%. The addition of RAG improved the summarization process, leading to a 6% increase and achieving an accuracy of 99.25%. The model also proved its capability in extracting risk factors with an accuracy of 90%. However, adding RAG did not further improve accuracy in this task. Overall, the model has shown a robust performance when information was explicitly stated in the notes; however, it could encounter hallucination limitations, particularly when details were not explicitly provided. CONCLUSION: This study demonstrates the high performance and limitations of applying zero-shot learning to generative AI models to automatic generation of structured summarization of EHRs data and extracting key clinical information. The inclusion of the RAG approach improved the model performance and mitigated the hallucination problem.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
RAG supports clinical summarization by integrating longitudinal patient data.
“RAG supports clinical summarization by integrating longitudinal patient data [63].”
From Retrieval-Augmented Generation (RAG) in Healthcare: A Comprehensive Review (2025) · cites this paper as [63]
Citations
Cites
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
Transformers: State-of-the-Art Natural Language Processing
Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., et al. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations · doi:10.18653/v1/2020.emnlp-demos.6 · Cited by 8,741
Large Language Models Encode Clinical Knowledge
Singhal, K., Azizi, S., Tu, T., Mahdavi, S. S., Wei, J., Chung, H. W., Scales, N., Tanwani, A., Cole-Lewis, H., Pfohl, S., et al. (2022) · arXiv · doi:10.48550/arxiv.2212.13138 · Cited by 4,083
QLoRA: Efficient Finetuning of Quantized LLMs
Dettmers, T., Pagnoni, A., Holtzman, A., & Zettlemoyer, L. (2023) · arXiv · doi:10.48550/arxiv.2305.14314 · Cited by 3,832
Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., Küttler, H., Lewis, M., Yih, W.-T., Rocktäschel, T., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.11401 · Cited by 3,101
Artificial hallucinations in ChatGPT: implications in scientific writing
Alkaissi, H., & McFarlane, S. I. (2023) · Cureus · doi:10.7759/cureus.35179 · Cited by 1,540
On Faithfulness and Factuality in Abstractive Summarization
Maynez, J., Narayan, S., Bohnet, B., & McDonald, R. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.173 · Cited by 1,185
Adapted large language models can outperform medical experts in clinical text summarization
Van Veen, D., Van Uden, C., Blankemeier, L., Delbrouck, J.-B., Aali, A., Bluethgen, C., Pareek, A., Polacin, M., Reis, E. P., Seehofnerová, A., et al. (2024) · Nature Medicine · doi:10.1038/s41591-024-02855-5 · Cited by 891
Retrieval-Augmented Generation for Large Language Models: A Survey
Gao, Y., Xiong, Y., Gao, X., Jia, K., Pan, J., Bi, Y., Dai, Y., Sun, J., Wang, M., & Wang, H. (2023) · arXiv · doi:10.48550/arxiv.2312.10997 · Cited by 876
Towards Reasoning in Large Language Models: A Survey
Huang, J., & Chang, K. C.-C. (2023) · Findings of the Association for Computational Linguistics: ACL 2023 · doi:10.18653/v1/2023.findings-acl.67 · Cited by 428
Bang, Y., Cahyawijaya, S., Lee, N., Dai, W., Su, D., Wilie, B., Lovenia, H., Ji, Z., Yu, T., Chung, W., et al. (2023) · arXiv · doi:10.48550/arxiv.2302.04023 · Cited by 337
Malnutrition Screening and Assessment
Serón-Arbeloa, C., Labarta-Monzón, L., Puzo-Foncillas, J., Mallor-Bonet, T., Lafita-López, A., Bueno-Vidales, N., & Montoro-Huguet, M. (2022) · doi:10.20944/preprints202205.0071.v1 · Cited by 312
Generation-Augmented Retrieval for Open-Domain Question Answering
Mao, Y., He, P., Liu, X., Shen, Y., Gao, J., Han, J., & Chen, W. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.316 · Cited by 146
A Survey of Hallucination in Large Foundation Models
Rawte, V., Sheth, A., & Das, A. (2023) · arXiv · doi:10.48550/arxiv.2309.05922 · Cited by 133
Generation-Augmented Retrieval for Open-domain Question Answering
Mao, Y., He, P., Liu, X., Shen, Y., Gao, J., Han, J., & Chen, W. (2020) · arXiv · doi:10.48550/arxiv.2009.08553 · Cited by 31
Bhate, N. J., Mittal, A., He, Z., & Luo, X. (2023) · 2023 IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata59044.2023.10386811 · Cited by 18
Flagged first, then by citations
Research works printed but not linked: 45
- Correia, 2003, The impact of malnutrition on morbidity, mortality, length of hospital stay and costs evaluated through a multivariate model analysis, Clin. Nutr., 22, 235, 10.1016/S0261-5614(02)00215-7 · DOI
- Stratton, 2006, ‘Malnutrition Universal Screening Tool’ predicts mortality and length of hospital stay in acutely ill elderly, Br. J. Nutr., 95, 325, 10.1079/BJN20051622 · DOI
- Ahmed, 2010, Assessment and management of nutrition in older people and its importance to health, Clin. Interv. Aging, 5, 207
- Agarwal, 2013, Malnutrition in the elderly: A narrative review, Maturitas, 76, 296, 10.1016/j.maturitas.2013.07.013 · DOI · Paper page
- Kellett, 2012, Malnutrition prevalence in aged care residences, Nutr. Diet., 69, 72
- Flint, 2021, Mealtime care and dietary intake in older psychiatric hospital inpatient: A multiple case study, J. Adv. Nurs., 77, 1490, 10.1111/jan.14728 · DOI
- Fashho, 2020, Investigating the prevalence of malnutrition, frailty and physical disability and the association between them amongst older care home residents, Clin. Nutr. ESPEN, 40, 231, 10.1016/j.clnesp.2020.09.014 · DOI
- Sahin, 2016, Prevalence of anemia and malnutrition and their association in elderly nursing home residents, Aging Clin. Exp. Res., 28, 857, 10.1007/s40520-015-0490-5 · DOI
- Robb, 2017, Malnutrition in the elderly residing in long-term care facilities: A cross sectional survey using the Mini Nutritional Assessment (MNA®) screening tool, South Afr. J. Clin. Nutr., 30, 34, 10.1080/16070658.2016.1248062 · DOI
- K. Lind et al., “Measuring the prevalence of 60 health conditions in older Australians in residential aged care with electronic health records: a retrospective dynamic cohort study,” pp. 1-9, 2020, doi: DOI: 10.21203/rs.2.21384/v1. · DOI
- Kharrazi, 2018, The value of unstructured electronic health record data in geriatric syndrome case identification, J. Am. Geriatr. Soc., 66, 1499, 10.1111/jgs.15411 · DOI · Paper page
- Murdoch, 2013, The inevitable application of big data to health care, JAMA – J. Am. Med. Assoc., 309, 1351, 10.1001/jama.2013.393 · DOI
- Kong, 2019, Managing unstructured big data in healthcare system, Healthcare Inform. Res., 25, 1, 10.4258/hir.2019.25.1.1 · DOI · Paper page
- Mustafa, 2021, Automated Machine Learning for Healthcare and Clinical Notes Analysis, Computers, 10, pp, 10.3390/computers10020024 · DOI
- A. Laxmisan et al., “Clinical Summarization Capabilities of Commercially-available and Internally-developed Electronic Health Records,” (in En), Appl. Clin. Inform., vol. 03, no. 01, pp. 80-93, 2017/12/16 2012, doi: DOI: 10.1055/s-0037-1618556. · DOI
- G. Adams et al., “What’s in a summary? laying the groundwork for advances in hospital-course summarization,” in Proceedings of the conference. Association for Computational Linguistics. North American Chapter. Meeting, 2021, vol. 2021: NIH Public Access, p. 4794. · DOI
- Hirsch, 2014, HARVEST, a longitudinal patient record summarizer, J. Am. Med. Informat. Assoc., 22, 263, 10.1136/amiajnl-2014-002945 · DOI · Paper page
- Favaro-Moreira, 2016, Risk Factors for Malnutrition in Older Adults: A Systematic Review of the Literature Based on Longitudinal Data, Adv. Nutr., 7, 507, 10.3945/an.115.011254 · DOI
- Steinkamp, 2020, Task definition, annotated dataset, and supervised natural language processing models for symptom extraction from unstructured clinical notes, J. Biomed. Informat., 102, 10.1016/j.jbi.2019.103354 · DOI
- Meijers, 2008, Malnutrition prevalence in The Netherlands: results of the Annual Dutch National Prevalence Measurement of Care Problems, Br. J. Nutr., 101, 417, 10.1017/S0007114508998317 · DOI
- Barker, 2011, Hospital Malnutrition: Prevalence, Identification and Impact on Patients and the Healthcare System, Int. J. Environ. Res. Public Health, 8, 514, 10.3390/ijerph8020514 · DOI · Paper page
- Song, 2023, Uncovering hidden trends: identifying time trajectories in risk factors documented in clinical notes and predicting hospitalizations and emergency department visits during home health care, J. Am. Med. Inform. Assoc., p. ocad101
- Topaz, 2020, Home health care clinical notes predict patient hospitalization and emergency department visits, Nurs. Res., 69, 448, 10.1097/NNR.0000000000000470 · DOI
- Hu, 2024, Improving large language models for clinical named entity recognition via prompt engineering, J. Am. Med. Inform. Assoc., 10.1093/jamia/ocad259 · DOI · Paper page
- Pivovarov, 2015, Automated methods for the summarization of electronic health records, J. Am. Med. Inform. Assoc., 22, 938, 10.1093/jamia/ocv032 · DOI
- Li, 2022, Neural Natural Language Processing for unstructured data in electronic health records: a review, Comput. Sci. Rev., 46, 10.1016/j.cosrev.2022.100511 · DOI
- Wang, 2018, Clinical information extraction applications: a literature review, J. Biomed. Inform., 77, 34, 10.1016/j.jbi.2017.11.011 · DOI · Paper page
- J. A. Banan, F. Chia Aziz, and H. Mzhda Yasin, “A Review of the Role and Challenges of Big Data in Healthcare Informatics and Analytics,” (in English), Computational Intelligence and Neuroscience : CIN, vol. 2022, 2022 2022, doi: DOI: 10.1155/2022/5317760. · DOI
- Yu, 2023, Leveraging generative ai and large language models: a comprehensive roadmap for healthcare integration, Healthcare, 11, pp, 10.3390/healthcare11202776 · DOI
- D. Van Veen et al., “Clinical Text Summarization: Adapting Large Language Models Can Outperform Human Experts,” p. arXiv:2309.07430doi: DOI: 10.48550/arXiv.2309.07430. · DOI
- Liu, 2023, Utility of ChatGPT in Clinical Practice, J. Med. Internet Res., 25, e48568, 10.2196/48568 · DOI · Paper page
- Hugo Touvron et al., “Llama 2- Open Foundation and Fine-Tuned Chat Models,” arXiv, vol. 2307, 2023, doi: DOI: 10.48550/arXiv.2307.09288. · DOI · Paper page
- V. K. Cody Bumgardner et al., “Local Large Language Models for Complex Structured Medical Tasks,” p. arXiv:2308.01727doi: DOI: 10.48550/arXiv.2308.01727. · DOI
- A. Toma et al., “Clinical Camel: An Open Expert-Level Medical Language Model with Dialogue-Based Knowledge Encoding,” p. arXiv:2305.12031doi: DOI: 10.48550/arXiv.2305.12031. · DOI · Paper page
- H. Wang et al., “DRG-LLaMA : Tuning LLaMA Model to Predict Diagnosis-related Group for Hospitalized Patients,” p. arXiv:2309.12625doi: DOI: 10.48550/arXiv.2309.12625. · DOI
- W. E. Thompson et al., “Large Language Models with Retrieval-Augmented Generation for Zero-Shot Disease Phenotyping,” arXiv e-prints, p. arXiv:2312.06457, 2023, doi: DOI: 10.48550/arXiv.2312.06457. · DOI
- M. Alkhalaf et al., “Malnutrition and its contributing factors for older people living in residential aged care facilities: Insights from natural language processing of aged care records,” Technology and Health Care, vol. Preprint, pp. 1-12, 2023, doi: DOI: 10.3233/THC-230229. · DOI
- X. Liu et al., “Prompting Frameworks for Large Language Models: A Survey,” arXiv e-prints, p. arXiv:2311.12785, 2023, doi: DOI: 10.48550/arXiv.2311.12785. · DOI · Paper page
- H. Touvron et al., “Llama 2: Open Foundation and Fine-Tuned Chat Models,” p. arXiv:2307.09288doi: DOI: 10.48550/arXiv.2307.09288. · DOI · Paper page
- J. Kaddour et al., “Challenges and Applications of Large Language Models,” p. arXiv:2307.10169doi: DOI: 10.48550/arXiv.2307.10169. · DOI · Paper page
- J. Wang et al., “NoteChat: A Dataset of Synthetic Doctor-Patient Conversations Conditioned on Clinical Notes,” p. arXiv:2310.15959doi: DOI: 10.48550/arXiv.2310.15959. · DOI
- Y. Hu et al., “Zero-shot Clinical Entity Recognition using ChatGPT,” p. arXiv:2303.16416doi: DOI: 10.48550/arXiv.2303.16416. · DOI
- Wei, 2022, Chain-of-thought prompting elicits reasoning in large language models, Adv. Neural Inf. Process. Syst., 35, 24824
- M. Mosbach et al., “Few-shot Fine-tuning vs. In-context Learning: A Fair Comparison and Evaluation,” arXiv preprint arXiv:2305.16938, 2023. · DOI
- Wang, 2023, Potential for GPT technology to optimize future clinical decision-making using retrieval-augmented generation, Ann. Biomed. Eng., 10.5772/intechopen.100663 · DOI
Cited by
Sorted by citations
Retrieval augmented generation for large language models in healthcare: A systematic review
Amugongo, L. M., Mascheroni, P., Brooks, S., Doering, S., & Seidel, J. (2025) · PLOS Digital Health · doi:10.1371/journal.pdig.0000877 · Cited by 267
Liu, S., McCoy, A. B., & Wright, A. (2025) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocaf008 · Cited by 200
Retrieval-Augmented Generation (RAG) in Healthcare: A Comprehensive Review
Neha, F., Bhati, D., & Shukla, D. K. (2025) · AI · doi:10.3390/ai6090226 · Cited by 117
“RAG supports clinical summarization by integrating longitudinal patient data [63].”
Scientific evidence for clinical text summarization using large language models: scoping review
Bednarczyk, L., Reichenpfader, D., Gaudet-Blavignac, C., Ette, A. K., Zaghir, J., Zheng, Y., Bensahla, A., Bjelogrlic, M., & Lovis, C. (2025) · Journal of Medical Internet Research · doi:10.2196/68998 · Cited by 83
Clay, B., Bergman, H. I., Salim, S., Pergola, G., Shalhoub, J., & Davies, A. H. (2025) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2025.109808 · Cited by 41
Retrieval-Augmented Generation: Advancing personalized care and research in oncology
Zarfati, M., Soffer, S., Nadkarni, G. N., & Klang, E. (2025) · European Journal of Cancer · doi:10.1016/j.ejca.2025.115341 · Cited by 20
ENTAgents: AI Agents for Complex Knowledge Otolaryngology
Chan, T. K., & Dinh, N.-D. (2025) · doi:10.1101/2025.01.01.25319863 · Cited by 9
Hooshiar, M. H. (2026) · The Journal of Prosthetic Dentistry · doi:10.1016/j.prosdent.2025.11.005 · Cited by 5
Enhancing Medical Question Answering with LLMs via a Hybrid Retrieval-Augmented Generation Framework
Aljohani, B., & Alsanoosy, T. (2026) · Information · doi:10.3390/info17020133 · Cited by 3
Metadata
- Authors
- Alkhalaf, M., Yu, P., Yin, M., & Deng, C.
- Affiliations
- Qassim University (SA)
- University of Wollongong (AU)
- Corresponding author
- Yu, Ping
- Venue
- Journal of Biomedical Informatics
- Publisher
- Elsevier BV
- Published
- 2024-08
- Volume
- 156
- Pages
- 104662
- ISSN
- 1532-0464 (ISSN-L)
- OpenAlex
- W4399660853
- Cited by
- 156
- References
- 16
- Citations by year
- 2026: 66 · 2025: 81 · 2024: 9
- FWCI
- 23.02
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Journal of Biomedical Informatics (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Research Online (University of Wollongong) (submittedVersion)
- Figshare (submittedVersion)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Keywords
- retrieval-augmented generation, generative AI, electronic health records, clinical summarization, clinical information extraction, malnutrition, aged care facilities, nutritional assessment, risk factors for malnutrition, Llama 2, hallucinations
- MeSH terms
- Algorithms, Artificial Intelligence, Australia, Humans, Information Storage and Retrieval, Malnutrition, Electronic Health Records
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
