Med-HEAL: Analyzing and Mitigating Hallucinations in Medical LLMs with Hallucination-Aware In-Context Learning
Open accessAbstract
Hallucinations in medical large language models (LLMs) pose serious risks for clinical decision support, particularly when models must reason over complex electronic health records (EHRs). However, existing benchmarks often lack a realistic clinical context and provide limited insight into how hallucinations can be mitigated in practice. We introduce Med-HEAL, a framework for systematically identifying, analyzing, and mitigating hallucinations in medical LLMs using clinically grounded data. Building on the EHRNoteQA benchmark derived from MIMIC-IV discharge summaries, we construct a hallucination dataset by evaluating BioMistral-7B on open-ended clinical question answering tasks. Model outputs are labeled through a dual evaluation pipeline that combines LLM-as-a-Judge assessment (GPT-4o) with human auditing by medical student reviewers, producing correctness judgments and annotations of reasoning errors via a custom web-based evaluation system. We then leverage this dataset to investigate mitigation strategies: a self-critique pipeline, in which the test model reviews its own answers to detect potential errors and regenerates responses for flagged cases, and retrieval-augmented in-context learning (RA-ICL), which exposes the model to hallucinated and corrected examples. Experiments across five open-source LLMs—BioMistral, Llama-3.1, DeepSeek, Qwen2.5, and Qwen3, show that the self-critique strategy improves accuracy for three of five models (p < 0.05) without requiring parame…
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Cites
Overcoming catastrophic forgetting in neural networks
Kirkpatrick, J., Pascanu, R., Rabinowitz, N., Veness, J., Desjardins, G., Rusu, A. A., Milan, K., Quan, J., Ramalho, T., Grabska-Barwinska, A., et al. (2017) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1611835114 · Cited by 7,899 · cited 1× · see citing works
“Second, repeated fine-tuning may introduce catastrophic forgetting [8, 12] or degrade general reasoning ability, potentially creating new forms of hallucination [7].”
Large language models encode clinical knowledge
Singhal, K., Azizi, S., Tu, T., Mahdavi, S. S., Wei, J., Chung, H. W., Scales, N., Tanwani, A., Cole-Lewis, H., Pfohl, S., et al. (2023) · Nature · doi:10.1038/s41586-023-06291-2 · Cited by 4,083 · see citing works
BioGPT: generative pre-trained transformer for biomedical text generation and mining
Luo, R., Sun, L., Xia, Y., Qin, T., Zhang, S., Poon, H., & Liu, T.-Y. (2022) · Briefings in Bioinformatics · doi:10.1093/bib/bbac409 · Cited by 1,150 · see citing works
An Empirical Investigation of Catastrophic Forgetting in Gradient-Based Neural Networks
Goodfellow, I. J., Mirza, M., Xiao, D., Courville, A., & Bengio, Y. (2013) · arXiv · doi:10.48550/arxiv.1312.6211 · Cited by 1,089 · see citing works
ESTIMATING PREVALENCE FROM THE RESULTS OF A SCREENING TEST
ROGAN, W. J., & GLADEN, B. (1978) · American Journal of Epidemiology · doi:10.1093/oxfordjournals.aje.a112510 · Cited by 1,001 · cited 1× · see citing works
“To account for judge noise, we apply Rogan-Gladen calibration [20] using GPT-4o’s estimated true positive rate (TPR) and false positive rate (FPR) on the human-verified subset: where ˆ 𝑝 is the observed GPT-4o-judged accuracy and 𝑝 is the calibrated accuracy.”
Jin, D., Pan, E., Oufattole, N., Weng, W.-H., Fang, H., & Szolovits, P. (2021) · Applied Sciences · doi:10.3390/app11146421 · Cited by 718 · see citing works
PubMedQA: A Dataset for Biomedical Research Question Answering
Jin, Q., Dhingra, B., Liu, Z., Cohen, W., & Lu, X. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1259 · Cited by 685 · see citing works
Grattafiori, A., Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Vaughan, A., et al. (2024) · arXiv · doi:10.48550/arxiv.2407.21783 · Cited by 505 · cited 1× · see citing works
“5-7B [25], Qwen3-8B, LLaMA-3-8B [6], and DeepSeek-R1-Distill-LLaMA-8B [5], served via vLLM (temperature=0.1, maxtokens=8,096), under 5-fold cross-validation on 962 questions from EHRNoteQA.”
InfluentialAlmanac — Retrieval-Augmented Language Models for Clinical Medicine
Zakka, C., Shad, R., Chaurasia, A., Dalal, A. R., Kim, J. L., Moor, M., Fong, R., Phillips, C., Alexander, K., Ashley, E., et al. (2024) · NEJM AI · doi:10.1056/aioa2300068 · Cited by 424 · see citing works
Large language models are few-shot clinical information extractors
Agrawal, M., Hegselmann, S., Lang, H., Kim, Y., & Sontag, D. (2022) · Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2022.emnlp-main.130 · Cited by 358 · see citing works
BioMistral: A Collection of Open-Source Pretrained Large Language Models for Medical Domains
Labrak, Y., Bazoge, A., Morin, E., Gourraud, P.-A., Rouvier, M., & Dufour, R. (2024) · Findings of the Association for Computational Linguistics ACL 2024 · doi:10.18653/v1/2024.findings-acl.348 · Cited by 269 · see citing works
Medical Hallucination in Foundation Models and Their Impact on HealthcarePreprint
Kim, Y., Jeong, H., Chen, S., Li, S. S., Park, C., Lu, M., Alhamoud, K., Mun, J., Grau, C., Jung, M., et al. (2025) · medRxiv · doi:10.1101/2025.02.28.25323115 · Cited by 253 · see citing works
A question-entailment approach to question answering
Abacha, A. B., & Demner‐Fushman, D. (2019) · BMC Bioinformatics · doi:10.1186/s12859-019-3119-4 · Cited by 221 · see citing works
(2022) · doi:10.18653/v1/2022.emnlp-main.669 · Cited by 177 · see citing works
Qwen, :, Yang, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Li, C., Liu, D., et al. (2024) · arXiv · doi:10.48550/arxiv.2412.15115 · Cited by 128 · see citing works
Abacha, A. B., Shivade, C., & Demner‐Fushman, D. (2019) · BioNLP Workshop and Shared Task (BioNLP) · doi:10.18653/v1/w19-5039 · Cited by 119 · cited 1× · see citing works
“Datasets such as Med-Halu [2] and MedHallu [18] draw questions from structured knowledge bases including PubMedQA [10], LiveQA [4], USMLE [9], and MedicationQA [1].”
HuaTuo: Tuning LLaMA Model with Chinese Medical Knowledge
Wang, H., Liu, C., Xi, N., Qiang, Z., Zhao, S., Qin, B., & Liu, T. (2023) · arXiv · doi:10.48550/arxiv.2304.06975 · Cited by 115 · see citing works
Does Fine-Tuning LLMs on New Knowledge Encourage Hallucinations?
Gekhman, Z., Yona, G., Aharoni, R., Eyal, M., Feder, A., Reichart, R., & Herzig, J. (2024) · Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2024.emnlp-main.444 · Cited by 86 · see citing works
Does Fine-Tuning LLMs on New Knowledge Encourage Hallucinations?
Gekhman, Z., Yona, G., Aharoni, R., Eyal, M., Feder, A., Reichart, R., & Herzig, J. (2024) · arXiv · doi:10.48550/arxiv.2405.05904 · Cited by 86 · see citing works
MedHallu: A Comprehensive Benchmark for Detecting Medical Hallucinations in Large Language Models
Pandit, S., Xu, J., Hong, J., Wang, Z., Chen, T., Xu, K., & Ding, Y. (2025) · arXiv · doi:10.48550/arxiv.2502.14302 · Cited by 31 · see citing works
EHRNoteQA: An LLM Benchmark for Real-World Clinical Practice Using Discharge Summaries
Kweon, S., Kim, J., Kwak, H., Cha, D., Yoon, H., Kim, K., Yang, J., Won, S., & Choi, E. (2024) · neural information processing systems · doi:10.52202/079017-3958 · Cited by 19 · see citing works
MedCaseReasoning: Evaluating and learning diagnostic reasoning from clinical case reportsPreprint
Wu, K., Wu, E., Thapa, R., Wei, K., Zhang, A., Suresh, A., Tao, J. J., Sun, M., Lozano, A., & Zou, J. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2505.11733 · Cited by 5 · see citing works
MedHalu: Hallucinations in Responses to Healthcare Queries by Large Language Models
Agarwal, V., Jin, Y., Chandra, M., et al. (2024) · arXiv · doi:10.48550/arxiv.2409.19492
Flagged first, then by citations
Research works printed but not linked: 3
- Patrick Lewis Ethan Perez Aleksandra Piktus Fabio Petroni Vladimir Karpukhin Naman Goyal Heinrich Küttler Mike Lewis Wen-tau Yih Tim Rocktäschel and others. 2020. Retrieval-augmented generation for knowledge-intensive nlp tasks. Advances in neural information processing systems 33 (2020) 9459–9474.
- 10.5555/1823431 · DOI
- Wenhui Wang Furu Wei Li Dong Hangbo Bao Nan Yang and Ming Zhou. 2020. Minilm: Deep self-attention distillation for task-agnostic compression of pre-trained transformers. Advances in neural information processing systems 33 (2020) 5776–5788.
Cited by
Sorted by citations
Yu, Z., Li, M.-H., Yu, L., Chow, K.-H., Qin, C.-W., Wu, X.-Y., Xing, W.-P., Xiong, S.-G., & Han, M. (2026) · doi:10.48550/arxiv.2610.05472
“For consequential claims, the relevant evidence may be a source, image region, formal constraint, or executable test [33, 85, 93, 141].”
- Authors
- Liao, Y., Franco, Z., Lizarraga Mazaba, J. E., & Chen, K.
- Affiliations
- Concordia University Wisconsin (US)
- Medical College of Wisconsin (US)
- Medical College of Wisconsin, Milwaukee, WI, USA
- University of Maryland, Baltimore County (US)
- Venue
- Proceedings of the 17th ACM International Conference on Bioinformatics, Computational Biology and Health Informatics
- Publisher
- ACM
- Published
- 2026-06-30
- Pages
- 1-6
- OpenAlex
- W7171539531
- Semantic Scholar
- f1d8b48cd17c7810e5d7e59bff4aefcb624ceb3e
- Cited by
- 0
- References
- 16
- FWCI
- 0.00
- Type
- conference paper
- Open access
- hybrid
- doi.org
- Locations
- ACM International Conference on Bioinformatics, Computational Biology, and Health Informatics (ACM-BCB) (publishedVersion, cc-by)
- Funders
- NSF (National Science Foundation) (2517121)
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Keywords
- hallucination mitigation, medical large language models, in-context learning, medical question answering, electronic health records, LLM-as-a-judge, clinical decision support, reasoning errors
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
