CoChat
Sign Up
PreprintCited by 8Artificial Intelligence in Healthcare and EducationTopic ModelingMultimodal Machine Learning Applications

Towards Reliable Medical Question Answering: Techniques and Challenges in Mitigating Hallucinations in Language Models

Open access
Pham, Duy Khoa, Vo, Bao Quoc·2024·arXiv
DOI 10.48550/arxiv.2408.13808arXiv 2408.13808
Open this paper in CoChat — free accountOpen PDF

Abstract

The rapid advancement of large language models (LLMs) has significantly impacted various domains, including healthcare and biomedicine. However, the phenomenon of hallucination, where LLMs generate outputs that deviate from factual accuracy or context, poses a critical challenge, especially in high-stakes domains. This paper conducts a scoping study of existing techniques for mitigating hallucinations in knowledge-based task in general and especially for medical domains. Key methods covered in the paper include Retrieval-Augmented Generation (RAG)-based techniques, iterative feedback loops, supervised fine-tuning, and prompt engineering. These techniques, while promising in general contexts, require further adaptation and optimization for the medical domain due to its unique demands for up-to-date, specialized knowledge and strict adherence to medical guidelines. Addressing these challenges is crucial for developing trustworthy AI systems that enhance clinical decision-making and patient safety as well as accuracy of biomedical scientific research.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2408.13808

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • RAG reduces the frequency of hallucinations and improves interpretability, especially for information-sensitive tasks such as question answering and multi-hop reasoning.

    “By combining both parametric (trained) and non-parametric (retrieved) sources, RAG reduces the frequency of hallucinations and improves interpretability, especially for information-sensitive tasks such as question answering and multi-hop reasoning [4, 25, 33].”

    From SKiM-GPT: combining biomedical literature-based discovery with large language model hypothesis evaluation (2025) · cites this paper as [25]

    The RAG section (page not provided) says that combining model knowledge with retrieved documents supports question answering and that grounding responses in external information can reduce hallucinations. It does not say that RAG improves interpretability or establish the claimed benefit specifically for multi-hop reasoning; it describes a multi-stage question-decomposition approach instead. (LLM, conf=0.87)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Still being checked.

Research works printed but not linked: 45

  1. Muhammad Aurangzeb Ahmad, Ilker Yaramis, and Taposh Dutta Roy. Creating trustworthy llms: Dealing with hallucinations in healthcare ai. arXiv preprint arXiv:2311.01463, 2023. · arXiv
  2. Rahul Aralikatte, Shashi Narayan, Joshua Maynez, Sascha Rothe, and Ryan McDonald. Focus attention: Promoting faithfulness and diversity in summarization. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli, editors, Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Lan- guage Processing (Volume 1: Long Papers), pages 6078- 6095, Online, August 2021. Association for Computa- tional Linguistics.
  3. Elliot Bolton, Abhinav Venigalla, Michihiro Yasunaga, David Hall, Betty Xiong, Tony Lee, Roxana Daneshjou, Jonathan Frankle, Percy Liang, Michael Carbin, et al. Biomedlm: A 2.7 b parameter language model trained on biomedical text. arXiv preprint arXiv:2403.18421, 2024. · arXiv
  4. Hejing Cao, Zhenwei An, Jiazhan Feng, Kun Xu, Liwei Chen, and Dongyan Zhao. A step closer to compre- hensive answers: Constrained multi-stage question de- composition with large language models. arXiv preprint arXiv:2311.07491, 2023. · arXiv
  5. Daixuan Cheng, Shaohan Huang, Junyu Bi, Yuefeng Zhan, Jianfeng Liu, Yujing Wang, Hao Sun, Furu Wei, Weiwei Deng, and Qi Zhang. UPRISE: Universal prompt retrieval for improving zero-shot evaluation. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Proceed- ings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 12318-12337, Sin- gapore, December 2023. Association for Computational Linguistics.
  6. Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih, editors, Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 2197-2214, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics.
  7. Yang Feng, Wanying Xie, Shuhao Gu, Chenze Shao, Wen Zhang, Zhengxin Yang, and Dong Yu. Modeling fluency and faithfulness for diverse neural machine translation. Proceedings of the AAAI Conference on Artificial Intel- ligence, 34(01):59-66, Apr. 2020.
  8. Zhenxiang Gao, Lingyao Li, Siyuan Ma, Qinyong Wang, Libby Hemphill, and Rong Xu. Examining the potential of chatgpt on biomedical information retrieval: Fact- checking drug-disease associations. Annals of Biomedi- cal Engineering, pages 1-9, 2023.
  9. Yucheng Hu and Yuxing Lu. Rag and rau: A survey on retrieval-augmented language model in natural language processing. arXiv preprint arXiv:2404.19543, 2024. · arXiv
  10. Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. Survey of hallucination in natural lan- guage generation. ACM Computing Surveys, 55(12):1- 38, 2023.
  11. Ziwei Ji, Tiezheng Yu, Yan Xu, Nayeon Lee, Etsuko Ishii, and Pascale Fung. Towards mitigating LLM hal- lucination via self reflection. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Findings of the Associa- tion for Computational Linguistics: EMNLP 2023, pages 1827-1843, Singapore, December 2023. Association for Computational Linguistics.
  12. Haoqiang Kang, Juntong Ni, and Huaxiu Yao. Ever: Mit- igating hallucination in large language models through real-time verification and rectification. arXiv preprint arXiv:2311.09114, 2023. · arXiv
  13. Tiziano Labruna, Jon Ander Campos, and Gorka Azkune. When to retrieve: Teaching llms to utilize information retrieval effectively. arXiv preprint arXiv:2404.19705, 2024. · arXiv
  14. Rémi Lebret, David Grangier, and Michael Auli. Neural text generation from structured data with application to the biography domain. In Jian Su, Kevin Duh, and Xavier Carreras, editors, Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1203-1213, Austin, Texas, November 2016. Association for Computational Linguistics.
  15. Katherine Lee, Daphne Ippolito, Andrew Nystrom, Chiyuan Zhang, Douglas Eck, Chris Callison-Burch, and Nicholas Carlini. Deduplicating training data makes language models better. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio, editors, Proceedings of the 60th Annual Meeting of the Association for Com- putational Linguistics (Volume 1: Long Papers), pages 8424-8445, Dublin, Ireland, May 2022. Association for Computational Linguistics.
  16. Deren Lei, Yaxi Li, Mingyu Wang, Vincent Yun, Emily Ching, Eslam Kamal, et al. Chain of natural language inference for reducing large language model ungrounded hallucinations. arXiv preprint arXiv:2310.03951, 2023. · arXiv
  17. Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. Retrieval-augmented generation for knowledge- intensive nlp tasks. Advances in Neural Information Processing Systems, 33:9459-9474, 2020.
  18. Kenneth Li, Oam Patel, Fernanda Viégas, Hanspeter Pfister, and Martin Wattenberg. Inference-time interven- tion: Eliciting truthful answers from a language model. Advances in Neural Information Processing Systems, 36, 2024.
  19. Shayne Longpre, Kartik Perisetla, Anthony Chen, Nikhil Ramesh, Chris DuBois, and Sameer Singh. Entity-based knowledge conflicts in question answering. In Marie- Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih, editors, Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 7052-7063. Association for Computa- tional Linguistics, November 2021.
  20. Yuning Mao, Pengcheng He, Xiaodong Liu, Yelong Shen, Jianfeng Gao, Jiawei Han, and Weizhu Chen. Generation-augmented retrieval for open-domain ques- tion answering. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli, editors, Proceedings of the 59th An- nual Meeting of the Association for Computational Lin- guistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 4089-4100, Online, August 2021. Association for Computational Linguistics.
  21. Joshua Maynez, Shashi Narayan, Bernd Bohnet, and Ryan McDonald. On faithfulness and factuality in abstractive summarization. In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel Tetreault, editors, Proceedings of the 58th Annual Meeting of the Association for Com- putational Linguistics, pages 1906-1919, Online, July 2020. Association for Computational Linguistics.
  22. Jessica Morley, Caio C.V. Machado, Christopher Burr, Josh Cowls, Indra Joshi, Mariarosaria Taddeo, and Lu- ciano Floridi. The ethics of ai in health care: A mapping review. Social Science & Medicine, 260:113172, 2020.
  23. Mengjia Niu, Hao Li, Jie Shi, Hamed Haddadi, and Fan Mo. Mitigating hallucinations in large language models via self-refinement-enhanced knowledge retrieval. arXiv preprint arXiv:2405.06545, 2024. · arXiv
  24. Ankit Pal, Logesh Kumar Umapathi, and Malaikannan Sankarasubbu. Med-halt: Medical domain hallucination test for large language models. arXiv preprint, 2023.
  25. Ankur Parikh, Xuezhi Wang, Sebastian Gehrmann, Man- aal Faruqui, Bhuwan Dhingra, Diyi Yang, and Dipan- jan Das. ToTTo: A controlled table-to-text generation dataset. In Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu, editors, Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1173-1186, Online, November 2020. Association for Computational Linguistics.
  26. Baolin Peng, Michel Galley, Pengcheng He, Hao Cheng, Yujia Xie, Yu Hu, Qiuyuan Huang, Lars Liden, Zhou Yu, Weizhu Chen, et al. Check your facts and try again: Improving large language models with external knowledge and automated feedback. arXiv preprint arXiv:2302.12813, 2023. · arXiv
  27. Patrik Puchert, Poonam Poonam, Christian van Onzenoodt, and Timo Ropinski. Llmmaps-a visual metaphor for stratified evaluation of large language models. arXiv preprint arXiv:2304.00457, 2023. · arXiv
  28. Krithi Pushpanathan, Zhi Wei Lim, Samantha Min Er Yew, David Ziyou Chen, Hazel Anne Hui'En Lin, Joce- lyn Hui Lin Goh, Wendy Meihua Wong, Xiaofei Wang, Marcus Chun Jin Tan, Victor Teck Chang Koh, et al. Popular large language model chatbots' accuracy, com- prehensiveness, and self-awareness in answering ocular symptom queries. Iscience, 26(11), 2023.
  29. Vipula Rawte, Swagata Chakraborty, Agnibh Pathak, Anubhav Sarkar, S.M Towhidul Islam Tonmoy, Aman Chadha, Amit Sheth, and Amitava Das. The troubling emergence of hallucination in large language models - an extensive definition, quantification, and prescriptive remediations. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Proceedings of the 2023 Conference on Em- pirical Methods in Natural Language Processing, pages 2541-2573, Singapore, December 2023. Association for Computational Linguistics.
  30. Chenglei Si, Zhe Gan, Zhengyuan Yang, Shuohang Wang, Jianfeng Wang, Jordan Boyd-Graber, and Lijuan Wang. Prompting gpt-3 to be reliable. arXiv preprint arXiv:2210.09150, 2022. · arXiv
  31. Lichao Sun, Yue Huang, Haoran Wang, Siyuan Wu, Qi- hui Zhang, Chujie Gao, Yixin Huang, Wenhan Lyu, Yix- uan Zhang, Xiner Li, et al. Trustllm: Trustworthiness in large language models. arXiv preprint arXiv:2401.05561, 2024. · arXiv
  32. Katherine Tian, Eric Mitchell, Huaxiu Yao, Christo- pher D Manning, and Chelsea Finn. Fine-tuning language models for factuality. arXiv preprint arXiv:2311.08401, 2023. · arXiv
  33. Ran Tian, Shashi Narayan, Thibault Sellam, and Ankur P Parikh. Sticking to the facts: Confident decoding for faithful data-to-text generation. arXiv preprint arXiv:1910.08684, 2019. · arXiv
  34. SM Tonmoy, SM Zaman, Vinija Jain, Anku Rani, Vipula Rawte, Aman Chadha, and Amitava Das. A compre- hensive survey of hallucination mitigation techniques in large language models. arXiv preprint arXiv:2401.01313, 2024. · arXiv
  35. Tu Vu, Mohit Iyyer, Xuezhi Wang, Noah Constant, Jerry Wei, Jason Wei, Chris Tar, Yun-Hsuan Sung, Denny Zhou, Quoc Le, et al. Freshllms: Refreshing large language models with search engine augmentation. arXiv preprint arXiv:2310.03214, 2023. · arXiv
  36. Benyou Wang, Qianqian Xie, Jiahuan Pei, Zhihong Chen, Prayag Tiwari, Zhao Li, and Jie Fu. Pre-trained language models in biomedical domain: A systematic survey. ACM Computing Surveys, 56(3):1-52, 2023.
  37. Kevin Wu, Eric Wu, and James Zou. How faithful are rag models? quantifying the tug-of-war between rag and llms' internal prior. arXiv preprint arXiv:2404.10198, 2024. · arXiv
  38. Tianhao Wu, Weizhe Yuan, Olga Golovneva, Jing Xu, Yuandong Tian, Jiantao Jiao, Jason Weston, and Sainba- yar Sukhbaatar. Meta-rewarding language models: Self- improving alignment with llm-as-a-meta-judge. arXiv preprint arXiv:2407.19594, 2024. · arXiv
  39. Betelhem Zewdu Wubineh, Fitsum Gizachew Deriba, and Michael Melese Woldeyohannis. Exploring the opportunities and challenges of implementing artificial intelligence in healthcare: A systematic literature review. In Urologic Oncology: Seminars and Original Investiga- tions. Elsevier, 2023.
  40. Ran Xu, Wenqi Shi, Yue Yu, Yuchen Zhuang, Yanqiao Zhu, May D Wang, Joyce C Ho, Chao Zhang, and Carl Yang. Bmretriever: Tuning large language models as better biomedical text retrievers. arXiv preprint arXiv:2404.18443, 2024. · arXiv
  41. Han Yang, Mingchen Li, Huixue Zhou, Yongkang Xiao, Qian Fang, and Rui Zhang. One llm is not enough: Harnessing the power of ensemble learning for medical question answering. medRxiv, 2023.
  42. Yuanshun Yao, Xiaojun Xu, and Yang Liu. Large language model unlearning. arXiv preprint arXiv:2310.10683, 2023. · arXiv
  43. Hanning Zhang, Shizhe Diao, Yong Lin, Yi R Fung, Qing Lian, Xingyao Wang, Yangyi Chen, Heng Ji, and Tong Zhang. R-tuning: Teaching large language models to refuse unknown questions. arXiv preprint arXiv:2311.09677, 2023. · arXiv
  44. Muru Zhang, Ofir Press, William Merrill, Alisa Liu, and Noah A Smith. How language model hallucinations can snowball. arXiv preprint arXiv:2305.13534, 2023. · arXiv
  45. Yue Zhang, Yafu Li, Leyang Cui, Deng Cai, Lemao Liu, Tingchen Fu, Xinting Huang, Enbo Zhao, Yu Zhang, Yulong Chen, et al. Siren's song in the ai ocean: a survey on hallucination in large language models. arXiv preprint arXiv:2309.01219, 2023. · arXiv

Cited by

Sorted by citations

Related

Authors
Pham, D. K., & Vo, B. Q.
ORCID
Pham, Duy Khoa
Venue
arXiv
Publisher
arXiv
Published
2024
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2408.13808
arXiv
2408.13808
OpenAlex
W4402701703
Cited by
8
References
0
Citations by year
2026: 3 · 2025: 5
Type
posted content
Language
en
Open access
Licence
Locations
  • arXiv (Cornell University) (submittedVersion, other-oa)
  • arXiv (Cornell University)
Topics
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
Keywords
hallucination mitigation, large language models, medical question answering, retrieval-augmented generation, iterative feedback loops, supervised fine-tuning, prompt engineering, clinical decision support, trustworthy AI, biomedical knowledge

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review