CoChat
Sign Up
Cited by 43Topic ModelingMultimodal Machine Learning ApplicationsNatural Language Processing Techniques

RAG-HAT: A hallucination-aware tuning pipeline for LLM in retrieval-augmented generation

Open access
Song, Juntong, Wang, Xingguang, Zhu, Juno, Wu, Yuanhao, Cheng, Xuxin, Zhong, Randy, Niu, Cheng·2024·Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: Industry Track
DOI 10.18653/v1/2024.emnlp-industry.113
Open this paper in CoChat — free accountOpen PDF

Abstract

Retrieval-augmented generation (RAG) has emerged as a significant advancement in the field of large language models (LLMs).By integrating up-to-date information not available during their initial training, RAG greatly enhances the practical utility of LLMs in realworld applications.However, even with RAG, LLMs can still produce inaccurate outputs, such as distorting or misinterpreting source content, posing risks in high-trust scenarios.To address these issues, we introduce a novel approach called Hallucination Aware Tuning (HAT).This method involves training hallucination detection models that generate detection labels and provide detailed descriptions of the detected hallucinations.Utilizing these detection results-particularly the hallucination descriptions-GPT-4 Turbo is employed to correct any detected hallucinations.The corrected outputs, free of hallucinations, along with the original versions, are used to create a preference dataset for Direct Preference Optimization (DPO) training.The fine-tuning through DPO leads to LLMs that exhibit a reduced rate of hallucinations and deliver improved answer quality.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.18653/v1/2024.emnlp-industry.113

Cites

0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • There is a risk of LLM hallucinations even with Retrieval-Augmented Generation (RAG).

    “Abstention is critical given that there is a risk of LLM hallucinations, even with Retrieval-Augmented Generation (RAG) (Niu et al. 2023;Song et al. 2024).”

    From Sacred or Synthetic? Evaluating LLM Reliability and Abstention for Religious Questions (2025) · cites this paper as Song et al. 2024

  • Systems typically assume retrieved passages suffice, failing to detect and handle cases where key evidence is missing.

    “No mechanism for insufficient-context detection: Systems typically assume retrieved passages suffice, failing to detect and handle cases where key evidence is missing (Song et al., 2024).”

    From VeReaFine: Iterative Verification Reasoning Refinement RAG for Hallucination-Resistant on Open-Ended Clinical QA (2025) · cites this paper as Song et al., 2024

    The Introduction (page unavailable) says hallucinations persist in RAG systems and identifies a lack of work integrating hallucination detection and mitigation. The detection discussion and prompts address whether generated statements are supported by the provided input, but do not say that systems typically assume retrieved passages are sufficient or fail to detect missing key evidence. Searches for insufficient context, missing evidence, and inadequate retrieved information found no passage establishing that claim. (LLM, conf=0.73)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

The publisher hasn’t deposited this paper’s reference list.

Research works printed but not linked: 3

  1. Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980. · arXiv
  2. Shashi Narayan, Shay B. Cohen, and Mirella Lapata. 2018. Don't give me the details, just the summary! Topic-aware convolutional neural networks for ex- treme summarization. In Proceedings of the 2018 Conference on Empirical Methods in Natural Lan- guage Processing, Brussels, Belgium.
  3. Yanli Zhao, Andrew Gu, Rohan Varma, Liang Luo, Chien-Chin Huang, Min Xu, Less Wright, Hamid Shojanazeri, Myle Ott, Sam Shleifer, et al. 2023. Py- torch fsdp: experiences on scaling fully sharded data parallel. arXiv preprint arXiv:2304.11277. · arXiv

Cited by

Sorted by citations

Related

Authors
Song, J., Wang, X., Zhu, J., Wu, Y., Cheng, X., Zhong, R., & Niu, C.
ORCID
Venue
Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: Industry Track
Publisher
Association for Computational Linguistics
Published
2024
Pages
1548-1558
DOI
10.18653/v1/2024.emnlp-industry.113
OpenAlex
W4404783652
Cited by
36
References
0
Citations by year
2026: 23 · 2025: 13
FWCI
10.62
Type
conference paper
Language
en
Open access
Licence
cc-by
Locations
Conference on Empirical Methods in Natural Language Processing (EMNLP) (publishedVersion, cc-by)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
Keywords
retrieval-augmented generation, hallucination detection, hallucination correction, preference dataset, direct preference optimization, large language models, GPT-4 Turbo

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review