Hallucination Detection: Robustly Discerning Reliable Answers in Large Language Models
Open accessThe DOI matches its Crossref record; 45 of the 45 references it cites were found in a registry or an index.
Abstract
Large language models (LLMs) have gained widespread adoption in various natural language processing tasks, including question answering and dialogue systems. However, a major drawback of LLMs is the issue of hallucination, where they generate unfaithful or inconsistent content that deviates from the input source, leading to severe consequences. In this paper, we propose a robust discriminator named RelD to effectively detect hallucination in LLMs' generated answers. RelD is trained on the constructed RelQA, a bilingual question-answering dialogue dataset along with answers generated by LLMs and a comprehensive set of metrics. Our experimental results demonstrate that the proposed RelD successfully detects hallucination in the answers generated by diverse LLMs. Moreover, it performs well in distinguishing hallucination in LLMs' generated answers from both in-distribution and out-of-distribution datasets. Additionally, we also conduct a thorough analysis of the types of hallucinations that occur and present valuable insights. This research significantly contributes to the detection of reliable answers generated by LLMs and holds noteworthy implications for mitigating hallucination in the future work.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
RoBERTa: A Robustly Optimized BERT Pretraining Approach
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019) · arXiv · doi:10.48550/arxiv.1907.11692 · Cited by 16,515
Lewis, M., Liu, Y., Goyal, N., Ghazvininejad, M., Mohamed, A., Levy, O., Stoyanov, V., & Zettlemoyer, L. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.703 · Cited by 8,821
Lewis, M., Liu, Y., Goyal, N., Ghazvininejad, M., Mohamed, A., Levy, O., Stoyanov, V., & Zettlemoyer, L. (2019) · arXiv · doi:10.48550/arxiv.1910.13461 · Cited by 8,816
SQuAD: 100,000+ Questions for Machine Comprehension of Text
Rajpurkar, P., Zhang, J., Lopyrev, K., & Liang, P. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1264 · Cited by 6,208 · cited 1×
“RelQA consists of nine sub-datasets: SQuAD [55], DuReader [24], HotpotQA[75],MSMARCO[46],NewsQA [69],QuAC[11],CoQA[58], TriviaQA-Web [29], and TriviaQA-Wikipedia [29].”
MethodSurvey of Hallucination in Natural Language Generation
Ji, Z., Lee, N., Frieske, R., Yu, T., Su, D., Xu, Y., Ishii, E., Bang, Y. J., Madotto, A., & Fung, P. (2023) · ACM Computing Surveys · doi:10.1145/3571730 · Cited by 5,025
LLaMA: Open and Efficient Foundation Language Models
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. (2023) · arXiv · doi:10.48550/arxiv.2302.13971 · Cited by 4,602
Training language models to follow instructions with human feedback
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. (2022) · arXiv · doi:10.48550/arxiv.2203.02155 · Cited by 4,208
A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference
Williams, A., Nangia, N., & Bowman, S. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1101 · Cited by 3,342
BERTScore: Evaluating Text Generation with BERT
Zhang, T., Kishore, V., Wu, F., Weinberger, K. Q., & Artzi, Y. (2019) · arXiv · doi:10.48550/arxiv.1904.09675 · Cited by 2,865
A Survey of Large Language Models
Zhao, W. X., Zhou, K., Li, J., Tang, T., Dong, Z., Hou, Y., Zhang, B., Min, Y., Zhang, J., Liu, P., et al. (2026) · Frontiers of Computer Science · doi:10.1007/s11704-026-60308-3 · Cited by 2,093
Self-citationELECTRA: Pre-training Text Encoders as Discriminators Rather Than Generators
Clark, K., Luong, M.-T., Le, Q. V., & Manning, C. D. (2020) · arXiv · doi:10.48550/arxiv.2003.10555 · Cited by 1,844
TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension
Joshi, M., Choi, E., Weld, D. S., & Zettlemoyer, L. (2017) · arXiv · doi:10.48550/arxiv.1705.03551 · Cited by 1,650
Sequence Level Training with Recurrent Neural Networks
Ranzato, M., Chopra, S., Auli, M., & Zaremba, W. (2015) · arXiv · doi:10.48550/arxiv.1511.06732 · Cited by 1,426 · cited 1×
“During training, ground-truth serves as the input prefix for predicting subsequent tokens.”
BackgroundSummary of ChatGPT-Related Research and Perspective Towards the Future of Large Language Models
Liu, Y., Han, T., Ma, S., Zhang, J., Yang, Y., Tian, J., He, H., Li, A., He, M., Liu, Z., et al. (2023) · arXiv · doi:10.48550/arxiv.2304.01852 · Cited by 1,172
BLEURT: Learning Robust Metrics for Text Generation
Sellam, T., Das, D., & Parikh, A. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.704 · Cited by 1,005
BLEURT: Learning Robust Metrics for Text Generation
Sellam, T., Das, D., & Parikh, A. P. (2020) · arXiv · doi:10.48550/arxiv.2004.04696 · Cited by 1,005
CoQA: A Conversational Question Answering Challenge
Reddy, S., Chen, D., & Manning, C. D. (2019) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00266 · Cited by 975
Recipes for Building an Open-Domain Chatbot
Roller, S., Dinan, E., Goyal, N., Ju, D., Williamson, M., Liu, Y., Xu, J., Ott, M., Smith, E. M., Boureau, Y.-L., et al. (2021) · Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume · doi:10.18653/v1/2021.eacl-main.24 · Cited by 784
Recipes for building an open-domain chatbot
Roller, S., Dinan, E., Goyal, N., Ju, D., Williamson, M., Liu, Y., Xu, J., Ott, M., Shuster, K., Smith, E. M., et al. (2020) · arXiv · doi:10.48550/arxiv.2004.13637 · Cited by 784 · cited 1×
“…contains statistical metrics [21, 66, 72], model-based metrics (including Information Extraction (IE)-based metric, QA-based metric [25, 57, 60], Natural Language Inference (NLI) Metrics [26, 32, 73], Faithfulness Classification Metrics [25, 41, 79], LM-based Metrics [19, 67]), and…”
Bang, Y., Cahyawijaya, S., Lee, N., Dai, W., Su, D., Wilie, B., Lovenia, H., Ji, Z., Yu, T., Chung, W., et al. (2023) · Proceedings of the 13th International Joint Conference on Natural Language Processing and the 3rd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2023.ijcnlp-main.45 · Cited by 682 · cited 1×
“Largelanguagemodels(LLMs) haverevolutionizedvariousfields[78], including logical reasoning [3, 40], question answering [48], code generation [30], and vertical domains [42].”
BackgroundJohnson, D., Goodman, R., Patrinely, J., Stone, C., Zimmerman, E., Donald, R., Chang, S., Berkowitz, S., Finn, A., Jahangir, E., et al. (2023) · doi:10.21203/rs.3.rs-2566942/v1 · Cited by 598
A Categorical Archive of ChatGPT Failures
Borji, A. (2023) · arXiv · doi:10.48550/arxiv.2302.03494 · Cited by 514
(2020) · doi:10.18653/v1/2020.acl-main.450 · Cited by 356
Asking and Answering Questions to Evaluate the Factual Consistency of Summaries
Wang, A., Cho, K., & Lewis, M. (2020) · arXiv · doi:10.48550/arxiv.2004.04228 · Cited by 356 · cited 1×
“Model-based evaluation incorporates various metrics, including Information Extraction (IE)-based metrics, QA-based metrics [16, 63, 70], Natural Language Inference (NLI) metrics [17, 18, 25], Faithfulness Classification metrics [25, 41, 79], and LM-based metrics [19, 67].”
BackgroundHaluEval: A Large-Scale Hallucination Evaluation Benchmark for Large Language Models
Li, J., Cheng, X., Zhao, W. X., Nie, J.-Y., & Wen, J.-R. (2023) · arXiv · doi:10.48550/arxiv.2305.11747 · Cited by 340
Can Large Language Models Be an Alternative to Human Evaluations?
Chiang, C.-H., & Lee, H.-Y. (2023) · Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2023.acl-long.870 · Cited by 331
Can Large Language Models Be an Alternative to Human Evaluations?
Chiang, C.-H., & Lee, H.-Y. (2023) · arXiv · doi:10.48550/arxiv.2305.01937 · Cited by 331
Falke, T., Ribeiro, L. F. R., Utama, P. A., Dagan, I., & Gurevych, I. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1213 · Cited by 265 · cited 1×
“Model-based evaluation incorporates various metrics, including Information Extraction (IE)-based metrics, QA-based metrics [16, 63, 70], Natural Language Inference (NLI) metrics [17, 18, 25], Faithfulness Classification metrics [25, 41, 79], and LM-based metrics [19, 67].”
BackgroundQuAC : Question Answering in Context
Choi, E., He, H., Iyyer, M., Yatskar, M., Yih, W.-T., Choi, Y., Liang, P., & Zettlemoyer, L. (2018) · arXiv · doi:10.48550/arxiv.1808.07036 · Cited by 260
SummaC: Re-Visiting NLI-based Models for Inconsistency Detection in Summarization
Laban, P., Schnabel, T., Bennett, P. N., & Hearst, M. A. (2022) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00453 · Cited by 238
Peng, B., Galley, M., He, P., Cheng, H., Xie, Y., Hu, Y., Huang, Q., Liden, L., Yu, Z., Chen, W., et al. (2023) · arXiv · doi:10.48550/arxiv.2302.12813 · Cited by 156
Detecting Hallucinated Content in Conditional Neural Sequence Generation
Zhou, C., Neubig, G., Gu, J., Diab, M., Guzmán, F., Zettlemoyer, L., & Ghazvininejad, M. (2021) · Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021 · doi:10.18653/v1/2021.findings-acl.120 · Cited by 122
Entity-Based Knowledge Conflicts in Question Answering
Longpre, S., Perisetla, K., Chen, A., Ramesh, N., DuBois, C., & Singh, S. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.565 · Cited by 101
Evaluating Factuality in Generation with Dependency-level Entailment
Goyal, T., & Durrett, G. (2020) · Findings of the Association for Computational Linguistics: EMNLP 2020 · doi:10.18653/v1/2020.findings-emnlp.322 · Cited by 93
TRUE: Re-evaluating Factual Consistency Evaluation
Honovich, O., Aharoni, R., Herzig, J., Taitelbaum, H., Kukliansy, D., Cohen, V., Scialom, T., Szpektor, I., Hassidim, A., & Matias, Y. (2022) · arXiv · doi:10.48550/arxiv.2204.04991 · Cited by 84
Controlled Hallucinations: Learning to Generate Faithfully from Noisy Data
Filippova, K. (2020) · Findings of the Association for Computational Linguistics: EMNLP 2020 · doi:10.18653/v1/2020.findings-emnlp.76 · Cited by 79
A Simple Recipe towards Reducing Hallucination in Neural Surface Realisation
Nie, F., Yao, J.-G., Wang, J., Pan, R., & Lin, C.-Y. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1256 · Cited by 68
Factuality Enhanced Language Models for Open-Ended Text Generation
Lee, N., Ping, W., Xu, P., Patwary, M., Fung, P., Shoeybi, M., & Catanzaro, B. (2022) · arXiv · doi:10.48550/arxiv.2206.04624 · Cited by 68
(2022) · doi:10.52202/068431-2506 · Cited by 68
Sticking to the Facts: Confident Decoding for Faithful Data-to-Text Generation
Tian, R., Narayan, S., Sellam, T., & Parikh, A. P. (2019) · arXiv · doi:10.48550/arxiv.1910.08684 · Cited by 60 · cited 2×
“For example, Honovich et al.[25] employ the Q 2 method of QA systems to assess the consistency between the response and external knowledge.”
BackgroundMethodTowards Faithful Neural Table-to-Text Generation with Content-Matching Constraints
Wang, Z., Wang, X., An, B., Yu, D., & Chen, C. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.101 · Cited by 59
Evaluating Attribution in Dialogue Systems: The BEGIN Benchmark
Dziri, N., Rashkin, H., Linzen, T., & Reitter, D. (2021) · arXiv · doi:10.48550/arxiv.2105.00071 · Cited by 47 · cited 1×
“Model-based evaluation incorporates various metrics, including Information Extraction (IE)-based metrics, QA-based metrics [16, 63, 70], Natural Language Inference (NLI) metrics [17, 18, 25], Faithfulness Classification metrics [25, 41, 79], and LM-based metrics [19, 67].”
BackgroundSummaC: Re-Visiting NLI-based Models for Inconsistency Detection in Summarization
Laban, P., Schnabel, T., Bennett, P. N., & Hearst, M. A. (2021) · arXiv · doi:10.48550/arxiv.2111.09525 · Cited by 38
A Token-level Reference-free Hallucination Detection Benchmark for Free-form Text Generation
Liu, T., Zhang, Y., Brockett, C., Mao, Y., Sui, Z., Chen, W., & Dolan, B. (2021) · arXiv · doi:10.48550/arxiv.2104.08704 · Cited by 12 · cited 3×
“For example, Honovich et al.[25] employ the Q 2 method of QA systems to assess the consistency between the response and external knowledge.”
BackgroundMethod
Cited by
"I look at it as the king of knowledge": How Blind People Use and Understand Generative AI Tools
Adnin, R., & Das, M. (2024) · The 26th International ACM SIGACCESS Conference on Computers and Accessibility · doi:10.1145/3663548.3675631 · Cited by 73
Shorinwa, O., Mei, Z., Lidard, J., Ren, A. Z., & Majumdar, A. (2026) · ACM Computing Surveys · doi:10.1145/3744238 · Cited by 61
Automating the Development of Task-oriented LLM-based Chatbots
Sánchez Cuadrado, J., Pérez-Soler, S., Guerra, E., & De Lara, J. (2024) · ACM Conversational User Interfaces 2024 · doi:10.1145/3640794.3665538 · Cited by 46
Zhao, Q., Wang, R., Cen, Y., Zha, D., Tan, S., Dong, Y., & Tang, J. (2024) · Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2024.emnlp-main.1259 · Cited by 32
Sood, A. K., Zeadally, S., & Hong, E. (2025) · Computers and Electrical Engineering · doi:10.1016/j.compeleceng.2025.110307 · Cited by 33
Metadata
- Authors
- Chen, Y., Fu, Q., Yuan, Y., Wen, Z., Fan, G., Liu, D., Zhang, D., Li, Z., & Xiao, Y.
- Venue
- Proceedings of the 32nd ACM International Conference on Information and Knowledge Management
- Publisher
- ACM
- Published
- 2023-10-21
- Pages
- 245-255
- OpenAlex
- W4387848774
- Cited by
- 94
- References
- 45
- FWCI
- 17.77
- Type
- conference paper
- Language
- en
- Open access
- hybrid
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.