CoChat
Sign Up
PreprintCited by 47Topic ModelingNatural Language Processing TechniquesAdversarial Robustness in Machine Learning

Self-contradictory Hallucinations of Large Language Models: Evaluation, Detection and Mitigation

Open access
Mündler, Niels, He, Jingxuan, Jenko, Slobodan, Vechev, Martin·2023·arXiv
DOI 10.48550/arxiv.2305.15852arXiv 2305.15852
Open this paper in CoChat — free accountOpen PDF

Abstract

Large language models (large LMs) are susceptible to producing text that contains hallucinated content. An important instance of this problem is self-contradiction, where the LM generates two contradictory sentences within the same context. In this work, we present a comprehensive investigation into self-contradiction for various instruction-tuned LMs, covering evaluation, detection, and mitigation. Our primary evaluation task is open-domain text generation, but we also demonstrate the applicability of our approach to shorter question answering. Our analysis reveals the prevalence of self-contradictions, e.g., in 17.7% of all sentences produced by ChatGPT. We then propose a novel prompting-based framework designed to effectively detect and mitigate self-contradictions. Our detector achieves high accuracy, e.g., around 80% F1 score when prompting ChatGPT. The mitigation algorithm iteratively refines the generated text to remove contradictory information while preserving text fluency and informativeness. Importantly, our entire framework is applicable to black-box LMs and does not require retrieval of external knowledge. Rather, our method complements retrieval-based methods, as a large portion of self-contradictions (e.g., 35.2% for ChatGPT) cannot be verified using online text. Our approach is practically effective and has been released as a push-button tool to benefit the public at https://chatprotect.ai/.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2305.15852

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • GPT 3.5 and other LLMs far surpass the 14.3% hallucination baseline on general QA queries when answering legal QA queries.

    “On our legal QA queries, GPT 3.5 and our other LLMs far surpass this baseline rate-and it is possible that the true hallucination rate is even higher.”

    From Large Legal Fictions: Profiling Legal Hallucinations in Large Language Models (2024) · cites this paper as Mündler et al. (2023)

    Section 6.3 reports self-contradictions in general question answering, and Table 5 gives rates of 18.2% for ChatGPT and 38.0% for Llama2-70B-Chat without retrieval augmentation. It does not give a 14.3% hallucination baseline. The legal-QA results are the citing authors’ own findings, not findings attributed to this source. (LLM, conf=0.78)

    Experimental — accuracy not yet measured

  • CoVe outperforms existing hallucination mitigation baselines (SelfCheckGPT NLI/LLM and ChatProtect) on longform generation.

    “Our results in Table 3 show that COVE outperforms all these baselines.”

    From Chain-of-Verification Reduces Hallucination in Large Language Models (2024) · cites this paper as Mündler et al., 2023

    The Comparison with SelfCheckGPT section reports that this paper’s detection method outperforms three SelfCheckGPT variants, and the Generality section identifies its ChatProtect tool. Neither those passages nor the open-domain text-generation results in Section 6.2 compare CoVe with SelfCheckGPT NLI/LLM or ChatProtect. Searches for CoVe and the named baselines found no such comparison in our copy. (LLM, conf=0.78)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

The citation lists haven’t been fetched yet

Cited by 47

Research works printed but not linked: 47

  1. Amos Azaria and Tom M. Mitchell. The internal state of an LLM knows when its lying. CoRR, abs/2304.13734, 2023. URL https://arxiv.org/abs/2304.13734. · arXiv
  2. Yejin Bang, Samuel Cahyawijaya, Nayeon Lee, Wenliang Dai, Dan Su, Bryan Wilie, Holy Lovenia, Ziwei Ji, Tiezheng Yu, Willy Chung, Quyet V. Do, Yan Xu, and Pascale Fung. A multitask, multilingual, multimodal evaluation of ChatGPT on reasoning, hallucination, and interactivity. CoRR, abs/2302.04023, 2023. URL https://arxiv.org/abs/2302.04023. · arXiv
  3. Michele Banko, Michael J. Cafarella, Stephen Soderland, Matthew Broadhead, and Oren Et- zioni. Open information extraction from the web. In IJCAI, 2007. URL http://ijcai.org/ Proceedings/07/Papers/429.pdf.
  4. Farima Fatahi Bayat, Nikita Bhutani, and H. V. Jagadish. Compactie: compact facts in open informa- tion extraction. In NAACL, 2022. URL https://doi.org/10.18653/v1/2022.naacl-main. · DOI
  5. Ali Borji. A categorical archive of chatgpt failures. CoRR, abs/2302.03494, 2023. URL https: //arxiv.org/abs/2302.03494. · DOI
  6. Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. A large annotated corpus for learning natural language inference. In EMNLP, 2015. URL https://doi.org/10. 18653/v1/d15-1075. · DOI · Paper page
  7. Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: scaling language modeling with pathways. CoRR, abs/2204.02311, 2022. URL https://arxiv. org/abs/2204.02311. · arXiv
  8. Jacob Cohen. A coefficient of agreement for nominal scales. Educational and psychological measurement, 20(1):37-46, 1960.
  9. Roi Cohen, Mor Geva, Jonathan Berant, and Amir Globerson. Crawling the internal knowledge-base of language models. In Findings of EACL, 2023a. URL https://aclanthology.org/2023. findings-eacl.139.
  10. Roi Cohen, May Hamri, Mor Geva, and Amir Globerson. LM vs LM: detecting factual errors via cross examination. CoRR, abs/2305.13281, 2023b. URL https://arxiv.org/abs/2305.13281. · arXiv
  11. Ido Dagan, Oren Glickman, and Bernardo Magnini. The PASCAL recognising textual entailment challenge. In MLCW, 2005. URL https://doi.org/10.1007/11736790_9. · DOI
  12. Emily Dinan, Stephen Roller, Kurt Shuster, Angela Fan, Michael Auli, and Jason Weston. Wizard of wikipedia: Knowledge-powered conversational agents. In ICLR, 2019. URL https:// openreview.net/forum?id=r1l73iRqKm.
  13. Bradley H. Dowden. Logical Reasoning, chapter 9, pp. 283-315. 2011. URL https://www.sfu. ca/~swartz/logical_reasoning/09_dowden_pp_283-315.pdf.
  14. Yanai Elazar, Nora Kassner, Shauli Ravfogel, Abhilasha Ravichander, Eduard H. Hovy, Hinrich Schütze, and Yoav Goldberg. Measuring and improving consistency in pretrained language models. Trans. Assoc. Comput. Linguistics, 9:1012-1031, 2021. URL https://doi.org/10.1162/ tacl_a_00410. · DOI · Paper page
  15. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In ICLR, 2021. URL https: //openreview.net/forum?id=d7KBjmI3GmQ.
  16. Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The curious case of neural text degeneration. In ICLR, 2020. URL https://openreview.net/forum?id=rygGQyrFvH.
  17. Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Yejin Bang, Andrea Madotto, and Pascale Fung. Survey of hallucination in natural language generation. ACM Comput. Surv., 55(12), 2023. URL https://doi.org/10.1145/3571730. · DOI · Paper page
  18. Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. In NeurIPS, 2022. URL https://arxiv.org/abs/ 2205.11916.
  19. Wojciech Kryscinski, Bryan McCann, Caiming Xiong, and Richard Socher. Evaluating the factual consistency of abstractive text summarization. In EMNLP, 2020. URL https://doi.org/10. 18653/v1/2020.emnlp-main.750. · DOI · Paper page
  20. Lorenz Kuhn, Yarin Gal, and Sebastian Farquhar. Semantic uncertainty: Linguistic invariances for uncertainty estimation in natural language generation. In ICLR, 2023. URL https:// openreview.net/pdf?id=VD-AYtP0dve.
  21. Rémi Lebret, David Grangier, and Michael Auli. Neural text generation from structured data with application to the biography domain. In EMNLP, 2016. URL https://doi.org/10.18653/ v1/d16-1128. · DOI
  22. Nayeon Lee, Wei Ping, Peng Xu, Mostofa Patwary, Pascale N. Fung, Mohammad Shoeybi, and Bryan Catanzaro. Factuality enhanced language models for open-ended text generation. In NeurIPS, 2022. URL https://arxiv.org/abs/2206.04624. · arXiv
  23. Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, et al. Holistic evaluation of language models. CoRR, abs/2211.09110, 2022. URL https://arxiv.org/abs/2211.09110. · arXiv
  24. Stephanie Lin, Jacob Hilton, and Owain Evans. TruthfulQA: measuring how models mimic human falsehoods. In ACL, 2022. URL https://doi.org/10.18653/v1/2022.acl-long.229. · DOI · Paper page
  25. Hanmeng Liu, Ruoxi Ning, Zhiyang Teng, Jian Liu, Qiji Zhou, and Yue Zhang. Evaluating the logical reasoning ability of chatgpt and GPT-4. CoRR, abs/2304.03439, 2023. URL https: //arxiv.org/abs/2304.03439. · arXiv
  26. Alex Mallen, Akari Asai, Victor Zhong, Rajarshi Das, Daniel Khashabi, and Hannaneh Hajishirzi. When not to trust language models: Investigating effectiveness of parametric and non-parametric memories. In ACL, 2023. URL https://doi.org/10.18653/v1/2023.acl-long.546. · DOI · Paper page
  27. Potsawee Manakul, Adian Liusie, and Mark J. F. Gales. MQAG: multiple-choice question answering and generation for assessing information consistency in summarization. CoRR, abs/2301.12307, 2023a. URL https://arxiv.org/abs/2301.12307. · arXiv
  28. Potsawee Manakul, Adian Liusie, and Mark J. F. Gales. SelfCheckGPT: zero-resource black-box hallucination detection for generative large language models. CoRR, abs/2303.08896, 2023b. URL https://arxiv.org/abs/2303.08896. · arXiv
  29. Joshua Maynez, Shashi Narayan, Bernd Bohnet, and Ryan T. McDonald. On faithfulness and factuality in abstractive summarization. In ACL, 2020. URL https://doi.org/10.18653/v1/ 2020.acl-main.173. · DOI · Paper page
  30. Sewon Min, Kalpesh Krishna, Xinxi Lyu, Mike Lewis, Wen-tau Yih, Pang Wei Koh, Mohit Iyyer, Luke Zettlemoyer, and Hannaneh Hajishirzi. Factscore: Fine-grained atomic evaluation of factual precision in long form text generation. CoRR, abs/2305.14251, 2023. URL https://arxiv. org/abs/2305.14251. · arXiv
  31. OpenAI. GPT-4 technical report. CoRR, abs/2303.08774, 2023b. URL https://arxiv.org/abs/ 2303.08774. · arXiv
  32. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. In NeurIPS, 2022. URL https://arxiv.org/abs/2203. 02155.
  33. Baolin Peng, Michel Galley, Pengcheng He, Hao Cheng, Yujia Xie, Yu Hu, Qiuyuan Huang, Lars Liden, Zhou Yu, Weizhu Chen, and Jianfeng Gao. Check your facts and try again: improving large language models with external knowledge and automated feedback. CoRR, abs/2302.12813, 2023. URL https://arxiv.org/abs/2302.12813. · arXiv
  34. Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick S. H. Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander H. Miller. Language models as knowledge bases? In EMNLP-IJCNLP, 2019. URL https://doi.org/10.18653/v1/D19-1250. · DOI · Paper page
  35. Chengwei Qin, Aston Zhang, Zhuosheng Zhang, Jiaao Chen, Michihiro Yasunaga, and Diyi Yang. Is chatgpt a general-purpose natural language processing task solver? CoRR, abs/2302.06476, 2023. URL https://arxiv.org/abs/2302.06476. · arXiv
  36. Kurt Shuster, Spencer Poff, Moya Chen, Douwe Kiela, and Jason Weston. Retrieval augmentation reduces hallucination in conversation. In Findings of EMNLP, 2021. URL https://doi.org/ 10.18653/v1/2021.findings-emnlp.320. · DOI · Paper page
  37. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurélien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. LLaMA: open and efficient foundation language models. CoRR, abs/2302.13971, 2023a. URL https://arxiv.org/abs/2302.13971. · arXiv
  38. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. CoRR, abs/2307.09288, 2023b. URL https://arxiv.org/abs/ 2307.09288. · arXiv
  39. Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V. Le, Ed H. Chi, and Denny Zhou. Self- consistency improves chain of thought reasoning in language models. In ICLR, 2023. URL https://arxiv.org/abs/2203.11171. · arXiv
  40. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V. Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models. In NeurIPS, 2022. URL https://arxiv.org/abs/2201.11903. · arXiv
  41. Laura Weidinger, John Mellor, Maribeth Rauh, Conor Griffin, Jonathan Uesato, Po-Sen Huang, Myra Cheng, Mia Glaese, Borja Balle, Atoosa Kasirzadeh, et al. Ethical and social risks of harm from language models. CoRR, abs/2112.04359, 2021. URL https://arxiv.org/abs/2112.04359. · arXiv
  42. Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona T. Diab, Xian Li, Xi Victoria Lin, et al. OPT: open pre-trained transformer language models. CoRR, abs/2205.01068, 2022. URL https://arxiv.org/abs/2205.01068. · arXiv
  43. Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. Bertscore: evaluating text generation with BERT. In ICLR, 2020. URL https://openreview.net/forum?id= SkeHuCVFDr.
  44. Qihuang Zhong, Liang Ding, Juhua Liu, Bo Du, and Dacheng Tao. Can chatgpt understand too? A comparative study on chatgpt and fine-tuned BERT. CoRR, abs/2302.10198, 2023. URL https://arxiv.org/abs/2302.10198. · arXiv
  45. Chunting Zhou, Graham Neubig, Jiatao Gu, Mona T. Diab, Francisco Guzmán, Luke Zettlemoyer, and Marjan Ghazvininejad. Detecting hallucinated content in conditional neural sequence generation. In Findings of ACL, 2021. URL https://doi.org/10.18653/v1/2021.findings-acl.120. · DOI · Paper page
  46. Pei Zhou, Karthik Gopalakrishnan, Behnam Hedayatnia, Seokhwan Kim, Jay Pujara, Xiang Ren, Yang Liu, and Dilek Hakkani-Tur. Think before you speak: explicitly generating implicit commonsense knowledge for response generation. In ACL, 2022. URL https://doi.org/10.18653/v1/ 2022.acl-long. · DOI
  47. Terry Yue Zhuo, Yujin Huang, Chunyang Chen, and Zhenchang Xing. Exploring AI ethics of chatgpt: A diagnostic analysis. CoRR, abs/2301.12867, 2023. URL https://arxiv.org/abs/2301. 12867. System: You are a description generator. You are given the start of a description. You return potential next sentences for the description. · DOI

Authors
Mündler, N., He, J., Jenko, S., & Vechev, M.
ORCID
Venue
arXiv
Publisher
arXiv
Published
2023
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2305.15852
arXiv
2305.15852
OpenAlex
W6947583425
Cited by
47
References
0
Citations by year
2026: 6 · 2025: 14 · 2024: 17 · 2023: 10
Type
posted content
Language
en
Open access
Licence
Locations
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • Repository for Publications and Research Data (ETH Zurich) (submittedVersion, other-oa)
  • arXiv (Cornell University) (cc-by)
  • ETH Zürich Research Collection
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
  • Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
Keywords
hallucinations, large language models, hallucination detection, hallucination mitigation, hallucination evaluation

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review