CoChat
Sign Up
PreprintCited by 8Topic ModelingAdversarial Robustness in Machine LearningNatural Language Processing Techniques

Trust Me, I'm Wrong: LLMs Hallucinate with Certainty Despite Knowing the Answer

Open access
Simhi, Adi, Itzhak, Itay, Barez, Fazl, Stanovsky, Gabriel, Belinkov, Yonatan·2025·arXiv
DOI 10.48550/arxiv.2502.12964arXiv 2502.12964
Open this paper in CoChat — free accountOpen PDF

Abstract

Prior work on large language model (LLM) hallucinations has associated them with model uncertainty or inaccurate knowledge. In this work, we define and investigate a distinct type of hallucination, where a model can consistently answer a question correctly, but a seemingly trivial perturbation, which can happen in real-world settings, causes it to produce a hallucinated response with high certainty. This phenomenon, which we dub CHOKE (Certain Hallucinations Overriding Known Evidence), is particularly concerning in high-stakes domains such as medicine or law, where model certainty is often used as a proxy for reliability. We show that CHOKE examples are consistent across prompts, occur in different models and datasets, and are fundamentally distinct from other hallucinations. This difference leads existing mitigation methods to perform worse on CHOKE examples than on general hallucinations. Finally, we introduce a probing-based mitigation that outperforms existing methods on CHOKE hallucinations. These findings reveal an overlooked aspect of hallucinations, emphasizing the need to understand their origins and improve mitigation strategies to enhance LLM safety. The code is available at https://github.com/technion-cs-nlp/Trust_me_Im_wrong .

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2502.12964

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Since LLM generations are prone to hallucinations, especially in long-context settings, the authors adopt a self-evaluation strategy to mitigate this issue.

    “Since LLM generations are prone to hallucinations [Simhi et al., 2025, Huang et al., 2025]-especially in long-context settings [Farquhar et al., 2024, Liu et al., 2025]-we adopt a self-evaluation strategy to mitigate this issue.”

    From DeepResearch Bench II: Diagnosing Deep Research Agents via Rubrics from Expert Reports (2026) · cites this paper as Simhi et al., 2025

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

No references could be resolved yet.

Research works printed but not linked: 64

  1. Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, and 1 others. 2023. Gpt-4 techni- cal report. arXiv preprint arXiv:2303.08774. · arXiv
  2. Maksym Andriushchenko, Francesco Croce, and Nico- las Flammarion. 2024. Jailbreaking leading safety- aligned llms with simple adaptive attacks. URL https://arxiv. org/abs/2404.02151.
  3. Amos Azaria and Tom Mitchell. 2023. The internal state of an llm knows when it's lying. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 967-976.
  4. Joris Baan, Nico Daheim, Evgenia Ilia, Dennis Ul- mer, Haau-Sing Li, Raquel Fernández, Barbara Plank, Rico Sennrich, Chrysoula Zerva, and Wilker Aziz. 2023. Uncertainty in natural language gener- ation: From theory to applications. arXiv preprint arXiv:2307.15703. · arXiv
  5. Patrice Béchard and Orlando Marquez Ayala. 2024. Reducing hallucination in structured outputs via retrieval-augmented generation. arXiv preprint arXiv:2404.08189. · arXiv
  6. Mohammad Beigi, Sijia Wang, Ying Shen, Zihao Lin, Adithya Kulkarni, Jianfeng He, Feng Chen, Ming Jin, Jin-Hee Cho, Dawei Zhou, and 1 others. 2024. Rethinking the uncertainty: A critical review and analysis in the era of large language models. arXiv preprint arXiv:2410.20199. · arXiv
  7. Lennart Bürger, Fred A Hamprecht, and Boaz Nadler. 2024. Truth is universal: Robust detection of lies in llms. arXiv preprint arXiv:2407.12831. · arXiv
  8. Sky CH-Wang, Benjamin Van Durme, Jason Eisner, and Chris Kedzie. 2023. Do androids know they're only dreaming of electric sheep? arXiv preprint arXiv:2312.17249. · arXiv
  9. Yung-Sung Chuang, Yujia Xie, Hongyin Luo, Yoon Kim, James R Glass, and Pengcheng He. 2023. Dola: Decoding by contrasting layers improves factuality in large language models. In The Twelfth International Conference on Learning Representations.
  10. Jeremy R Cole, Michael JQ Zhang, Daniel Gillick, Ju- lian Martin Eisenschlos, Bhuwan Dhingra, and Jacob Eisenstein. 2023. Selectively answering ambiguous questions. arXiv preprint arXiv:2305.14613. · arXiv
  11. Matthew Dahl, Varun Magesh, Mirac Suzgun, and Daniel E. Ho. 2024. Large legal fictions: Profiling legal hallucinations in large language models. ArXiv, abs/2401.01301. · arXiv
  12. Yuntian Deng, Wenting Zhao, Jack Hessel, Xiang Ren, Claire Cardie, and Yejin Choi. 2024. Wildvis: Open source visualizer for million-scale chat logs in the wild. arXiv preprint arXiv:2409.03753. · arXiv
  13. Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, and 1 others. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783. · arXiv
  14. Shangbin Feng, Weijia Shi, Yike Wang, Wenxuan Ding, Vidhisha Balachandran, and Yulia Tsvetkov. 2024. Don't hallucinate, abstain: Identifying llm knowl- edge gaps via multi-llm collaboration. arXiv preprint arXiv:2402.00367. · arXiv
  15. Jakob Gawlikowski, Cedrique Rovile Njieutcheu Tassi, Mohsin Ali, Jongseok Lee, Matthias Humt, Jianxi- ang Feng, Anna Kruspe, Rudolph Triebel, Peter Jung, Ribana Roscher, and 1 others. 2023. A survey of uncertainty in deep neural networks. Artificial Intel- ligence Review, 56(Suppl 1):1513-1589.
  16. Tao Ge, Xin Chan, Xiaoyang Wang, Dian Yu, Haitao Mi, and Dong Yu. 2024. Scaling synthetic data cre- ation with 1,000,000,000 personas. arXiv preprint arXiv:2406.20094. · arXiv
  17. Zorik Gekhman, Eyal Ben David, Hadas Orgad, Eran Ofek, Yonatan Belinkov, Idan Szpektor, Jonathan Herzig, and Roi Reichart. 2025. Inside-out: Hidden factual knowledge in llms. Preprint, arXiv:2503.15299. · arXiv
  18. Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Wein- berger. 2017. On calibration of modern neural net- works. In International conference on machine learn- ing, pages 1321-1330. PMLR.
  19. Jia He, Mukund Rungta, David Koleczek, Arshdeep Sekhon, Franklin X Wang, and Sadid Hasan. 2024. Does prompt formatting have any impact on llm per- formance? arXiv preprint arXiv:2411.10541. · arXiv
  20. Jinwen He, Yujia Gong, Kai Chen, Zijin Lin, Chengan Wei, and Yue Zhao. 2023. Llm factoscope: Uncov- ering llms' factual discernment through intermediate data analysis. arXiv preprint arXiv:2312.16374. · arXiv
  21. Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. 2020. Deberta: Decoding-enhanced bert with disentangled attention. arXiv preprint arXiv:2006.03654. · arXiv
  22. Mengting Hu, Zhen Zhang, Shiwan Zhao, Minlie Huang, and Bingzhe Wu. 2023. Uncertainty in natu- ral language processing: Sources, quantification, and applications. arXiv preprint arXiv:2306.04459. · arXiv
  23. Yuheng Huang, Jiayang Song, Zhijie Wang, Shengming Zhao, Huaming Chen, Felix Juefei-Xu, and Lei Ma. 2023. Look before you leap: An exploratory study of uncertainty measurement for large language models. arXiv preprint arXiv:2307.10236. · arXiv
  24. Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. 2023. Survey of halluci- nation in natural language generation. ACM Comput- ing Surveys, 55(12):1-38.
  25. Ziwei Ji, Lei Yu, Yeskendir Koishekenov, Yejin Bang, Anthony Hartshorn, Alan Schelten, Cheng Zhang, Pascale Fung, and Nicola Cancedda. 2025. Calibrat- ing verbal uncertainty as a linear feature to reduce hallucinations. arXiv preprint arXiv:2503.14477. · arXiv
  26. Albert Q Jiang, Alexandre Sablayrolles, Arthur Men- sch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guil- laume Lample, Lucile Saulnier, and 1 others. 2023. Mistral 7b. arXiv preprint arXiv:2310.06825. · arXiv
  27. Mandar Joshi, Eunsol Choi, Daniel S Weld, and Luke Zettlemoyer. 2017. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehen- sion. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Vol- ume 1: Long Papers), pages 1601-1611.
  28. Nitish Joshi, Javier Rando, Abulhair Saparov, Najoung Kim, and He He. 2023. Personas as a way to model truthfulness in language models. arXiv preprint arXiv:2310.18168. · arXiv
  29. Adam Tauman Kalai and Santosh S Vempala. 2023. Calibrated language models must hallucinate. arXiv preprint arXiv:2311.14648. · arXiv
  30. Jannik Kossen, Jiatong Han, Muhammed Razzak, Lisa Schut, Shreshth Malik, and Yarin Gal. 2024. Seman- tic entropy probes: Robust and cheap hallucination detection in llms. arXiv preprint arXiv:2406.15927. · arXiv
  31. Lorenz Kuhn, Yarin Gal, and Sebastian Farquhar. 2023. Semantic uncertainty: Linguistic invariances for un- certainty estimation in natural language generation. arXiv preprint arXiv:2302.09664. · arXiv
  32. Kenneth Li, Tianle Liu, Naomi Bashkansky, David Bau, Fernanda Viégas, Hanspeter Pfister, and Martin Wattenberg. 2024. Measuring and controlling per- sona drift in language model dialogs. arXiv preprint arXiv:2402.10962. · arXiv
  33. Kenneth Li, Oam Patel, Fernanda Viégas, Hanspeter Pfister, and Martin Wattenberg. 2023. Inference-time intervention: Eliciting truthful answers from a lan- guage model. NeurIPS.
  34. Edward Loper and Steven Bird. 2002. Nltk: The natural language toolkit. arXiv preprint cs/0205028. · arXiv
  35. Potsawee Manakul, Adian Liusie, and Mark Gales. 2023. Selfcheckgpt: Zero-resource black-box hallucina- tion detection for generative large language models. In Proceedings of the 2023 Conference on Empiri- cal Methods in Natural Language Processing, pages 9004-9017.
  36. Moran Mizrahi, Guy Kaplan, Dan Malkin, Rotem Dror, Dafna Shahaf, and Gabriel Stanovsky. 2024. State of what art? a call for multi-prompt llm evaluation. Transactions of the Association for Computational Linguistics, 12:933-949.
  37. Hadas Orgad, Michael Toker, Zorik Gekhman, Roi Re- ichart, Idan Szpektor, Hadas Kotek, and Yonatan Belinkov. 2024. Llms know more than they show: On the intrinsic representation of llm hallucinations. arXiv preprint arXiv:2410.02707. · arXiv
  38. Lorenzo Pacchiardi, Alex James Chan, Sören Minder- mann, Ilan Moscovitz, Alexa Yue Pan, Yarin Gal, Owain Evans, and Jan M Brauner. 2023. How to catch an ai liar: Lie detection in black-box llms by asking unrelated questions. In The Twelfth Interna- tional Conference on Learning Representations.
  39. Fabian Pedregosa, Gaël Varoquaux, Alexandre Gram- fort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vin- cent Dubourg, and 1 others. 2011. Scikit-learn: Ma- chine learning in python. the Journal of machine Learning research, 12:2825-2830.
  40. Gabrijela Perković, Antun Drobnjak, and Ivica Botički. 2024. Hallucinations in llms: Understanding and addressing challenges. In 2024 47th MIPRO ICT and Electronics Convention (MIPRO), pages 2084-2088. IEEE.
  41. Vipula Rawte, Prachi Priya, SM Tonmoy, SM Zaman, Amit Sheth, and Amitava Das. 2023. Exploring the relationship between llm hallucinations and prompt linguistic nuances: Readability, formality, and con- creteness. arXiv preprint arXiv:2309.11064. · arXiv
  42. Thomas Savage, John Wang, Robert J Gallo, Ab- dessalem Boukil, Vishwesh Patel, Seyed Amir Ah- mad Safavi-Naini, Ali Soroush, and Jonathan H Chen. 2024. Large language model uncertainty proxies: dis- crimination and calibration for medical diagnosis and treatment. Journal of the American Medical Infor- matics Association : JAMIA.
  43. Mrinank Sharma, Meg Tong, Tomasz Korbak, David Duvenaud, Amanda Askell, Samuel R Bowman, Esin DURMUS, Zac Hatfield-Dodds, Scott R Johnston, Shauna M Kravec, and 1 others. 2023. Towards understanding sycophancy in language models. In The Twelfth International Conference on Learning Representations.
  44. Xinyue Shen, Zeyuan Chen, Michael Backes, Yun Shen, and Yang Zhang. 2024. " do anything now": Charac- terizing and evaluating in-the-wild jailbreak prompts on large language models. In Proceedings of the 2024 on ACM SIGSAC Conference on Computer and Communications Security, pages 1671-1685.
  45. Aryan Shrivastava, Jessica Hullman, and Max Lam- parth. 2024. Measuring free-form decision-making inconsistency of language models in military crisis simulations. ArXiv, abs/2410.13204. · arXiv
  46. Chenglei Si, Zhe Gan, Zhengyuan Yang, Shuohang Wang, Jianfeng Wang, Jordan Lee Boyd-Graber, and Lijuan Wang. 2022. Prompting gpt-3 to be reliable. In The Eleventh International Conference on Learn- ing Representations.
  47. Adi Simhi, Jonathan Herzig, Idan Szpektor, and Yonatan Belinkov. 2024. Distinguishing ignorance from error in llm hallucinations. arXiv preprint arXiv:2410.22071. · arXiv
  48. K. Singhal, Shekoofeh Azizi, T. Tu, S. Mahdavi, Jason Wei, Hyung Won Chung, Nathan Scales, A. Tan- wani, H. Cole-Lewis, S. Pfohl, P. Payne, Martin G. Seneviratne, P. Gamble, C. Kelly, Nathaneal Scharli, Aakanksha Chowdhery, P. A. Mansfield, B. A. Y. Ar- cas, D. Webster, and 11 others. 2022. Large language models encode clinical knowledge. Nature, 620:172 -180. · DOI · Paper page
  49. Pittawat Taveekitworachai, Febri Abdullah, and Ruck Thawonmas. 2024. Null-shot prompting: rethinking prompting large language models with hallucination. In Proceedings of the 2024 Conference on Empiri- cal Methods in Natural Language Processing, pages 13321-13361.
  50. Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupati- raju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, and 1 others. 2024. Gemma 2: Improving open language models at a practical size. arXiv preprint arXiv:2408.00118. · arXiv
  51. Benedict Aaron Tjandra, Muhammed Razzak, Jan- nik Kossen, Kunal Handa, and Yarin Gal. 2024. Fine-tuning large language models to appropriately abstain with semantic entropy. arXiv preprint arXiv:2410.17234. · arXiv
  52. Christian Tomani, Kamalika Chaudhuri, Ivan Evti- mov, Daniel Cremers, and Mark Ibrahim. 2024. Uncertainty-based abstention in llms improves safety and reduces hallucinations. arXiv preprint arXiv:2404.10960. · arXiv
  53. Jiaqi Wang, Huan Zhao, Zhenyuan Yang, Peng Shu, Junhao Chen, Haobo Sun, Ruixi Liang, Shixin Li, Pengcheng Shi, Longjun Ma, Zongjia Liu, Zheng Liu, Tianyang Zhong, Yutong Zhang, Chong-Yi Ma, Xin Zhang, Tuo Zhang, Tianli Ding, Yudan Ren, and 3 others. 2024. Legal evalutions and challenges of large language models. ArXiv, abs/2411.10137. · arXiv
  54. Alexander Wei, Nika Haghtalab, and Jacob Steinhardt. 2023. Jailbroken: How does llm safety training fail? Advances in Neural Information Processing Systems, 36:80079-80110.
  55. Bingbing Wen, Jihan Yao, Shangbin Feng, Chenjun Xu, Yulia Tsvetkov, Bill Howe, and Lucy Lu Wang. 2024. Know your limits: A survey of abstention in large language models. arXiv preprint arXiv:2407.18418. · arXiv
  56. Yijun Xiao and William Yang Wang. 2019. Quantifying uncertainties in natural language processing tasks. In Proceedings of the AAAI conference on artificial intelligence, volume 33, pages 7322-7329.
  57. Hongshen Xu, Zixv yang, Zichen Zhu, Kunyao Lan, Zi- han Wang, Mengyue Wu, Ziwei Ji, Lu Chen, Pascale Fung, and Kai Yu. 2025. Delusions of large language models. Preprint, arXiv:2503.06709. · arXiv
  58. Rongwu Xu, Brian S Lin, Shujian Yang, Tianqi Zhang, Weiyan Shi, Tianwei Zhang, Zhixuan Fang, Wei Xu, and Han Qiu. 2023. The earth is flat be- cause...: Investigating llms' belief towards misinfor- mation via persuasive conversation. arXiv preprint arXiv:2312.09085. · arXiv
  59. Yasin Abbasi Yadkori, Ilja Kuzborskij, András György, and Csaba Szepesvári. 2024a. To believe or not to believe your llm. arXiv preprint arXiv:2406.02543. · arXiv
  60. Yasin Abbasi Yadkori, Ilja Kuzborskij, David Stutz, An- drás György, Adam Fisch, Arnaud Doucet, Iuliya Be- loshapka, Wei-Hung Weng, Yao-Yuan Yang, Csaba Szepesvári, and 1 others. 2024b. Mitigating llm hal- lucinations via conformal abstention. arXiv preprint arXiv:2405.01563. · arXiv
  61. Yongjin Yang, Haneul Yoo, and Hwaran Lee. 2024. Maqa: Evaluating uncertainty quantification in llms regarding data uncertainty. arXiv preprint arXiv:2408.06816. · arXiv
  62. Jia-Yu Yao, Kun-Peng Ning, Zhen-Hui Liu, Mu-Nan Ning, and Li Yuan. 2023. Llm lies: Hallucinations are not bugs, but features as adversarial examples. arXiv preprint arXiv:2310.01469. · arXiv
  63. Xi Ye and Greg Durrett. 2022. The unreliability of explanations in few-shot prompting for textual rea- soning. Advances in neural information processing systems, 35:30378-30392.
  64. Gal Yona, Roee Aharoni, and Mor Geva. 2024. Can large language models faithfully express their in- trinsic uncertainty in words? arXiv preprint arXiv:2405.16908. · arXiv

Cited by

Sorted by citations

Authors
Simhi, A., Itzhak, I., Barez, F., Stanovsky, G., & Belinkov, Y.
Affiliations
  • Technion – Israel Institute of Technology (IL)
  • University of Oxford (GB)
  • Hebrew University of Jerusalem (IL)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2025
Pages
14665-14688
DOI
10.48550/arxiv.2502.12964
arXiv
2502.12964
OpenAlex
W4416034653
Cited by
8
References
0
Citations by year
2026: 7 · 2025: 1
FWCI
7.63
Type
conference paper
Language
en
Open access
Licence
Locations
  • Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) (publishedVersion, cc-by)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
Keywords
large language model hallucination, hallucination mitigation, prompt perturbation, LLM safety

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review