CoChat
Sign Up
PreprintCited by 1Topic ModelingNatural Language Processing TechniquesAdversarial Robustness in Machine Learning

Do LLMs Know When to NOT Answer? Investigating Abstention Abilities of Large Language Models

Open access
Madhusudhan, Nishanth, Madhusudhan, Sathwik Tejaswi, Yadav, Vikas, Hashemi, Masoud·2024·arXiv
DOI 10.48550/arxiv.2407.16221arXiv 2407.16221
Open this paper in CoChat — free accountOpen PDF

Abstract

Abstention Ability (AA) is a critical aspect of Large Language Model (LLM) reliability, referring to an LLM's capability to withhold responses when uncertain or lacking a definitive answer, without compromising performance. Although previous studies have attempted to improve AA, they lack a standardised evaluation method and remain unsuitable for black-box models where token prediction probabilities are inaccessible. This makes comparative analysis challenging, especially for state-of-the-art closed-source commercial LLMs. This paper bridges this gap by introducing a black-box evaluation approach and a new dataset, Abstain-QA, crafted to rigorously assess AA across varied question types (answerable and unanswerable), domains (well-represented and under-represented), and task types (fact centric and reasoning). We also propose a new confusion matrix, the ''Answerable-Unanswerable Confusion Matrix'' (AUCM) which serves as the basis for evaluating AA, by offering a structured and precise approach for assessment. Finally, we explore the impact of three prompting strategies-Strict Prompting, Verbal Confidence Thresholding, and Chain-of-Thought (CoT)-on improving AA. Our results indicate that even powerful models like GPT-4, Mixtral 8x22b encounter difficulties with abstention; however, strategic approaches such as Strict prompting and CoT can enhance this capability.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2407.16221

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

No references could be resolved yet.

Research works printed but not linked: 34

  1. Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774. · arXiv
  2. Amos Azaria and Tom Mitchell. 2023. The internal state of an llm knows when its lying. arXiv preprint arXiv:2304.13734. · arXiv
  3. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901.
  4. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. 2021. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168. · arXiv
  5. Yu Gui, Ying Jin, and Zhimei Ren. 2024. Conformal alignment: Knowing when to trust foundation models with guarantees. arXiv preprint arXiv:2405.10301. · arXiv
  6. Sankalp Gulati, Joan Serra, Vignesh Ishwar, Sertan Sen- türk, and Xavier Serra. 2016. Phrase-based rāga recognition using vector space modeling. In 2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 66-70. IEEE.
  7. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2020. Measuring massive multitask language under- standing. arXiv preprint arXiv:2009.03300. · arXiv
  8. Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bam- ford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al. 2024. Mixtral of experts. arXiv preprint arXiv:2401.04088. · arXiv
  9. Mandar Joshi, Eunsol Choi, Daniel S Weld, and Luke Zettlemoyer. 2017. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehen- sion. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Vol- ume 1: Long Papers), pages 1601-1611.
  10. Daniel Khashabi, Sewon Min, Tushar Khot, Ashish Sabharwal, Oyvind Tafjord, Peter Clark, and Han- naneh Hajishirzi. 2020. Unifiedqa: Crossing for- mat boundaries with a single qa system. Preprint, arXiv:2005.00700. · arXiv
  11. Jannik Kossen, Jiatong Han, Muhammed Razzak, Lisa Schut, Shreshth Malik, and Yarin Gal. 2024. Seman- tic entropy probes: Robust and cheap hallucination detection in llms. arXiv preprint arXiv:2406.15927. · arXiv
  12. TM Krishna and Vignesh Ishwar. 2012. Carnatic mu- sic: Svara, gamaka, motif and raga identity. In Serra X, Rao P, Murthy H, Bozkurt B, editors. Proceed- ings of the 2nd CompMusic Workshop; 2012 Jul 12- 13; Istanbul, Turkey. Barcelona: Universitat Pompeu Fabra; 2012. Universitat Pompeu Fabra.
  13. Cheng Li, Jindong Wang, Yixuan Zhang, Kaijie Zhu, Wenxin Hou, Jianxun Lian, Fang Luo, Qiang Yang, and Xing Xie. 2023a. Large language models un- derstand and can be enhanced by emotional stimuli. arXiv preprint arXiv:2307.11760. · arXiv
  14. Yinheng Li, Shaofei Wang, Han Ding, and Hang Chen. 2023b. Large language models in finance: A sur- vey. In Proceedings of the Fourth ACM International Conference on AI in Finance, pages 374-382.
  15. Stephanie Lin, Jacob Hilton, and Owain Evans. 2021. Truthfulqa: Measuring how models mimic human falsehoods. arXiv preprint arXiv:2109.07958. · arXiv
  16. Sathwik Tejaswi Madhusudhan and Girish Chowdhary. 2019. Deepsrgm-sequence classification and ranking in indian classical music with deep learning. In Pro- ceedings of the 20th International Society for Music Information Retrieval Conference, pages 533-540.
  17. Alex Mallen, Akari Asai, Victor Zhong, Rajarshi Das, Daniel Khashabi, and Hannaneh Hajishirzi. 2023. When not to trust language models: Investigating effectiveness of parametric and non-parametric mem- ories. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Vol- ume 1: Long Papers), pages 9802-9822.
  18. Pouya Pezeshkpour and Estevam Hruschka. 2023. Large language models sensitivity to the order of options in multiple-choice questions. Preprint, arXiv:2308.11483. · arXiv
  19. S Samsekai Manjabhat, Shashidhar G Koolagudi, KS Rao, and Pravin Bhaskar Ramteke. 2017. Raga and tonic identification in carnatic music. Journal of New Music Research, 46(3):229-245.
  20. Rajeswari Sridhar and TV Geetha. 2009. Raga iden- tification of carnatic music for music information retrieval. International Journal of recent trends in Engineering, 1(1):571.
  21. Katherine Tian, Eric Mitchell, Allan Zhou, Archit Sharma, Rafael Rafailov, Huaxiu Yao, Chelsea Finn, and Christopher Manning. 2023. Just ask for cali- bration: Strategies for eliciting calibrated confidence scores from language models fine-tuned with human feedback. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Process- ing, pages 5433-5442, Singapore. Association for Computational Linguistics. · DOI · Paper page
  22. Christian Tomani, Kamalika Chaudhuri, Ivan Evti- mov, Daniel Cremers, and Mark Ibrahim. 2024. Uncertainty-based abstention in llms improves safety and reduces hallucinations. arXiv preprint arXiv:2404.10960. · arXiv
  23. Hugo Touvron, Louis Martin, Kevin Stone, Peter Al- bert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023. Llama 2: Open founda- tion and fine-tuned chat models. arXiv preprint arXiv:2307.09288. · arXiv
  24. Dennis Ulmer, Jes Frellsen, and Christian Hardmeier. 2022. Exploring predictive uncertainty and calibra- tion in NLP: A study on the impact of method & data scarcity. In Findings of the Association for Computa- tional Linguistics: EMNLP 2022, pages 2707-2735, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics. · DOI
  25. Neeraj Varshney and Chitta Baral. 2023. Post- abstention: Towards reliably re-attempting the abstained instances in qa. arXiv preprint arXiv:2305.01812. · arXiv
  26. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022. Chain-of-thought prompting elicits rea- soning in large language models. Advances in Neural Information Processing Systems, 35:24824-24837.
  27. Laura Weidinger, Jonathan Uesato, Maribeth Rauh, Conor Griffin, Po-Sen Huang, John Mellor, Amelia Glaese, Myra Cheng, Borja Balle, Atoosa Kasirzadeh, et al. 2022. Taxonomy of risks posed by language models. In Proceedings of the 2022 ACM Confer- ence on Fairness, Accountability, and Transparency, pages 214-229.
  28. Johannes Welbl, Nelson F Liu, and Matt Gardner. 2017. Crowdsourcing multiple choice science questions. In Proceedings of the 3rd Workshop on Noisy User- generated Text, pages 94-106.
  29. Lisa Wimmer, Yusuf Sale, Paul Hofman, Bernd Bischl, and Eyke Hüllermeier. 2023. Quantifying aleatoric and epistemic uncertainty in machine learning: Are conditional entropy and mutual information appro- priate measures? In Uncertainty in Artificial Intelli- gence, pages 2282-2292.
  30. PMLR.
  31. Miao Xiong, Zhiyuan Hu, Xinyang Lu, YIFEI LI, Jie Fu, Junxian He, and Bryan Hooi. 2024. Can LLMs express their uncertainty? an empirical evaluation of confidence elicitation in LLMs. In The Twelfth Inter- national Conference on Learning Representations.
  32. HanXiang Xu, ShenAo Wang, Ningke Li, Yanjie Zhao, Kai Chen, Kailong Wang, Yang Liu, Ting Yu, and HaoYu Wang. 2024a. Large language models for cyber security: A systematic literature review. arXiv preprint arXiv:2405.04760. · arXiv
  33. Tianyang Xu, Shujin Wu, Shizhe Diao, Xiaoze Liu, Xingyao Wang, Yangyi Chen, and Jing Gao Gao. 2024b. Sayself: Teaching llms to express confi- dence with self-reflective rationales. arXiv preprint arXiv:2405.20974v.
  34. Fanghua Ye, Mingming Yang, Jianhui Pang, Longyue Wang, Derek F Wong, Emine Yilmaz, Shuming Shi, and Zhaopeng Tu. 2024. Benchmarking llms via uncertainty quantification. arXiv preprint arXiv:2401.12794. · arXiv

Cited by

Sorted by citations

Authors
Madhusudhan, N., Madhusudhan, S. T., Yadav, V., & Hashemi, M.
Venue
arXiv
Publisher
arXiv
Published
2024
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2407.16221
arXiv
2407.16221
OpenAlex
W6966698728
Cited by
0
References
0
Type
posted content
Language
en
Open access
Licence
Locations
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • arXiv (Cornell University) (cc-by)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
  • Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
Keywords
large language models, abstention, uncertainty estimation, black-box evaluation, unanswerable questions, prompting strategies, chain-of-thought prompting, LLM reliability, GPT-4, Mixtral

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review