CoChat
Sign Up
Cited by 56

The DOI matches its Crossref record; 45 of the 45 references it cites were found in a registry or an index.

Know Your Limits: A Survey of Abstention in Large Language Models

Wen, Bingbing, Yao, Jihan, Feng, Shangbin, Xu, Chenjun, Tsvetkov, Yulia, Howe, Bill, Wang, Lucy Lu·2025·Transactions of the Association for Computational Linguistics
DOI 10.1162/tacl_a_00754
Open this paper in CoChat — free account

Abstract

Abstention, the refusal of large language models (LLMs) to provide an answer, is increasingly recognized for its potential to mitigate hallucinations and enhance safety in LLM systems. In this survey, we introduce a framework to examine abstention from three perspectives: the query, the model, and human values. We organize the literature on abstention methods, benchmarks, and evaluation metrics using this framework, and discuss merits and limitations of prior work. We further identify and motivate areas for future research, such as whether abstention can be achieved as a meta-capability that transcends specific tasks or domains, and opportunities to optimize abstention abilities in specific contexts. In doing so, we aim to broaden the scope and impact of abstention methodologies in AI systems.1

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1162/tacl_a_00754

Cites

45 of 45 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Flagged first, then by citations

Show all 45 references

Research works printed but not linked: 117

  1. Achiam, 2023, GPT-4 technical report, arXiv preprint arXiv:2303.08774 · arXiv
  2. Ahdritz, 2024, Distinguishing the knowable from the unknowable with language models, arXiv preprint arXiv:2402.03563 · arXiv
  3. Aliannejadi, 2019, Asking clarifying questions in open-domain information-seeking conversations, 475, 10.1145/3331184.3331265 · DOI
  4. Amayuelas, 2023, Knowledge of knowledge: Exploring known-unknowns uncertainty with large language models, arXiv preprint arXiv:2305.13712, 10.18653/v1/2024.findings-acl.383 · DOI
  5. Anwar, 2024, Foundational challenges in assuring alignment and safety of large language models, arXiv preprint arXiv:2404.09932 · arXiv
  6. Asai, 2021, Challenges in information-seeking QA: Unanswerable questions and paragraph retrieval, 1492, 10.18653/v1/2021.acl-long.118 · DOI
  7. Bhardwaj, 2024, Language models are Homer Simpson! Safety re-alignment of fine-tuned language models through task arithmetic, 10.18653/v1/2024.acl-long.762 · DOI
  8. Bianchi, 2024, Safety-tuned LLaMAs: Lessons from improving the safety of large language models that follow instructions
  9. Brahman, 2024, The art of saying no: Contextual noncompliance in language models, arXiv preprint arXiv:2407.12043 · arXiv
  10. Cao, 2023, Defending against alignment-breaking attacks via robustly aligned LLM, arXiv preprint arXiv:2309.14348 · arXiv
  11. Cao, 2024, Learn to refuse: Making large language models more controllable and reliable through knowledge scope limitation and refusal mechanism, 3628, 10.18653/v1/2024.emnlp-main.212 · DOI
  12. Chao, 2023, Jailbreaking black box large language models in twenty queries, arXiv preprint arXiv:2310.08419 · arXiv
  13. Chen, 2023, Jailbreaker in jail: Moving target defense for large language models, 29, 10.1145/3605760.3623764 · DOI
  14. Chen, 2024, INSIDE: LLMs’ internal states retain the power of hallucination detection
  15. Cheng, 2024, Can AI assistants know what they don’t know?
  16. Clark, 2020, TyDi QA: A benchmark for information- seeking question answering in typologically diverse languages, Transactions of the Association for Computational Linguistics, 8, 454, 10.1162/tacl_a_00317 · DOI
  17. Cole, 2023, Selectively answering ambiguous questions, arXiv preprint arXiv:2305.14613 · arXiv
  18. Dai, 2024, Safe RLHF: Safe reinforcement learning from human feedback
  19. Dasigi, 2021, A dataset of information-seeking questions and answers anchored in research papers, 4599, 10.18653/v1/2021.naacl-main.365 · DOI · Paper page
  20. Deng, 2024, Don’t just say “I don’t know”! self-aligning large language models for responding to unknown questions with explanations, 13652, 10.18653/v1/2024.emnlp-main.757 · DOI
  21. Duan, 2023, Shifting attention to relevance: Towards the uncertainty estimation of large language models, arXiv preprint arXiv:2307.01379, 10.18653/v1/2024.acl-long.276 · DOI
  22. ElSherief, 2021, Latent hatred: A benchmark for understanding implicit hate speech, 345, 10.18653/v1/2021.emnlp-main.29 · DOI
  23. Feng, 2024, Teaching LLMs to abstain across languages via multilingual feedback, arXiv preprint arXiv:2406.15948, 10.18653/v1/2024.emnlp-main.239 · DOI
  24. Gao, 2023, Scaling laws for reward model overoptimization, 10835
  25. Gehman, 2020, RealToxicityPrompts: Evaluating neural toxic degeneration in language models, 3356, 10.18653/v1/2020.findings-emnlp.301 · DOI · Paper page
  26. Guo, 2017, On calibration of modern neural networks, 1321
  27. Guo, 2024, Controllable preference optimization: Toward controllable multi-objective alignment, arXiv preprint arXiv:2402.19085 · arXiv
  28. Han, 2024, Wildguard: Open one-stop moderation tools for safety risks, jailbreaks, and refusals of llms, arXiv preprint arXiv:2406.18495 · arXiv
  29. Hestness, 2017, Deep learning scaling is predictable, empirically, arXiv preprint arXiv:1712.00409 · arXiv
  30. Hoffmann, 2022, Training compute-optimal large language models, arXiv preprint arXiv: 2203.15556 · arXiv
  31. Hou, 2023, Decomposing uncertainty for large language models through input clarification ensembling, arXiv preprint arXiv:2311.08718 · arXiv
  32. Huang, 2024, Catastrophic jailbreak of open-source LLMs via exploiting generation
  33. Huang, 2024, Calibrating long-form generations from large language models, arXiv preprint arXiv:2402.06544, 10.18653/v1/2024.findings-emnlp.785 · DOI
  34. Jain, 2023, Baseline defenses for adversarial attacks against aligned language models, arXiv preprint arXiv:2309.00614 · arXiv
  35. Ji, 2023, Beavertails: Towards improved safety alignment of LLM via a human-preference dataset
  36. Jin, 2019, PubMedQA: A dataset for biomedical research question answering, 2567, 10.18653/v1/D19-1259 · DOI · Paper page
  37. Jung, 2024, Trust or escalate: Llm judges with provable guarantees for human agreement, arXiv preprint arXiv:2407.18370 · arXiv
  38. Kamath, 2020, Selective question answering under domain shift, 5684, 10.18653/v1/2020.acl-main.503 · DOI
  39. Kang, 2024, Unfamiliar finetuning examples control how language models hallucinate, arXiv preprint arXiv:2403.05612 · arXiv
  40. Kim, 2024, Epistemology of language models: Do language models have holistic knowledge?, arXiv preprint arXiv:2403.12862, 10.18653/v1/2024.findings-acl.751 · DOI
  41. Kirk, 2023, The past, present and better future of feedback learning in large language models for subjective human preferences and values, 2409, 10.18653/v1/2023.emnlp-main.148 · DOI
  42. Kirk, 2023, Personalisation within bounds: A risk taxonomy and policy framework for the alignment of large language models with personalised feedback, arXiv preprint arXiv:2303.05453 · arXiv
  43. Kuhn, 2023, Semantic uncertainty: Linguistic invariances for uncertainty estimation in natural language generation
  44. Li, 2024, Salad-bench: A hierarchical and comprehensive safety benchmark for large language models, arXiv preprint arXiv:2402.05044, 10.18653/v1/2024.findings-acl.235 · DOI
  45. Li, 2024, MediQ: Question-asking LLMs for adaptive and reliable medical reasoning, arXiv preprint arXiv:2406.00922 · arXiv
  46. Liang, 2024, Learning to trust your feelings: Leveraging self-awareness in LLMs for hallucination mitigation, 10.18653/v1/2024.knowledgenlp-1.4 · DOI
  47. Lin, 2022, Teaching models to express their uncertainty in words, arXiv preprint arXiv:2205.14334 · arXiv
  48. Zi, 2023, ToxicChat: Unveiling hidden challenges of toxicity detection in real-world user-AI conversation, 4694, 10.18653/v1/2023.findings-emnlp.311 · DOI
  49. Lyu, 2024, Keeping LLMs aligned after fine-tuning: The crucial role of prompt templates
  50. Mazeika, 2024, Harmbench: A standardized evaluation framework for automated red teaming and robust refusal, arXiv preprint arXiv:2402.04249 · arXiv
  51. Min, 2020, AmbigQA: Answering ambiguous open-domain questions, 5783, 10.18653/v1/2020.emnlp-main.466 · DOI
  52. Mo, 2024, Fight back against jailbreaking via prompt adversarial tuning
  53. Neeman, 2023, DisentQA: Disentangling parametric and contextual knowledge with counterfactual question answering, 10056, 10.18653/v1/2023.acl-long.559 · DOI
  54. Phute, 2024, LLM self defense: By self examination, LLMs know they are being tricked
  55. Pisano, 2023, Bergeron: Combating adversarial attacks through a conscience-based alignment framework, arXiv preprint arXiv:2312.00029 · arXiv
  56. Qi, 2021, ONION: A simple and effective defense against textual backdoor attacks, 9558, 10.18653/v1/2021.emnlp-main.752 · DOI
  57. Qi, 2024, Fine-tuning aligned language models compromises safety, even when users do not intend to!
  58. Qiu, 2023, Latent jailbreak: A benchmark for evaluating text safety and output robustness of large language models, arXiv preprint arXiv:2307.08487 · arXiv
  59. Rafailov, 2023, Direct preference optimization: Your language model is secretly a reward model
  60. Rein, 2023, Gpqa: A graduate-level google-proof q&a benchmark, arXiv preprint arXiv:2311.12022 · arXiv
  61. Ren, 2023, Self-evaluation improves selective generation in large language models, 49
  62. Röttger, 2024, XSTest: A test suite for identifying exaggerated safety behaviours in large language models, 5377, 10.18653/v1/2024.naacl-long.301 · DOI · Paper page
  63. Röttger, 2024, Safetyprompts: A systematic review of open datasets for evaluating and improving large language model safety, arXiv preprint arXiv:2404.05399, 10.1609/aaai.v39i26.34975 · DOI
  64. Shah, 2023, Scalable and transferable black-box jailbreaks for language models via persona modulation
  65. Shao, 2021, Bddr: An effective defense against textual backdoor attacks, Computers & Security, 110, 102433, 10.1016/j.cose.2021.102433 · DOI
  66. Shen, 2023, “do anything now”: Characterizing and evaluating in-the-wild jailbreak prompts on large language models, arXiv preprint arXiv:2308.03825, 10.1145/3658644.3670388 · DOI · Paper page
  67. Shi, 2024, Safer-instruct: Aligning language models with automated preference data, 10.18653/v1/2024.naacl-long.422 · DOI
  68. Shrivastava, 2023, LLaMAs know what gpts don’t show: Surrogate models for confidence estimation, arXiv preprint arXiv:2311.08877 · arXiv
  69. Si, 2023, Getting moRE out of mixture of language model reasoning experts, 10.18653/v1/2023.findings-emnlp.552 · DOI
  70. Slobodkin, 2023, The curious case of hallucinatory (un)answerability: Finding truths in the hidden states of over- confident large language models, 3607, 10.18653/v1/2023.emnlp-main.220 · DOI
  71. Souly, 2024, A strongreject for empty jailbreaks, arXiv preprint arXiv:2402.10260 · arXiv
  72. Stengel-Eskin, 2024, Lacie: Listener-aware finetuning for confidence calibration in large language models, arXiv preprint arXiv:2405.21028 · arXiv
  73. Sun, 2023, Defending against backdoor attacks in natural language generation, 5257, 10.1609/aaai.v37i4.25656 · DOI
  74. Szegedy, 2016, Rethinking the inception architecture for computer vision, 10.1109/CVPR.2016.308 · DOI · Paper page
  75. Tian, 2023, Just ask for calibration: Strategies for eliciting calibrated confidence scores from language models fine-tuned with human feedback, 5433, 10.18653/v1/2023.emnlp-main.330 · DOI · Paper page
  76. Tomani, 2024, Uncertainty-based abstention in LLMs improves safety and reduces hallucinations, arXiv preprint arXiv:2404.10960 · arXiv
  77. Varshney, 2022, Investigating selective prediction approaches across several tasks in IID, OOD, and adversarial settings, 1995, 10.18653/v1/2022.findings-acl.158 · DOI
  78. Wang, 2024, Inferaligner: Inference-time alignment for harmlessness through cross-model guidance, arXiv preprint arXiv:2401.11206, 10.18653/v1/2024.emnlp-main.585 · DOI
  79. Wang, 2023, All languages matter: On the multilingual safety of large language models, arXiv preprint arXiv:2310.00905, 10.18653/v1/2024.findings-acl.349 · DOI
  80. Wang, 2024, Defending LLMs against jailbreaking attacks via backtranslation, arXiv preprint arXiv:2402.16459, 10.18653/v1/2024.findings-acl.948 · DOI
  81. Wang, 2023, Resolving knowledge conflicts in large language models, ArXiv, abs/2310.00935 · arXiv
  82. Wang, 2024, Do- not-answer: Evaluating safeguards in LLMs, 896
  83. Wang, 2024, SELF-GUARD: Empower the LLM to safeguard itself, 1648, 10.18653/v1/2024.naacl-long.92 · DOI
  84. Wei, 2022, Finetuned language models are zero-shot learners
  85. Wei, 2022, Chain-of-thought prompting elicits reasoning in large language models, 24824
  86. Wen, 2020, Batchensemble: An alternative approach to efficient ensemble and lifelong learning
  87. Wester, 2024, “as an AI language model, i cannot”: Investigating llm denials of user requests, 10.1145/3613904.3642135 · DOI
  88. Whitehead, 2022, Reliable visual question answering: Abstain rather than answer incorrectly, 148, 10.1007/978-3-031-20059-5_9 · DOI
  89. Xi, 2023, Defending pre-trained language models as few-shot learners against backdoor attacks
  90. Xiao, 2022, Uncertainty quantification with pre-trained language models: A large-scale empirical analysis, 10.18653/v1/2022.findings-emnlp.538 · DOI
  91. Xie, 2024, Sorry-bench: Systematically evaluating large language model safety refusal behaviors, arXiv preprint arXiv:2406.14598 · arXiv
  92. Xie, 2023, Defending ChatGPT against jailbreak attack via self-reminders, Nature Machine Intelligence, 5, 1486, 10.1038/s42256-023-00765-8 · DOI
  93. Ji, 2021, The art of abstention: Selective prediction and error regularization for natural language processing, 1040, 10.18653/v1/2021.acl-long.84 · DOI · Paper page
  94. Albert, 2021, Detoxifying language models risks marginalizing minority voices, 2390
  95. Guohai, 2023, Cvalues: Measuring the values of chinese large language models from safety to responsibility, arXiv preprint arXiv:2307.09705 · arXiv
  96. Rongwu, 2024, Knowledge conflicts for llms: A survey, arXiv preprint arXiv:2403.08319 · arXiv
  97. Yang, 2023, Alignment for honesty, arXiv preprint arXiv:2312.07000 · arXiv
  98. Yi, 2024, A survey on recent advances in LLM-Based multi-turn dialogue systems, arXiv preprint arXiv:2402.18013 · arXiv
  99. Yin, 2023, Do large language models know what they don’t know?, 8653, 10.18653/v1/2023.findings-acl.551 · DOI · Paper page
  100. Yong, 2023, Low-resource languages jailbreak gpt-4, arXiv preprint arXiv:2310.02446 · arXiv
  101. Yoshikawa, 2023, Selective-lama: Selective prediction for confidence-aware evaluation of language models, 1972, 10.18653/v1/2023.findings-eacl.150 · DOI
  102. Yuan, 2024, GPT-4 is too smart to be safe: Stealthy chat with LLMs via cipher
  103. Yuan, 2024, RigorLLM: Resilient guardrails for large language models against undesired content
  104. Zablotskaia, 2023, On uncertainty calibration and selective generation in probabilistic neural summarization: A benchmark study, 2980, 10.18653/v1/2023.findings-emnlp.197 · DOI
  105. Zeng, 2024, Autodefense: Multi-agent LLM defense against jailbreak attacks, arXiv preprint arXiv:2403.04783 · arXiv
  106. Zhang, 2024, Controllable safety alignment: Inference-time adaptation to diverse safety requirements, arXiv preprint arXiv:2410.08968 · arXiv
  107. Zhang, 2021, SituatedQA: Incorporating extra-linguistic contexts into QA, 7371, 10.18653/v1/2021.emnlp-main.586 · DOI
  108. Zhang, 2024, Self-alignment for factuality: Mitigating hallucinations in LLMs via self-evaluation, 10.18653/v1/2024.acl-long.107 · DOI
  109. Zhang, 2023, Defending large language models against jailbreaking attacks through goal prioritization, arXiv preprint arXiv:2311.09096, 10.18653/v1/2024.acl-long.481 · DOI
  110. Zhao, 2024, I could’ve asked that: Reformulating unanswerable questions, 4207, 10.18653/v1/2024.emnlp-main.242 · DOI
  111. Zhao, 2022, Calibrating sequence likelihood improves conditional language generation
  112. Zhao, 2024, Knowing what LLMs DO NOT know: A simple yet effective self- detection method, 7051, 10.18653/v1/2024.naacl-long.390 · DOI
  113. Zheng, 2024, Prompt-driven llm safeguarding via directed representation optimization, arXiv preprint arXiv:2401.18018 · arXiv
  114. Zhou, 2024, Relying on the unreliable: The impact of language models’ reluctance to express uncertainty, arXiv preprint arXiv:2401.06730, 10.18653/v1/2024.acl-long.198 · DOI
  115. Zhou, 2023, Navigating the grey area: How expressions of uncertainty and overconfidence affect language models, 5506, 10.18653/v1/2023.emnlp-main.335 · DOI · Paper page
  116. Zhou, 2023, Context-faithful prompting for large language models, 14544, 10.18653/v1/2023.findings-emnlp.968 · DOI
  117. Zhou, 2024, Defending jailbreak prompts via in-context adversarial game, arXiv preprint arXiv:2402 .13148, 10.18653/v1/2024.emnlp-main.1121 · DOI

Cited by

Sorted by citations

See all 56

Metadata

Authors
Wen, B., Yao, J., Feng, S., Xu, C., Tsvetkov, Y., Howe, B., & Wang, L. L.
Venue
Transactions of the Association for Computational Linguistics
Publisher
MIT Press
Published
2025
Volume
13
Pages
529-556
ISSN
2307-387X
DOI
10.1162/tacl_a_00754
Cited by
56
References
45
Language
en
Licence
creativecommons.org/licenses/by/4.0/ (vor, 182 d)

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review