Quantifying Uncertainty in Answers from any Language Model and Enhancing their Trustworthiness
Open accessAbstract
We introduce BSDETECTOR, a method for detecting bad and speculative answers from a pretrained Large Language Model by estimating a numeric confidence score for any output it generated.Our uncertainty quantification technique works for any LLM accessible only via a black-box API, whose training data remains unknown.By expending a bit of extra computation, users of any LLM API can now get the same response as they would ordinarily, as well as a confidence estimate that cautions when not to trust this response.Experiments on both closed and open-form Question-Answer benchmarks reveal that BSDETECTOR more accurately identifies incorrect LLM responses than alternative uncertainty estimation procedures (for both GPT-3 and ChatGPT).By sampling multiple responses from the LLM and considering the one with the highest confidence score, we can additionally obtain more accurate responses from the same LLM, without any extra training steps.In applications involving automated evaluation with LLMs, accounting for our confidence scores leads to more reliable evaluation in both human-in-the-loop and fullyautomated settings (across both GPT 3.5 and 4).... Observed Consistency + Ans...
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The faithfulness of LLMs in establishing judgment is established in previous works.
“The faithfulness of LLMs in establishing judgment is established in previous works33–35.”
From AgentClinic: a multimodal benchmark for tool-using clinical AI agents (2026) · cites this paper as [34]
Section 5 (page unavailable) says LLMs are often used to judge answer correctness, but explicitly notes that the reliability of those judgments remains in question. Section 6.3 (page unavailable) reports more reliable average evaluations after low-confidence judgments are discarded. The paper therefore supports improving judgment reliability under a specific method, not the unqualified claim that LLM faithfulness in making judgments is established. (LLM, conf=0.83)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
WizardLM: Empowering Large Pre-Trained Language Models to Follow Complex InstructionsPreprint
Xu, C., Sun, Q., Zheng, K., Geng, X.-B., Zhao, P., Feng, J., Tao, C., Lin, Q.-W., & Jiang, D. (2023) · International Conference on Learning Representations · doi:10.48550/arxiv.2304.12244 · Cited by 1,362 · see citing works
Training Verifiers to Solve Math Word Problems
Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., et al. (2021) · arXiv · doi:10.48550/arxiv.2110.14168 · Cited by 575 · see citing works
A gentle introduction to conformal prediction and distribution-free uncertainty quantification
Angelopoulos, A. N., & Bates, S. (2021) · arXiv · doi:10.48550/arxiv.2107.07511 · Cited by 240 · see citing works
Language Models (Mostly) Know What They Know
Kadavath, S., Conerly, T., Askell, A., Henighan, T., Drain, D., Perez, E., Schiefer, N., Hatfield-Dodds, Z., DasSarma, N., Tran-Johnson, E., et al. (2022) · arXiv · doi:10.48550/arxiv.2207.05221 · Cited by 187 · see citing works
Peng, B., Li, C., He, P., Galley, M., & Gao, J. (2023) · arXiv · doi:10.48550/arxiv.2304.03277 · Cited by 183 · see citing works
Bayesian Recurrent Neural Networks
Fortunato, M., Blundell, C., & Vinyals, O. (2017) · arXiv · doi:10.48550/arxiv.1704.02798 · Cited by 145 · see citing works
Generating with Confidence: Uncertainty Quantification for Black-box Large Language ModelsPreprint
Lin, Z., Trivedi, S., & Sun, J. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2305.19187 · Cited by 133 · see citing works
When do you need Chain-of-Thought Prompting for ChatGPT?Preprint
Chen, J.-H., Chen, L., Huang, H., & Zhou, T.-Y. (2023) · arXiv.org · doi:10.48550/arxiv.2304.03262 · Cited by 65 · see citing works
Self-citationInstructZero: Efficient Instruction Optimization for Black-Box Large Language ModelsPreprint
Chen, L., Chen, J., Goldstein, T., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2306.03082 · Cited by 11 · see citing works
Self-citation
Flagged first, then by citations
Research works printed but not linked: 20
- Jiuhai Chen, Lichang Chen, Chen Zhu, and Tianyi Zhou. 2023b. How many demonstrations do you need for in-context learning? In Findings of the Association for Computational Linguistics: EMNLP 2023, Sin- gapore, December 6-10, 2023, pages 11149-11159. Association for Computational Linguistics.
- Yarin Gal and Zoubin Ghahramani. 2016a. Dropout as a bayesian approximation: Representing model uncertainty in deep learning. In Proceedings of the 33nd International Conference on Machine Learning, ICML 2016, New York City, NY, USA, June 19-24, 2016, volume 48 of JMLR Workshop and Conference Proceedings, pages 1050-1059. JMLR.org.
- Yarin Gal and Zoubin Ghahramani. 2016b. A theoret- ically grounded application of dropout in recurrent neural networks. In Advances in Neural Information Processing Systems 29: Annual Conference on Neu- ral Information Processing Systems 2016, December 5-10, 2016, Barcelona, Spain, pages 1019-1027.
- Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Wein- berger. 2017. On calibration of modern neural net- works. In Proceedings of the 34th International Con- ference on Machine Learning, ICML 2017, Sydney, NSW, Australia, 6-11 August 2017, volume 70 of Proceedings of Machine Learning Research, pages 1321-1330. PMLR.
- Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. 2021. Deberta: decoding-enhanced bert with disentangled attention. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenRe- view.net.
- Dan Hendrycks and Kevin Gimpel. 2017. A baseline for detecting misclassified and out-of-distribution ex- amples in neural networks. In 5th International Con- ference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net.
- Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Yejin Bang, Andrea Madotto, and Pascale Fung. 2023. Survey of halluci- nation in natural language generation. ACM Comput. Surv., 55(12):248:1-248:38. · DOI · Paper page
- Mandar Joshi, Eunsol Choi, Daniel S. Weld, and Luke Zettlemoyer. 2017. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehen- sion. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, ACL 2017, Vancouver, Canada, July 30 -August 4, Volume 1: Long Papers, pages 1601-1611. Association for Computational Linguistics. · DOI · Paper page
- Lorenz Kuhn, Yarin Gal, and Sebastian Farquhar. 2023. Semantic uncertainty: Linguistic invariances for un- certainty estimation in natural language generation. 5194 In The Eleventh International Conference on Learn- ing Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net.
- Volodymyr Kuleshov, Nathan Fenner, and Stefano Er- mon. 2018. Accurate uncertainties for deep learning using calibrated regression. In Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmässan, Stockholm, Sweden, July 10-15, 2018, volume 80 of Proceedings of Ma- chine Learning Research, pages 2801-2809.
- PMLR.
- Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. 2017. Simple and scalable pre- dictive uncertainty estimation using deep ensembles. In Advances in Neural Information Processing Sys- tems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pages 6402-6413.
- Shiyu Liang, Yixuan Li, and R. Srikant. 2018. En- hancing the reliability of out-of-distribution image detection in neural networks. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 -May 3, 2018, Con- ference Track Proceedings. OpenReview.net.
- Stephanie Lin, Jacob Hilton, and Owain Evans. 2022. Teaching models to express their uncertainty in words. Trans. Mach. Learn. Res., 2022.
- Andrey Malinin and Mark J. F. Gales. 2021. Uncertainty estimation in autoregressive structured prediction. In 9th International Conference on Learning Represen- tations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net.
- Potsawee Manakul, Adian Liusie, and Mark J. F. Gales. 2023. Selfcheckgpt: Zero-resource black-box hal- lucination detection for generative large language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Process- ing, EMNLP 2023, Singapore, December 6-10, 2023, pages 9004-9017. Association for Computational Linguistics.
- Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel M. Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F. Christiano. 2020. Learn- ing to summarize with human feedback. In Advances in Neural Information Processing Systems 33: An- nual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual.
- Katherine Tian, Eric Mitchell, Allan Zhou, Archit Sharma, Rafael Rafailov, Huaxiu Yao, Chelsea Finn, and Christopher D. Manning. 2023. Just ask for cali- bration: Strategies for eliciting calibrated confidence scores from language models fine-tuned with human feedback. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Process- ing, EMNLP 2023, Singapore, December 6-10, 2023, pages 5433-5442. Association for Computational Linguistics.
- Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V. Le, Ed H. Chi, Sharan Narang, Aakanksha Chowd- hery, and Denny Zhou. 2023. Self-consistency improves chain of thought reasoning in language models. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net.
- Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V. Le, and Denny Zhou. 2022. Chain-of-thought prompting elicits reasoning in large language models. In Ad- vances in Neural Information Processing Systems 35: Annual Conference on Neural Information Process- ing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, November 28 -December 9, 2022.
Cited by
Sorted by citations
Non-Determinism of “Deterministic” LLM System Settings in Hosted Environments
Atıl, B., Aykent, S., Chittams, A., Fu, L., Passonneau, R. J., Radcliffe, E., Rajagopal, G. R., Sloan, A., Tudrej, T., Ture, F., et al. (2025) · Proceedings of the 5th Workshop on Evaluation and Comparison of NLP Systems · doi:10.18653/v1/2025.eval4nlp-1.12 · Cited by 36
Content moderation by LLM: from accuracy to legitimacy
Huang, T. (2025) · Artificial Intelligence Review · doi:10.1007/s10462-025-11328-1 · Cited by 24 · see citing works
AgentClinic: a multimodal benchmark for tool-using clinical AI agents
Schmidgall, S., Ziaei, R., Harris, C., Kim, J. W., Reis, E. P., Jopling, J., & Moor, M. (2026) · npj Digital Medicine · doi:10.1038/s41746-026-02674-7 · Cited by 11 · see citing works
- Authors
- Chen, J., & Mueller, J.
- ORCID
- Mueller, Jonas
- Venue
- Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)
- Published
- 2024
- Pages
- 5186-5200
- ISSN
- 0736-587X (ISSN-L)
- OpenAlex
- W4402671817
- Cited by
- 33
- References
- 9
- Citations by year
- 2026: 10 · 2025: 18 · 2024: 4 · 2023: 1
- FWCI
- 9.44
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Proceedings of the conference - Association for Computational Linguistics. Meeting (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- uncertainty quantification, large language models, confidence estimation, answer reliability, question answering, GPT-3, ChatGPT
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
