Reducing conversational agents' overconfidence through linguistic calibration
Open accessAbstract
While improving neural dialogue agents' factual accuracy is the object of much research, another important aspect of communication, less studied in the setting of neural dialogue, is transparency about ignorance. In this work, we analyze to what extent state-of-the-art chit-chat models are linguistically calibrated in the sense that their verbalized expression of doubt (or confidence) matches the likelihood that the model's responses are factually incorrect (or correct). We find that these models are poorly calibrated, yet we show that likelihood of correctness can accurately be predicted. By incorporating such metacognitive features into the training of a controllable generation model, we obtain a dialogue agent with greatly improved linguistic calibration. While improving neural dialogue agents' factual accuracy is the object of much research, another important aspect of communication, less studied in the setting of neural dialogue, is transparency about ignorance. In this work, we analyze to what extent state-of-the-art chit-chat models are linguistically calibrated in the sense that their verbalized expression of doubt (or confidence) matches the likelihood that the model's responses are factually incorrect (or correct). We find that these models are poorly calibrated, yet we show that likelihood of correctness can accurately be predicted. By incorporating such metacognitive features into the training of a controllable generation model, we obtain a dialogue agent with greatly improved linguistic calibration.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Post-training should align the model to 'know what it knows' rather than add knowledge.
“We follow the principle that post-training should align the model to 'know what it knows' rather than add knowledge (Gekhman et al., 2024; Mielke et al., 2020).”
From The Llama 3 Herd of Models (2024) · cites this paper as Mielke et al., 2020
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · see citing works
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 48,293 · see citing works
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372 · see citing works
Grice, H. P. (1975) · Speech Acts · doi:10.1163/9789004368811_003 · Cited by 23,221 · see citing works
RoBERTa: A Robustly Optimized BERT Pretraining Approach
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019) · arXiv · doi:10.48550/arxiv.1907.11692 · Cited by 16,515 · see citing works
Advances in Neural Information Processing Systems 29
Teymur, O., Zygalakis, K., & Calderhead, B. (2016) · Cited by 13,881 · see citing works
Proceedings of the 24th international conference on Machine learning
(2007) · doi:10.1145/1273496 · Cited by 11,869 · see citing works
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., & Liu, P. J. (2019) · arXiv · doi:10.48550/arxiv.1910.10683 · Cited by 10,929 · see citing works
On the Dangers of Stochastic Parrots: Can Language Models Be Too Big? 🦜
Bender, E. M., Gebru, T., McMillan-Major, A., & Shmitchell, S. (2021) · Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3442188.3445922 · Cited by 6,988 · see citing works
DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter
Sanh, V., Debut, L., Chaumond, J., & Wolf, T. (2019) · arXiv · doi:10.48550/arxiv.1910.01108 · Cited by 4,653 · see citing works
Gaussian Error Linear Units (GELUs)
Hendrycks, D., & Gimpel, K. (2016) · arXiv · doi:10.48550/arxiv.1606.08415 · Cited by 3,537 · see citing works
On calibration of modern neural networks
Guo, C., Pleiss, G., Sun, Y., & Weinberger, K. Q. (2017) · arXiv · doi:10.48550/arxiv.1706.04599 · Cited by 3,435 · see citing works
Semantic Parsing on Freebase from Question-Answer Pairs
Berant, J., Chou, A., Frostig, R., & Liang, P. (2013) · Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d13-1160 · Cited by 2,318 · see citing works
Natural Questions: A benchmark for question answering research
Kwiatkowski, T., Palomaki, J., Redfield, O., Collins, M., Parikh, A., Alberti, C., Epstein, D., Polosukhin, I., Devlin, J., Lee, K., et al. (2019) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00276 · Cited by 2,069 · see citing works
Predicting good probabilities with supervised learning
Niculescu-Mizil, A., & Caruana, R. (2005) · Proceedings of the 22nd international conference on Machine learning - ICML '05 · doi:10.1145/1102351.1102430 · Cited by 1,884 · see citing works
TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension
Joshi, M., Choi, E., Weld, D., & Zettlemoyer, L. (2017) · Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p17-1147 · Cited by 1,690 · see citing works
DIALOGPT : Large-Scale Generative Pre-training for Conversational Response Generation
Zhang, Y., Sun, S., Galley, M., Chen, Y.-C., Brockett, C., Gao, X., Gao, J., Liu, J., & Dolan, B. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations · doi:10.18653/v1/2020.acl-demos.30 · Cited by 1,076 · see citing works
Baumgartner, J., Zannettou, S., Keegan, B., Squire, M., & Blackburn, J. (2020) · Proceedings of the International AAAI Conference on Web and Social Media · doi:10.1609/icwsm.v14i1.7347 · Cited by 881 · see citing works
Recipes for Building an Open-Domain Chatbot
Roller, S., Dinan, E., Goyal, N., Ju, D., Williamson, M., Liu, Y., Xu, J., Ott, M., Smith, E. M., Boureau, Y.-L., et al. (2021) · Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume · doi:10.18653/v1/2021.eacl-main.24 · Cited by 784 · see citing works
Self-citationCTRL: A Conditional Transformer Language Model for Controllable Generation
Keskar, N. S., McCann, B., Varshney, L. R., Xiong, C., & Socher, R. (2019) · arXiv · doi:10.48550/arxiv.1909.05858 · Cited by 751 · see citing works
Towards a Human-like Open-Domain Chatbot
Adiwardana, D., Luong, M.-T., So, D. R., Hall, J., Fiedel, N., Thoppilan, R., Yang, Z., Kulshreshtha, A., Nemade, G., Lu, Y., et al. (2020) · arXiv · doi:10.48550/arxiv.2001.09977 · Cited by 614 · see citing works
Wizard of Wikipedia: Knowledge-Powered Conversational agentsPreprint
Dinan, E., Roller, S., Shuster, K., Fan, A., Auli, M., & Weston, J. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1811.01241 · Cited by 610 · see citing works
Self-citationIgnorance and Uncertainty: Emerging Paradigms
Smithson, M. J. (2012) · Medical Entomology and Zoology · Cited by 476 · see citing works
The Role of Individual Differences in the Accuracy of Confidence Judgments
Pallier, G., Wilkinson, R., Danthiir, V., Kleitman, S., Knežević, G., Stankov, L., & Roberts, R. (2002) · The Journal of general psychology · doi:10.1080/00221300209602099 · Cited by 340 · see citing works
Juslin, P. (1994) · Organizational Behavior and Human Decision Processes · doi:10.1006/obhd.1994.1013 · Cited by 327 · see citing works
ParlAI: A Dialog Research Software Platform
Miller, A., Feng, W., Batra, D., Bordes, A., Fisch, A., Lu, J., Parikh, D., & Weston, J. (2017) · Conference on Empirical Methods in Natural Language Processing: System Demonstrations (EMNLP) · doi:10.18653/v1/d17-2014 · Cited by 326 · see citing works
Neural Text Generation with Unlikelihood Training
Welleck, S., Kulikov, I., Roller, S., Dinan, E., Cho, K., & Weston, J. (2019) · arXiv · doi:10.48550/arxiv.1908.04319 · Cited by 254 · see citing works
Self-citationWhat makes a good conversation? How controllable attributes affect human judgments
See, A., Roller, S., Kiela, D., & Weston, J. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1170 · Cited by 247 · see citing works
Calibration of Pre-trained Transformers
Desai, S., & Durrett, G. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.21 · Cited by 209 · see citing works
Topical-Chat: Towards Knowledge-Grounded Open-Domain Conversations
Gopalakrishnan, K., Hedayatnia, B., Chen, Q., Gottardi, A., Kwatra, S., Venkatesh, A., Gabriel, R., & Hakkani-Tür, D. (2019) · Interspeech 2019 · doi:10.21437/interspeech.2019-3079 · Cited by 207 · see citing works
Multiple-Attribute Text Rewriting.
Lample, G., Subramanian, S., Smith, E. M., Denoyer, L., Ranzato, M., & Boureau, Y.-L. (2018) · International Conference on Learning Representations · Cited by 181 · see citing works
Self-citationCan You Put it All Together: Evaluating Conversational Agents’ Ability to Blend Skills
Smith, E. M., Williamson, M., Shuster, K., Weston, J., & Boureau, Y.-L. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.183 · Cited by 179 · see citing works
Self-citationJiang, Z., Araki, J., Ding, H., & Neubig, G. (2021) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00407 · Cited by 177 · see citing works
Retrieve and Refine: Improved Sequence Generation Models For Dialogue
Weston, J., Dinan, E., & Miller, A. (2018) · Proceedings of the 2018 EMNLP Workshop SCAI: The 2nd International Workshop on Search-Oriented Conversational AI · doi:10.18653/v1/w18-5713 · Cited by 165 · see citing works
Self-citationSelective Question Answering under Domain Shift
Kamath, A., Jia, R., & Liang, P. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.503 · Cited by 153 · see citing works
Recipes for Safety in Open-domain ChatbotsPreprint
Xu, J., Ju, D., Li, M., Boureau, Y.-L., Weston, J., & Dinan, E. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.07079 · Cited by 130 · see citing works
Self-citationConfidence judgments in studies of individual differences
Stankov, L., & Crawford, J. D. (1996) · Personality and Individual Differences · doi:10.1016/s0191-8869(96)00130-4 · Cited by 129 · see citing works
Don’t Say That! Making Inconsistent Dialogue Unlikely with Unlikelihood Training
Li, M., Roller, S., Kulikov, I., Welleck, S., Boureau, Y.-L., Cho, K., & Weston, J. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.428 · Cited by 128 · see citing works
Self-citationEcological and person‐oriented aspects of metacognitive processes in test‐taking
Kleitman, S., & Stankov, L. (2001) · Applied Cognitive Psychology · doi:10.1002/acp.705 · Cited by 108 · see citing works
Calibration curves, scatterplots and the distinction between general knowledge and perceptual tasks
Stankov, L. (1998) · Learning and Individual Differences · doi:10.1016/s1041-6080(99)80141-1 · Cited by 62 · see citing works
Ravetz, J. R. (1993) · Science Communication · doi:10.1177/107554709301500203 · Cited by 62 · see citing works
Plug-and-Play Conversational Models
Madotto, A., Ishii, E., Lin, Z., Dathathri, S., & Fung, P. (2020) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2020.findings-emnlp.219 · Cited by 47 · see citing works
Controlling Style in Generated DialoguePreprint
Smith, E. M., Gonzalez-Rico, D., Dinan, E., & Boureau, Y.-L. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2009.10855 · Cited by 38 · see citing works
Self-citationNguyen, D., & Doğruöz, A. S. (2013) · The Association for Computational Linguistics · Cited by 32 · see citing works
Calibrating Structured Output Predictors for Natural Language Processing
Jagannatha, A., & Yu, H. Q. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.188 · Cited by 22 · see citing works
Flagged first, then by citations
1 of the 46 references couldn’t be matched to a known paper.
Research works printed but not linked: 1
- - 2078-2092. Association for Computational Linguistics.
Cited by
Sorted by citations
Survey of Hallucination in Natural Language Generation
Ji, Z., Lee, N., Frieske, R., Yu, T., Su, D., Xu, Y., Ishii, E., Bang, Y. J., Madotto, A., & Fung, P. (2023) · ACM Computing Surveys · doi:10.1145/3571730 · Cited by 5,025 · see citing works
Kim, S. S. Y., Liao, Q. V., Vorvoreanu, M., Ballard, S., & Vaughan, J. W. (2024) · The 2024 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3630106.3658941 · Cited by 147 · see citing works
Tian, K., Mitchell, E., Zhou, A., Sharma, A., Rafailov, R., Yao, H., Finn, C., & Manning, C. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.330 · Cited by 123 · see citing works
Do large language models have a legal duty to tell the truth?
Wachter, S., Mittelstadt, B., & Russell, C. (2024) · Royal Society Open Science · doi:10.1098/rsos.240197 · Cited by 85 · see citing works
Mirages. On Anthropomorphism in Dialogue Systems
Abercrombie, G., Curry, A., Dinkar, T., Rieser, V., & Talat, Z. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.290 · Cited by 83 · see citing works
FaithDial : A Faithful Benchmark for Information-Seeking Dialogue
Dziri, N., Kamalloo, E., Milton, S., Zai͏̈ane, O. R., Yu, M., Ponti, E. M., & Reddy, S. (2022) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00529 · Cited by 68 · see citing works
Shorinwa, O., Mei, Z., Lidard, J., Ren, A. Z., & Majumdar, A. (2025) · ACM Computing Surveys · doi:10.1145/3744238 · Cited by 85 · see citing works
To protect science, we must use LLMs as zero-shot translators
Mittelstadt, B., Wachter, S., & Russell, C. (2023) · Nature Human Behaviour · doi:10.1038/s41562-023-01744-0 · Cited by 58 · see citing works
Kuhn, L., Gal, Y., & Farquhar, S. (2023) · arXiv · doi:10.48550/arxiv.2302.09664 · Cited by 52 · see citing works
Towards Human-centered Proactive Conversational Agents
Deng, Y., Liao, L., Zheng, Z., Yang, G. H., & Chua, T. (2024) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3626772.3657843 · Cited by 51 · see citing works
Prompting is not a substitute for probability measurements in large language models
Hu, J., & Levy, R. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.306 · Cited by 60 · see citing works
Know Your Limits: A Survey of Abstention in Large Language Models
Wen, B., Yao, J., Feng, S., Xu, C., Tsvetkov, Y., Howe, B., & Wang, L. L. (2025) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00754 · Cited by 70 · see citing works
Navigating the Grey Area: How Expressions of Uncertainty and Overconfidence Affect Language Models
Zhou, K., Jurafsky, D., & Hashimoto, T. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.335 · Cited by 38 · see citing works
RHO: Reducing Hallucination in Open-domain Dialogues with Knowledge Grounding
Ji, Z., Liu, Z., Lee, N., Yu, T., Wilie, B., Zeng, M., & Fung, P. (2023) · Findings of the Association for Computational Linguistics: ACL 2023 · doi:10.18653/v1/2023.findings-acl.275 · Cited by 37 · see citing works
Lee, Y., Son, K., Kim, T. S., Kim, J., Chung, J. J. Y., Adar, E., & Kim, J. (2024) · The 2024 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3630106.3662681 · Cited by 36 · see citing works
Fostering effective hybrid human-LLM reasoning and decision making
Passerini, A., Gema, A. P., Minervini, P., Sayin, B., & Tentori, K. (2025) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2024.1464690 · Cited by 35 · see citing works
Evaluating large language models for accuracy incentivizes hallucinations
Kalai, A. T., Nachum, O., Vempala, S., & Zhang, E. (2026) · Nature · doi:10.1038/s41586-026-10549-w · Cited by 308 · see citing works
Generation Probabilities Are Not Enough: Uncertainty Highlighting in AI Code Completions
Vasconcelos, H., Bansal, G., Fourney, A., Liao, Q. V., & Vaughan, J. W. (2024) · ACM Transactions on Computer-Human Interaction · doi:10.1145/3702320 · Cited by 43 · see citing works
Diving Deep into Modes of Fact Hallucinations in Dialogue Systems
Das, S., Saha, S., & Srihari, R. K. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.48 · Cited by 27 · see citing works
Xu, Z., Song, T., & Lee, Y.-C. (2025) · International Journal of Human-Computer Studies · doi:10.1016/j.ijhcs.2025.103455 · Cited by 18 · see citing works
Chatbots With Attitude: Enhancing Chatbot Interactions Through Dynamic Personality Infusion
Kovačević, N., Boschung, T., Holz, C., Groß, M. H., & Wampfler, R. (2024) · ACM Conference on Conversational User Interfaces (CUI) · doi:10.1145/3640794.3665543 · Cited by 16 · see citing works
Creating General User Models from Computer Use
Shaikh, O., Sapkota, S., Rizvi, S., Horvitz, E., Park, J. S., Yang, D., & Bernstein, M. S. (2025) · ACM Symposium on User Interface Software and Technology (UIST) · doi:10.1145/3746059.3747722 · Cited by 22 · see citing works
Shen, K., & Kejriwal, M. (2025) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2025.112253 · Cited by 13 · see citing works
ChatGPT Incorrectness Detection in Software Reviews
Tanzil, M. H., Khan, J. Y., & Uddin, G. (2024) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3597503.3639194 · Cited by 12 · see citing works
Sestino, A., Rizzo, C., Irgang, L., & Stehlíková, B. (2025) · Business Process Management Journal · doi:10.1108/bpmj-07-2024-0655 · Cited by 13 · see citing works
Zhang, D., Zhang, X., Bansal, C., Las-Casas, P., Fonseca, R., & Rajmohan, S. (2024) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3663529.3663858 · Cited by 15 · see citing works
Training Models to Generate, Recognize, and Reframe Unhelpful Thoughts
Maddela, M., Ung, M., Xu, J., Madotto, A., Foran, H., & Boureau, Y.-L. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.763 · Cited by 11 · see citing works
Self-citationReliable Text-to-SQL with Adaptive Abstention
Chen, K., Chen, Y., Koudas, N., & Yu, X. (2025) · Proceedings of the ACM on Management of Data · doi:10.1145/3709719 · Cited by 12 · see citing works
Calibrated Interpretation: Confidence Estimation in Semantic Parsing
Stengel-Eskin, E., & Durme, B. V. (2022) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00598 · Cited by 48 · see citing works
Schmidt, S., Rolff, T., Voigt, H., Offe, M., & Steinicke, F. (2024) · ACM Symposium on User Interface Software and Technology (UIST) · doi:10.1145/3654777.3676454 · Cited by 8 · see citing works
Simple Token-Level Confidence Improves Caption Correctness
Petryk, S., Whitehead, S., Gonzalez, J. E., Darrell, T., Rohrbach, A., & Rohrbach, M. (2024) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv57701.2024.00564 · Cited by 8 · see citing works
Understanding Knowledge Drift in LLMs Through Misinformation
Fastowski, A., & Kasneci, G. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-82346-6_5 · Cited by 10 · see citing works
Passi, S., Dhanorkar, S., & Vorvoreanu, M. (2025) · Springer eBooks · doi:10.1007/978-981-95-3658-0_98 · Cited by 6 · see citing works
Examining epistemological challenges of large language models in law
Paseri, L., & Durante, M. (2025) · Cambridge Forum on AI Law and Governance · doi:10.1017/cfl.2024.7 · Cited by 5 · see citing works
Confidence Estimation For LLM-Based Dialogue State Tracking
Sun, Y.-J., Dey, S., Hakkani-Tür, D. Z., & Tür, G. (2024) · IEEE Spoken Language Technology Workshop (SLT) · doi:10.1109/slt61566.2024.10832237 · Cited by 8 · see citing works
Conformal Autoregressive Generation: Beam Search with Coverage Guarantees
Deutschmann, N., Alberts, M., & Martínez, M. R. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i10.29062 · Cited by 6 · see citing works
EfficientEQA: An Efficient Approach to Open-Vocabulary Embodied Question Answering
Cheng, K., Li, Z., Sun, X., Min, B., Bedi, A. S., & Bera, A. (2025) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros60139.2025.11247438 · Cited by 4 · see citing works
Refining medical large language models: key insights from instruction tuning
Alqahtani, M. Q., Al-Barakati, A. A., & Alotaibi, F. S. (2025) · PeerJ Computer Science · doi:10.7717/peerj-cs.3216 · Cited by 6 · see citing works
When to Read Documents or QA History: On Unified and Selective Open-domain QA
Lee, K., Han, S.-E., Hwang, S.-W., & Lee, M. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.401 · Cited by 4 · see citing works
Exploring In-Context Learning for Knowledge Grounded Dialog Generation
Chen, Q., Wu, W., & Li, S. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.675 · Cited by 4 · see citing works
Li, H., Bezemer, C., & Hassan, A. E. (2025) · IEEE/ACM International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP) · doi:10.1109/icse-seip66354.2025.00033 · Cited by 5 · see citing works
Wang, H., Qi, Z., Wang, N., Yu, Z., Li, Z., & Lv, W. (2025) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn64981.2025.11227721 · Cited by 2 · see citing works
CarICL: Mitigating causal hallucinations to enhance event causality identification
Feng, Y., Li, L., Qin, X., & Li, Z. (2025) · Information Processing & Management · doi:10.1016/j.ipm.2025.104500 · Cited by 2 · see citing works
Ren, J. F., Li, B., Xu, Z., Zhang, X., Fayek, H. M., & Li, X. (2026) · International World Wide Web Conference (WWW) · doi:10.1145/3774904.3792358 · Cited by 5 · see citing works
Towards In-Context Non-Expert Evaluation of Reflection Generation for Counselling Conversations
Wu, Z., Balloccu, S., Helaoui, R., Recupero, D. R., & Riboni, D. (2022) · Workshop on Generation, Evaluation and Metrics (GEM) · doi:10.18653/v1/2022.gem-1.9 · Cited by 2 · see citing works
Liu, K., Casper, S., Hadfield-Menell, D., & Andreas, J. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.291 · Cited by 2 · see citing works
Elsheikh, M., Schoch, N., Palacio, S., Strem, N., Stärk, K. D. C., & Hoernicke, M. (2025) · IEEE International Conference on Emerging Technologies and Factory Automation (ETFA) · doi:10.1109/etfa65518.2025.11205595 · Cited by 1 · see citing works
Robust Uncertainty Quantification for Factual Generation of Large Language Models
Zhang, Y., Yang, Z., & Zhou, L. (2025) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn64981.2025.11227634 · Cited by 1 · see citing works
Li, T. W., Song, Y., Sundaram, H., & Karahalios, K. G. (2026) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3772318.3791940 · Cited by 1 · see citing works
Improved Instruction Ordering in Recipe-Grounded Conversation
Le, D., Guo, R., Xu, W. H., & Ritter, A. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.561 · Cited by 5 · see citing works
An Improved Chinese Pause Fillers Prediction Module Based on RoBERTa
Yu, L. H., Zhou, X., & Niu, F. (2023) · Applied Sciences · doi:10.3390/app131910652
Estimating the Confidence of Language Model Generation using Training Data
Yoshikawa, H., & Okazaki, N. (2025) · Journal of Natural Language Processing · doi:10.5715/jnlp.32.91
Can I Trust You? Advancing GUI Task Automation with Action Trust Score
Mei, H., Gao, D., Wei, X., Yang, X., & Shou, M. Z. (2025) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3746027.3755618
Step-Aware Confidence Estimation in Natural Language Reasoning
Xia, C., Liu, Z., Chen, F., Zhou, H., & Wang, C. (2025) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn64981.2025.11228321
Overreliance on AI in Oil and Gas Cybersecurity: A Review of Risks and Mitigation Strategies
Alkhalaf, D., Shareef, R., Almousa, S., & Alshamrani, S. (2026) · IPTC Summit on AI for the Energy Industry · doi:10.2523/iptc-25174-ms
Wang, X., & Sarand, S. F. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.6321223
Towards Uncertainty-Aware Low-Bit Quantized LLMs for On-Device Inference
Sparrenberg, L., Schneider, T., Deußer, T., Berger, A., & Sifa, R. (2025) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata66926.2025.11400739
Socratic Elenchus-inspired multi-agent debate for mitigating hallucinations in large language models
Huang, J., Han, T., Yu, N., & Yi, X. (2026) · Expert Systems with Applications · doi:10.1016/j.eswa.2026.132208
Zheng, Z., Yu, C., Chen, W., Song, M., Liu, B., Liu, J., Wang, S., Wang, X. Q., Cai, J., & Shi, Y. (2026) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3772318.3790318
Beyond Answers: Trajectory Semantic Entropy for Reliable Uncertainty Quantification in LLMs
Song, B., Feng, Z., Wang, S., & Zhou, Y. (2026) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp55912.2026.11463008
Ojewale, V., Ryan, J., Venkatasubramanian, S., & Boykin, C. M. (2026) · Computers in Human Behavior Artificial Humans · doi:10.1016/j.chbah.2026.100307
Alsaleh, O. I. (2026) · Telecommunication Systems · doi:10.1007/s11235-026-01462-8
Radtke, A., & Schnaubert, L. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-29760-0_54
Beyond Benchmarks: A User-Centric Framework for Evaluating Large Language Models
Abdrabou, Y., Abdelrahman, Y., Bozkir, E., Mazen, Y., Alt, F., & Kasneci, E. (2026) · ACM Conference on Conversational User Interfaces (CUI) · doi:10.1145/3816046.3816227
Delaflor, M., & Toxtli, C. (2026) · ACM Conference on Conversational User Interfaces (CUI) · doi:10.1145/3816046.3816233
TRACE-QA: Task-routed constraint elimination for auditable multi-agent question answering
Lu, J., Chen, H., Zhu, Y., Xu, J., & Xu, A. (2026) · Journal of King Saud University - Computer and Information Sciences · doi:10.1007/s44443-026-01141-3 · Cited by 1 · see citing works
Wang, Z., & Hu, K. (2026) · ACM Transactions on Social Computing · doi:10.1145/3843742
Dynamic adaptive ensemble decoding under uncertainty for large language models
Ma, W., Feng, X., Huang, L., Huang, Y., Ye, Y., Feng, X., & Qin, B. (2026) · Knowledge-Based Systems · doi:10.1016/j.knosys.2026.116931
Liu, D., & Niu, X. (2026) · Journal of theoretical and applied electronic commerce research · doi:10.3390/jtaer21090291
Savant, S., Deshpande, U. U., & D’Silva, J. (2026) · Discover Artificial Intelligence · doi:10.1007/s44163-026-01750-x
Why we believe chatbots: trust calibration as a design problem
Jaidka, K., & Cai, M. (2026) · Frontiers in Psychology · doi:10.3389/fpsyg.2026.1935527 · Cited by 1 · see citing works
The circular economy, explained by AI
Morseletto, P., & Bocken, N. M. P. (2026) · Journal of Industrial Ecology · doi:10.1007/s44498-026-00192-z
Firt, E., & Nir, B. (2026) · Pragmatics and Society · doi:10.1075/ps.25161.fir
CoSu-UQ: Uncertainty Quantification for Reasoning via Confidence and Semantic Support
Feng, Y., Lin, Q., Zhang, Y., Yang, Z., & Zhou, L. (2026) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn69854.2026.11730028
Grattafiori, A., Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Vaughan, A., et al. (2024) · arXiv · doi:10.48550/arxiv.2407.21783 · Cited by 505 · see citing works
Self-citation
- Authors
- Mielke, S. J., Szlam, A., Dinan, E., & Boureau, Y.-L.
- Affiliations
- Johns Hopkins University (US)
- Meta (United States) (US)
- Corresponding author
- Mielke, Sabrina J.
- Szlam, Arthur
- Dinan, Emily
- Boureau, Y-Lan
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2020
- Volume
- 10
- Pages
- 857-872
- ISSN
- 2307-387X (ISSN-L)
- arXiv
- 2012.14983
- OpenAlex
- W4290994954
- Cited by
- 72
- References
- 46
- Citations by year
- 2026: 22 · 2025: 22 · 2024: 12 · 2023: 13 · 2022: 3
- FWCI
- 4.11
- Type
- journal article
- Language
- en
- Open access
- diamond
- doi.org
- Licence
- Locations
- Transactions of the Association for Computational Linguistics (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Speech and dialogue systems
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Intelligent Tutoring Systems and Adaptive Learning
- Keywords
- chatbots, overconfidence, metacognition, factual accuracy, controllable generation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
