How Can We Know When Language Models Know? On the Calibration of Language Models for Question Answering
Open accessThe DOI matches its Crossref record; 72 of the 72 references it cites were found in a registry or an index.
Abstract
Recent works have shown that language models (LM) capture different types of knowledge regarding facts or common sense. However, because no model is perfect, they still fail to provide appropriate answers in many cases. In this paper, we ask the question, “How can we know when language models know, with confidence, the answer to a particular query?” We examine this question from the point of view of calibration, the property of a probabilistic model’s predicted probabilities actually being well correlated with the probabilities of correctness. We examine three strong generative models—T5, BART, and GPT-2—and study whether their probabilities on QA tasks are well calibrated, finding the answer is a relatively emphatic no. We then examine methods to calibrate such models to make their confidence scores correlate better with the likelihood of correctness through fine-tuning, post-hoc probability modification, or adjustment of the predicted outputs or inputs. Experiments on a diverse range of datasets demonstrate the effectiveness of our methods. We also perform analysis to study the strengths and limitations of these methods, shedding light on further improvements that may be made in methods for calibrating LMs. We have released the code at https://github.com/jzbjyb/lm-calibration.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source doesn’t support the claim as written.
In the citing paper
“Another line of work that sheds light on the nature of hallucinations shows that LMs know when they’re hallucinating [1,3,18,20].”
From Calibrated Language Models Must Hallucinate (2024) · cites this paper as [18]
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredThe cited source doesn’t support the claim as written.
In the citing paper
“Jiang et al. [51] observed the probabilities of pre-trained LMs (e.g., BART, T5, and GPT-2) on QA tasks are well calibrated.”
From Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing (2023) · cites this paper as [51]
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Past work on uncertainty for LLMs has focused on simpler settings, such as classifiers16,17 and regressors18,19, whereas the most exciting applications of LLMs relate to free-form generations.”
From Detecting hallucinations in large language models using semantic entropy (2024) · cites this paper as [16]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“find only a weak correlation between the log-likelihood models assign to their answer and the answer's correctness.”
From Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generation (2023) · cites this paper as Jiang et al. (2021)
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“More complex decision-making is currently bottlenecked by issues such as over-confidence and miscalibration (Jiang et al., 2021; Braverman et al., 2020; Chen et al., 2022), misalignment with human values or bias (Liang et al., 2021; Feng et al., 2023), hallucinations in self-evaluation (Shinn et al., 2023), and lack of human-in-the-loop mechanisms in face of uncertainties (Nguyen et al., 2022a; R”
From Cognitive Architectures for Language Agents (2023) · cites this paper as Jiang et al., 2021
8 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · cited 2×
“We found that larger LMs usually achieve both higher accuracy and better calibration performance, which is contradictory to the observation in image clas-sification (Guo et al., 2017) where larger models such as ResNet (He et al., 2016) are no longer well calibrated compared to smaller models like LeNet (Lecun et al.,…”
MethodVaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · cited 1×
“The T5 model is a sequence-to-sequence model with both encoder and decoder using trans-formers (Vaswani et al., 2017), and the largest version has 11 billion parameters, allowing it to realize state-of-the-art performance on tasks such as question answering and natural language understanding…”
BackgroundGradient-based learning applied to document recognition
Lecun, Y., Bottou, L., Bengio, Y., & Haffner, P. (1998) · Proceedings of the IEEE · doi:10.1109/5.726791 · Cited by 58,805 · cited 2×
“We found that larger LMs usually achieve both higher accuracy and better calibration performance, which is contradictory to the observation in image clas-sification (Guo et al., 2017) where larger models such as ResNet (He et al., 2016) are no longer well calibrated compared to smaller models like LeNet (Lecun et al.,…”
MethodXGBoost: A Scalable Tree Boosting System
Chen, T., & Guestrin, C. (2016) · Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2939672.2939785 · Cited by 54,569 · cited 2×
“We follow previous works (Jagannatha and Yu, 2020; Dong et al., 2018) and use gradient boosted decision trees (Chen and Guestrin, 2016) as our regres-sor to estimate the confidence based on features.”
MethodInfluentialBERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325 · cited 3×
“LMs are now a ubiquitous tool in not only natu-ral language generation, but also natural language understanding (NLU), where they are largely used for unsupervised representation learning in pre-trained models such as BERT (Devlin et al., 2019).”
MethodInfluentialLanguage Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
Kaugeranna, R., Kaugeranna, E., & Aion, (. S. 4. 6. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.giscience.2023.43 · Cited by 13,690
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., & Liu, P. J. (2019) · arXiv · doi:10.48550/arxiv.1910.10683 · Cited by 10,929 · cited 3×
“No-tably, large models trained on massive crawls of internet text (such as T5 (Raffel et al., 2019) and GPT-3 (Brown et al., 2020)) have been shown to be able to perform quite sophisticated knowledge-based tasks simply through prompting the model to predict the next words given a particular cue.”
BackgroundMethodScaling Laws for Neural Language Models
Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., & Amodei, D. (2020) · arXiv · doi:10.48550/arxiv.2001.08361 · Cited by 9,550 · cited 1×
“Unlike ResNet trained on CIFAR-100, the training of LMs is not bottlenecked by the dataset, and larger LMs have a stronger capacity to model text distribution and memorize facts, which leads to better calibration performance overall (Kaplan et al., 2020).”
MethodLewis, M., Liu, Y., Goyal, N., Ghazvininejad, M., Mohamed, A., Levy, O., Stoyanov, V., & Zettlemoyer, L. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.703 · Cited by 8,821
Research works printed but not linked: 4
- Hewitt, 2019, A structural probe for finding syntax in word representations, 4129
- Ng, 2019, Facebook fair’s WMT19 news translation task submission, 314
- Quach, 2020, Researchers made an OpenAI GPT-3 medical chatbot as an experiment. It told a mock patient to kill themselves, The Register
- Radford, 2019, Language models are unsupervised multitask learners, OpenAI Blog, 1
Cited by
Items 201–300 of 307
Pretrained LLMs Learn Multiple Types of UncertaintyPreprint
Cohen, R., Fahn, O., & de Melo, G. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2505.21218 · Cited by 2
“Common approaches to calibration are to perform various transformations on a model’s output logits [Desai and Durrett, 2020, Jiang et al., 2021] and measuring uncertainty [e.g., see Kuhn et al., 2023].”
Detecting Student Intent for Chat-Based Intelligent Tutoring SystemsPreprint
Cutler, E., Levonian, Z., & Christie, S. T. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2502.15096 · Cited by 5
“Incorporating research on probing models for their certainty [12,7] might enable the wider use of LLM-based intent detection models.”
Calibrating Semantic Uncertainty from Observable Language-Model ProbabilitiesPreprint
Dixon, M. F. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2607.17447 · Cited by 2
“Continuation likelihoods can support calibration in question answering, and fitted models sometimes distinguish what they know from what they do not [Jiang et al., 2021, Kadavath et al., 2022].”
How Much Does a Reasoning Summary Reveal? An Observability Ladder for Large Language ModelsPreprint
Algaba, A., Carlon, F., Delcon, L., Ballon, M., Verbruggen, B., & Ginis, V. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2608.02089 · Cited by 3
MAE I Trust Myself? Self-Evaluating VLA Action Generation with Markov Attention EntropyPreprint
Aniri, Yilin, C., Bi, J., Jin, Z., Wang, Y., Bian, X., Tresp, V., Shen, F., Chua, T.-S., & Ma, Y. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2608.16697 · Cited by 4
“Calibration-based work evaluates whether model confidence is aligned with answer correctness, including probability-based calibration and uncertainty expressed through natural-language confidence statements [20, 28, 39].”
Unveiling Inner Doubts beyond ChatGPT’s Apparent Overconfidence
Hamroun, A., Amouyel, P., Bentegeac, R., Kuchcinski, G., & Guellec, B. L. (2025) · Radiology · doi:10.1148/radiol.241766 · Cited by 1
Wang, M., Li, Z., Zhao, X., & Guo, Q. (2025) · Information Processing & Management · doi:10.1016/j.ipm.2025.104422 · Cited by 2
CADRAG: Credibility-Aware Dynamic Retrieval-Augmented Generation for LLMs
He, L., & Wei, Z. (2025) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn64981.2025.11228947 · Cited by 1
AN APPROACH TO ASSESSING THE QUALITY OF PENTEST SCENARIO GENERATION USING LARGE LANGUAGE MODELS
Konev, A., & Игоревна, П. Т. (2025) · Voprosy kiberbezopasnosti · doi:10.21681/2311-3456-2025-6-147-157 · Cited by 1
AutoTheme: A Multi-Agent Framework for Inductive Thematic Analysis with LLMs
Wahbeh, A., El-Gayar, O., Al-Ramahi, M. A., Nasralah, T., & Elnoshokaty, A. (2026) · Hawaii International Conference on System Sciences (HICSS) · doi:10.24251/hicss.2026.214 · Cited by 1
Hallucination detection in large language models based on multi-granularity consistency
Zhang, J., Ma, X., Yang, Q., Cui, T., Chen, X., Lin, Q., & Zhao, B. (2026) · Journal of Intelligent Information Systems · doi:10.1007/s10844-026-01047-6 · Cited by 1
“Recent studies have proposed to inspire confidence through self-assessment cues (Jiang et al., 2021; Xiong et al., 2024), in which the model estimates its own correctness through responses such as “P(True)”.”
Indonesian Hoax News Detection Using IndoBERT and Knowledge Graph
Maulidya, L., Wasista, S., & Ruswiansari, M. (2026) · Indonesian Journal of Computer Science · doi:10.33022/ijcs.v15i4.5195 · Cited by 1
Calibration of the Multiple Choice Machine Reading Comprehension
Xing, L., Hu, Y., Xie, Y., Peng, W., Sun, Y., & Zhang, C. (2022) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn55064.2022.9892434 · Cited by 1
“In the Natural Language Processing (NLP) area, towards calibrating NLP models, previous works utilize temperature-based scaling [14], [15], feature-based decision trees, and MaxProb [16] to calibrate models in the postprocessing manner.”
MethodTwo Examples are Better than One: Context Regularization for Gradient-based Prompt Tuning
Ha, H., Jung, S., Park, J., Seo, M., Hwang, S.-W., & Chun, B.-G. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.206 · Cited by 1
“…pre-trained LMs to numerous downstream tasks such as Natural Language Inference (Schick and Schütze, 2020), text classification (Gao et al., 2021; Puri and Catanzaro, 2019; Schick and Schütze, 2020), question answering (Jiang et al., 2021; Khashabi et al., 2020; Raffel et al., 2020) and many more.”
BackgroundCross-task Knowledge Transfer for Extremely Weakly Supervised Text Classification
Park, S., Kim, K., & Lee, J. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.328 · Cited by 2
“Such overconfidence in label quality estimates can hinder downstream classifier performance (Wei et al., 2022; Jiang et al., 2021).”
BackgroundEnhancing Large Model Document Question Answering Through Retrieval Augmentation
Zeng, J., Zheng, R., Wang, C., Xue, W., Yu, X., & Zhang, T. (2024) · International Conference on Artificial Intelligence and Power Systems (AIPS) · doi:10.1109/aips64124.2024.00057 · Cited by 1
“To address this issue, some methods [12,13,14,15,16] adopt external knowledge to alleviate the illusions of large models.”
BackgroundMethodHigher-order representation in AI
Butlin, P. (2026) · Philosophy and the Mind Sciences · doi:10.33735/phimisci.2026.12032 · Cited by 2
Towards Language Modelling in the Speech Domain Using Sub-word Linguistic UnitsPreprint
Katakkar, A., & Black, A. W. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2111.00610 · Cited by 1
“Studies that probe text neural language models for real world knowledge have established methods for understanding what information is stored in these LMs and at what point during training they begin to know it [6, 7, 8].”
A Taxonomy for Understanding and Identifying Uncertainty in AI-Generated ResponsesPreprint
Prabhudesai, S., Goldstein, D. G., Hofman, J. M., & Rothschild, D. M. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.4836380 · Cited by 1
A Monte Carlo Framework for Calibrated Uncertainty Estimation in Sequence PredictionPreprint
Yang, Q., Zhu, W., Keslin, J., Zanna, L., Rudner, T. G. J., & Fernandez‐Granda, C. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2410.23272 · Cited by 2
“Monte Carlo methods are also used for uncertainty estimation on large language model outputs [Malinin and Gales, 2021, Jiang et al., 2021, Kuhn et al., 2023, Xiong et al., 2024], where multiple sequences are generated to assess the confidence of factual outputs and mitigate hallucinations.”
MethodBuilding Understandable Messaging for Policy and Evidence Review (BUMPER) with AIPreprint
Rosenfeld, K., Sonnewald, M., Jindal, S. J., McCarthy, K. A., & Proctor, J. L. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2407.12812 · Cited by 1
Quantifying Uncert-AI-nty: Testing the Accuracy of LLMs’ Confidence JudgmentsPreprint
Cash, T. N., Oppenheimer, D. M., Christie, S., & Devgan, M. (2025) · PsyArXiv (OSF Preprints) · doi:10.31234/osf.io/47df5_v3 · Cited by 1
Gu, J., Pang, L., Shen, H., & Cheng, X. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2601.17865 · Cited by 1
“Calibration-based concepts can assess LLM uncertainty; [18] compares the calibration effects of verbalized probabilities versus probabilities extracted from log-its, and [14] confirms that model probabilities significantly deviate from actual accuracy rates in traditional QA tasks.”
Dixon, M. F. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2607.23130 · Cited by 1
“Language-model work documents miscalibration across token, sequence and answer levels (Braverman et al., 2020; Guo et al., 2017; Jiang et al., 2021); elicited confidence can differ from native probabilities (Tian et al., 2023; Giovannotti and Gammerman, 2024).”
READY or Not: Reliable Enterprise Agent DeploymentPreprint
Chatrath, V., Zhu, B., Fan, J., Pu, G., Tiwari, S. D., Dan, S., Young, R., Yuan, Li, Yao, Y., et al. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2609.02095 · Cited by 2
CausalVerify: End-to-End Verification of Causal Analyses by Language ModelsPreprint
Zhang, Y., Correia, R., Parra, I. M., & Xie, Y. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2609.07944 · Cited by 2
When Confidence Signals Disagree: Local and Global Confidence in Autoregressive Language ModelsPreprint
Amador, J. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2609.16933 · Cited by 1
“In language models, Jiang et al. [7] show that probability-based confidence can vary in reliability across question types and model scales, while Zhao et al. [22] study the relationship between sequence likelihood and generation quality.”
A Novel Gradient Accumulation Method for Calibration of Named Entity Recognition Models
Jouet, G., Duhart, C., Staiano, J., Rousseaux, F., & de Runz, C. (2022) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn55064.2022.9892324
“The Expected Calibration Error (ECE) [9] is one of the most popular calibration measures experimented with in various use cases, e.g. Question-Answering (QA) [12, 24], Computer Vision (CV) [23, 24] and Natural Language Processing (NLP) in general [13, 16].”
BackgroundMethodCorpus-Based Task-Specific Relation Discovery
Ramanan, K. (2023) · doi:10.18653/v1/2023.matching-1.5
“Prompting can also be used to probe the inherent relational knowledge of pretrained language models by aggregating the masked language model generations from multiple hand-crafted prompts (Jiang et al., 2021).”
BackgroundEnhancing Genetic Prompt Search Using Language Model Probability ExploitationPreprint
Song, Z., & Huang, S. (2024) · Research Square · doi:10.21203/rs.3.rs-4511610/v1
Gadiraju, V. M., Tsai, H.-Y., Wu, H., Singha, M., Huang, C.-H. S., Liu, G., Chang, S. Y., & Wu, Y. (2024) · IEEE International Symposium on Broadband Multimedia Systems and Broadcasting (BMSB) · doi:10.1109/bmsb62888.2024.10608341
SENTENCE REPRESENTATION USING LSTM FOR FINDING QUESTION
Linh, D. K., & Huy, T. Q. (2024) · Vinh University Journal of Science · doi:10.56824/vujs.2024a063a
Efficient pipeline for retrieval-augmented generation system under big data
Yu, R., Yang, Y., Zhou, J., & Wu, F. (2025) · Scientia Sinica Informationis · doi:10.1360/ssi-2024-0331
ScenarioQA: Evaluating Test Scenario Reasoning Capabilities of Large Language Models
Sinha, S., Paranjape, I., & Whitehead, J. (2024) · IEEE International Conference on Intelligent Transportation Systems (ITSC) · doi:10.1109/itsc58415.2024.10919946
“A few advantages are the capability of in-context learning and reasoning as a result of which they are capable of arriving at solutions to problems without pre-written instructions [7].”
Quantifying Uncert-AI-nty: Testing the Accuracy of LLMs’ Confidence JudgmentsPreprint
Cash, T. N., Oppenheimer, D. M., Christie, S., & Devgan, M. (2025) · PsyArXiv (OSF Preprints) · doi:10.31234/osf.io/47df5_v2
Robust Fine-Tuning and Adaptation of Zero-Shot Models via Adaptive Weight-Space Ensembling
Döbler, M., Feil, M., Marsden, R. A., & Yang, B. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-91672-4_21
Guariso, D., Adewoyin, R. A., Aguilar, G. R., Guerrero, O. A., & Davies, A. R. (2025) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2025.103203
Evaluating Log-Likelihood for Confidence Estimation in LLM-Based Multiple-Choice Question Answering
Boseak, C. (2025) · Innovative Journal of Applied Science · doi:10.70844/ijas.2025.2.29
Stable LLM Ensemble: Interaction Between Example Representativeness and Diversity
Niimi, J. (2026) · Lecture notes in networks and systems · doi:10.1007/978-3-032-20897-2_46
Optimización de ChatGPT para la clasificación de noticias falsas mediante FineTuning
Alcívar, C., Tomalá, D. F. R., & Bernabé, M. D. C. T. (2025) · Revista Científica Multidisciplinar G-nerando · doi:10.60100/rcmg.v6i2.816
“Sin embargo, se requiere una investigación continua para abordar las limitaciones actuales y mejorar la robustez del modelo (Jiang et al., 2021).”
Research on Judgment Questions Based on Label Guided Prompt Generation and Adapter Mapping
Peng, X., & Zhang, Y. (2025) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn64981.2025.11228332
“Jiang [8] introduced a machine translation method, translating candidate words into another language and back to increase diversity, which improves model accuracy.”
Jostmann, M. (2024) · URN:NBN Resolver for Germany & Switzerland (German National Library) · doi:10.17879/25968508220 · Cited by 2
Enhancing Text Representation by Minimizing Bias for Downstream Natural Language Processing Tasks
Zhang, L. (2024) · UNSWorks (University of New South Wales, Sydney, Australia) · doi:10.26190/unsworks/30770
Behzad, S. (2024) · DigitalGeorgetown (Georgetown University Library) · Cited by 1
Towards Efficient and Robust Knowledge-Intensive NLP Systems
Wu, Y. (2025) · UCL Discovery (University College London)
Tamkivi, K. H. (2025) · DSpace repository (University of Tartu)
Acknowledging Valued Perspectives! User-Assisted Summarization by Leveraging Opinions and LLMs
Kumar, R., Gupta, T., Kalra, K., & Saha, S. (2026) · IEEE Transactions on Artificial Intelligence · doi:10.1109/tai.2026.3654914
“Its proven performance across multiple NLP tasks [40], [41], [42], [43] reinforces its suitability for OACSF.”
ReToP: Learning to Rewrite Electronic Health Records for Clinical Prediction
Lovón-Melgarejo, J., Moreno, J. G., Damase‐Michel, C., & Tamine, L. (2026) · ACM International Conference on Web Search and Data Mining (WSDM) · doi:10.1145/3773966.3777969 · Cited by 2
ReToP: Learning to Rewrite Electronic Health Records for Clinical PredictionPreprint
Lovón-Melgarejo, J., Moreno, J. G., Damase‐Michel, C., & Tamine, L. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2601.19286
Suo, J., Fang, J., Wang, H., & Xiao, Y. (2026) · Research Square · doi:10.21203/rs.3.rs-8872718/v1
AdaGen: Adaptive Domain-Specific Classification via Zero-Shot Dataset Generation
Xu, Z.-H., Liang, F.-Y., Zhou, H.-Y., & Wang, C. (2025) · Proceedings · doi:10.1109/icdm65498.2025.00088
“Studies demonstrate that PLMs can generate data without human annotations [25] by leveraging carefully designed prompts [26]–[28].”
Moros-Daza, A., Jubiz‐Diaz, M., Coneo, L., & Voß, S. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.6360426
Jasem, A. N. A. A. (2026) · International Journal of Research · doi:10.26643/ijr/22
A Formal Framework for Uncertainty Analysis of Text Generation with Large Language ModelsPreprint
Herbold, S., & Lemmerich, F. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2603.26363
“To capture a basic notion of uncertainty by the probability of the last token (Jiang et al., 2021), we add a single token filter f st v ′ .”
Barkovich, A. A. (2026) · International Conference on Artificial Intelligence in Information and Communication (ICAIIC) · doi:10.1109/icaiic68212.2026.11454416
“The fundamental objective in this context is the strategic creation of a robust instrumental base —founded on the frequency and distribution of linguistic items [22].”
Facing Data Scarcity in Dialogues for Discourse Structure Discovery and Prediction
Li, C. (2023) · theses.fr (ABES) · doi:10.70675/c379d1f5z5324z4865z9c13z84c9508836f1 · Cited by 7
Prompt-based Relation Extraction for Pharmacovigilance
Yeh, H.-S. (2024) · theses.fr (ABES) · doi:10.70675/762a3598ze739z4c47za72ez62e5a986fd1f · Cited by 1
Contributions to evaluating and improving the faithfulness for text generation
Zhang, K. (2025) · theses.fr (ABES) · doi:10.70675/28453a20z78f2z4fa2za6edz700065e69e97
Less Data Approximates More: Earning Faithful Confidence in High-Stakes DomainsPreprint
Ma, H., Zhen, L. Y., Yang, G., Chen, Y., Ma, Y., Chua, T.-S., & Chang, E. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2604.08454
“Prior work estimates confidence via token-level entropy (Farquhar et al., 2024), calibration metrics (Minderer et al., 2021), and probabilistic approximations (Xiong et al., 2024), yet LLMs often remain miscalibrated and overconfident on incorrect answers (Kadavath et al., 2022; Jiang et al., 2021).”
Empowering University AI Assistants: Improving LLM Performance with RAG and Self-evaluation
Shnaider, P., Chernysheva, A., Говоров, А. И., & Khlopotov, M. (2026) · Lecture notes in networks and systems · doi:10.1007/978-3-032-13153-9_3
Cai, Z., Ye, A., Liao, F., Xu, G., & Wu, H. (2026) · Expert Systems with Applications · doi:10.1016/j.eswa.2026.132766
ECUAS$_n$: A family of metrics for principled evaluation of uncertainty-augmented systemsPreprint
Estienne, L., Ernst, E., Vera, M., Piantanida, P., & Ferrer, L. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2605.20490
“These metrics include the area under the ROC curve (AUC) [15, 91, 90, 19], expected calibration error (ECE) [43, 74, 47, 26], Brier score [45, 77, 57, 80], and cross-entropy [52, 60, 62, 40].”
Jostmann, M., & Winkelmann, H. (2026) · Lecture notes in information systems and organisation · doi:10.1007/978-3-032-08489-7_15
Yagci, E. A., & Yildiz, S. (2026) · Research Square · doi:10.21203/rs.3.rs-9691516/v1
Alsaleh, O. I. (2026) · Telecommunication Systems · doi:10.1007/s11235-026-01462-8
Token Mines: A Defense Against Agents and Large Language Models
Pasternak, A., Ashkenazy, T., Arad, I., Weiss, R., & Mirsky, Y. (2026) · doi:10.1145/3803629.3803674
“…substantial body of NLP research studies the robustness and reliability of language models under distribution shift, noisy inputs, or out-of-distribution prompts, typically evaluating task performance degradation or calibration behavior rather than failures in the generation process itself [4, 11].”
Chinese Cyberbullying Detection Based on Hierarchical Multi-Committee Ensemble
Hou, Y., Liu, W., Wang, S., & Xiong, S. (2026) · Information · doi:10.3390/info17060565
“Kong et al. and Jiang et al. proposed calibration frameworks based on manifold regularization and post-hoc fine-tuning, respectively, to improve probability estimation for out-of-distribution (OOD) data and question-answering tasks [24,25].”
Liu, C., Qian, L., Jadhav, C. D., & Zhao, J. Z. (2026) · IEEE Conference on Computer Communications (INFOCOM) · doi:10.1109/infocom59046.2026.11571647
“Defensive work so far has focused on content filtering [20], [21] or response calibration [22]–[24].”
Narayankeri, S., Sapar, N., Buchade, A., Taralekar, S. S., & Nalavade, J. E. (2026) · International Conference on Computing, Communication and Green Engineering (CCGE) · doi:10.1109/ccge67142.2026.11581753
“Jiang et al. discovered that confidence scores do not reliably show the true level of accuracy [11].”
Multicalibration for LLM-based Code Generation
Campos, V., Kuschnereit, R., & Ulges, A. (2026) · doi:10.1145/3786175.3788347 · Cited by 2
“A key challenge in our setup is selecting an appropriate scoring method to derive initial confidences ˆ 𝑝 as a starting point for calibration.”
Harnessing noisy LLM annotations: Confidence-calibrated node selection on text-attributed graphs
Fang, Z., Du, S., Wu, Z. W., Cai, Z., Tan, Y., Wang, S., & Lin, Z. (2026) · Pattern Recognition · doi:10.1016/j.patcog.2026.114379
TRIAGE: Trustworthy Retrieval Instrumentation And Graph EvaluationPreprint
Tahmasebimoradi, A., Schott, L., & Royer, M. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2607.03447
“Second, TCS relies on LLM token log-probabilities as a confidence proxy, which are known to be imperfectly calibrated for factual correctness [30,29]; a mis-calibrated TCS may flag correct triples or pass incorrect ones.”
Knowledgeless Language Models: Suppressing Parametric Recall for Evidence-Grounded Language ModelingPreprint
Cohen, R., Carré, Y., Lechtenbörger, N., Droste, H., Kerschke, L., Biswas, R., de Melo, G., & Buys, J. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2607.12831
“Prior approaches to calibration often operate at the logit level through post-hoc transformations [Desai and Durrett, 2020, Jiang et al., 2021], or rely on uncertainty estimation methods [Kuhn et al., 2023].”
Delaflor, M., & Toxtli, C. (2026) · ACM Conference on Conversational User Interfaces (CUI) · doi:10.1145/3816046.3816233
Wu, J., Li, J., Ning, Z., & Wang, L. (2026) · Machine Learning · doi:10.1007/s10994-026-07109-8
“Prior work on QA calibra - tion shows that model confidence can be poorly aligned with correctness, especially under distribution shifts or hallucination prone settings (Jiang et al., 2021 ).”
Zhang, V. (2026) · Stout in Agriculture and Animal Science · doi:10.61424/xbzn8d61
TrAC: Trace-Conditioned Answer Consistency for Efficient Uncertainty Quantification in LLMsPreprint
Yu, D., Jiang, L., Xu, R., & Wang, G. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2608.00422
Tavakolian, M. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.7222978
Liu, Y., Huang, H., Yang, Y., Zeng, Z., Zhou, Y., Wu, Z., & Gao, Y. (2026) · Expert Systems with Applications · doi:10.1016/j.eswa.2026.133921
AI Evaluation Should Measure Verification Cost, Not Correctness AlonePreprint
Crescitelli, V., Immediato, G., Persia, F., & Costantini, S. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.7263200 · Cited by 1
Liu, H. Y., Wei, Y. N., & Zhu, P. H. (2026) · Advanced Electromagnetics · doi:10.7716/aem.v15i3.3180
Liu, O. (2026) · Stout in Medical Science Biology and Chemistry · doi:10.61424/k0qrad59
Towards principled knowledge editing methods for large language model reasoning
Yao, Y., Qin, J., Zhang, N., Xu, H., Zhu, Y., Yu, Z., Wang, M., Tang, Y., Gu, J.-C., Deng, S., et al. (2026) · Nature Machine Intelligence · doi:10.1038/s42256-026-01276-y · Cited by 15
“This subsection presents a second framework, reasoning-augmented belief revision, which treats causal justification, contextual scaffolding and consistency verification as mechanisms for making edits coherent; event-based and virtual-world editing studies motivate parts of this design but do not…”
Retrieval-augmented LLM framework for automated construction site layout planning
Li, R., Lin, H., Chi, H.-L., Li, F., Yi, W., & Chen, L. (2026) · Automation in Construction · doi:10.1016/j.autcon.2026.107211
Mendez, J., Cargnelutti, M., Lowry-Duda, D., Brobston, C., Ismail, S., Leppert, G., Watson, A., & Zittrain, J. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2608.18957
Fact and Ideology in the Machine: Modelling Knowledge and Belief in Neural Models from Text
Saynova, D. (2025) · Chalmers Research (Chalmers University of Technology)
Risk Is Not Review Value: Wrong-Answer Exposure Under Bounded Review BudgetsPreprint
Park, S., Choi, M., 김진억, & Kang, M. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2609.07095 · Cited by 1
“Calibration methods seek confidence signals aligned with answer correctness [11, 13, 29].”
Pal, A. (2026) · Discover Artificial Intelligence · doi:10.1007/s44163-026-02240-w
Lyu, J., Xue, J., Baker, T., Meng, W., Tan, Y., Chen, J., & Liu, Z. (2026) · Expert Systems with Applications · doi:10.1016/j.eswa.2026.134466
Detecting folktales in historical newspapers using large language models
Rejec, J. (2026) · Repository of the University of Ljubljana (University of Ljubljana)
Offline Routing for Health Language Systems with Clarification and Escalation
Cao, Z., & Gao, J. (2025) · Journal of Technology Informatics and Engineering · doi:10.51903/jtie.v4i2.591
Wijaya, S. N., & Anderies (2026) · International Conference on Information Management and Technology (ICIMTech) · doi:10.1109/icimtech202671078.2026.11698984
“Zhao et al. [16] showed that language model confidence is skewed by label bias, while Jiang et al. [13] found that calibration degrades across diverse downstream tasks.”
Metamorphic Evidence as a Runtime Reliability Signal for Small Language Models
Jung, B., & Chang, J. (2026) · Sensors · doi:10.3390/s26216918
O., E. F. T. (2026) · Research Square · doi:10.21203/rs.3.rs-11210583/v1
Kawiak, A., Kwaśniewicz, Ł., Wójcik, G. M., & Danek, B. (2026) · Research Square · doi:10.21203/rs.3.rs-10983955/v1
Frahm, N., Patel, P., Zhang, Y., Yu, S., Bansal, M., & Sengupta, R. (2026) · doi:10.1109/cvprf71909.2026.00303
“Token-level probabilities offer a well-established and reliable confidence signal [6, 13, 17, 22] and avoid complex output parsing.”
Large Language Models in Cybersecurity: Explainability, Trust, and Responsible Deployment
Sharma, D., Girdhar, N., & Modi, A. (2026) · Advances in computational intelligence and robotics book series · doi:10.4018/979-8-2600-1970-2.ch009
Metadata
- Authors
- Jiang, Z., Araki, J., Ding, H., & Neubig, G.
- Affiliations
- Carnegie Mellon University (US)
- Robert Bosch (United States) (US)
- Corresponding author
- Jiang, Zhengbao
- Araki, Jun
- Ding, Haibo
- Neubig, Graham
- Venue
- Transactions of the Association for Computational Linguistics
- Publisher
- MIT Press
- Published
- 2021
- Volume
- 9
- Pages
- 962-977
- ISSN
- 2307-387X (ISSN-L)
- OpenAlex
- W3199958362
- Cited by
- 177
- References
- 71
- Citations by year
- 2026: 43 · 2025: 49 · 2024: 32 · 2023: 37 · 2022: 14 · 2021: 2
- FWCI
- 7.48
- Type
- journal article
- Language
- en
- Open access
- diamond
- doi.org
- Locations
- Transactions of the Association for Computational Linguistics (publishedVersion, cc-by)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- question answering, confidence calibration, probabilistic reasoning, T5, BART, GPT-2, fine-tuning, knowledge capture, factual knowledge, commonsense knowledge
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.