CoChat
Sign Up
PreprintCited by 52Adversarial Robustness in Machine LearningTopic ModelingExplainable Artificial Intelligence (XAI)

Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMs

Open access
Xiong, Miao, Hu, Zhiyuan, Lu, Xinyang, Li, Yifei, Fu, Jie, He, Junxian, Hooi, Bryan·2023·arXiv
DOI 10.48550/arxiv.2306.13063arXiv 2306.13063
Open this paper in CoChat — free accountOpen PDF

Abstract

Empowering large language models to accurately express confidence in their answers is essential for trustworthy decision-making. Previous confidence elicitation methods, which primarily rely on white-box access to internal model information or model fine-tuning, have become less suitable for LLMs, especially closed-source commercial APIs. This leads to a growing need to explore the untapped area of black-box approaches for LLM uncertainty estimation. To better break down the problem, we define a systematic framework with three components: prompting strategies for eliciting verbalized confidence, sampling methods for generating multiple responses, and aggregation techniques for computing consistency. We then benchmark these methods on two key tasks-confidence calibration and failure prediction-across five types of datasets (e.g., commonsense and arithmetic reasoning) and five widely-used LLMs including GPT-4 and LLaMA 2 Chat. Our analysis uncovers several key insights: 1) LLMs, when verbalizing their confidence, tend to be overconfident, potentially imitating human patterns of expressing confidence. 2) As model capability scales up, both calibration and failure prediction performance improve. 3) Employing our proposed strategies, such as human-inspired prompts, consistency among multiple responses, and better aggregation strategies can help mitigate this overconfidence from various perspectives. 4) Comparisons with white-box methods indicate that while white-box methods perform better, the gap is narrow, e.g., 0.522 to 0.605 in AUROC. Despite these advancements, none of these techniques consistently outperform others, and all investigated methods struggle in challenging tasks, such as those requiring professional knowledge, indicating significant scope for improvement. We believe this study can serve as a strong baseline and provide insights for eliciting confidence in black-box LLMs.

AI snapshot

Written from the paper’s full text.

TL;DR

This paper empirically evaluates black-box confidence elicitation for LLMs within a unified framework of prompting, sampling, and aggregation strategies. It finds that LLMs are often overconfident when verbalizing confidence, but human-inspired prompts, multiple-response consistency, and hybrid aggregation can partially improve calibration and failure prediction.

Key findings

  • LLMs tend to be overconfident when verbalizing confidence: confidence values mostly fall between 80% and 100%, often in multiples of 5, while accuracy in those bins is much lower. GPT-3, GPT-3.5, and Vicuna show average ECE exceeding 0.377; GPT-4 has lower ECE but an average AUROC of only 62.7%, close to random guessing.
  • Scaling improves confidence quality: moving from GPT-3 to GPT-4 yields about a 22.2% improvement in AUROC, though performance remains far from ideal.
  • Sampling multiple responses and using consistency improves failure prediction, especially on arithmetic tasks: on GSM8K, AUROC rises from 54.8% with CoT M=1 to 92.7% with Self-Random sampling M=5 plus consistency.
  • Aggregation matters: Pair-Rank achieves the lowest average ECE and can reduce ECE to as low as 0.028 on one dataset, while Avg-Conf improves AUROC in five of six datasets. White-box methods perform better than black-box methods, but the gap is narrow, e.g., 0.522 to 0.605 in AUROC.

Methods

  • The paper defines a black-box confidence elicitation framework with three components: prompting strategies (Vanilla, CoT, Self-Probing, Multi-Step, Top-K), sampling strategies (Self-Random, Prompting, Misleading), and aggregation strategies (Consistency, Avg-Conf, Pair-Rank).
  • It benchmarks these methods on five reasoning types across eight datasets—commonsense, arithmetic, symbolic, professional knowledge, and ethical knowledge—using five LLMs: Vicuna 13B, GPT-3 175B, GPT-3.5-turbo, GPT-4, and LLaMA 2 70B.
  • Evaluation uses calibration and failure prediction metrics: ECE, AUROC, AUPRC-Positive, and AUPRC-Negative. Sampling experiments use M=5 responses and temperature 0.7 for Self-Random.

Limitations

  • The paper states that its scope is mainly fixed-form and freeform QA tasks with a unique ground-truth answer, leaving summarization and open-ended QA to future work.
  • The paper states that black-box approaches remain suboptimal and that integrating black-box methods with limited white-box access, such as model logits, is a promising direction.
  • The text makes evident that no single technique consistently outperforms others, and all investigated methods struggle on challenging tasks requiring professional knowledge; failure prediction can remain close to random guessing.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

DataCite · 10.48550/arxiv.2306.13063

Cites

37 of 37 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

9 of 16 cited sources checked · 5 supported · 1 flagged · accuracy not yet measured

  • Premisecitation

    Verbalized confidence is intrinsically tied to the semantic meaning of the answer instead of its syntactic or lexical form.

    “the verbalized confidence is intrinsically tied to the semantic meaning of the answer instead of its syntactic or lexical form (Kuhn et al., 2023).”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites Kuhn et al., 2023 source

      The Introduction (page unavailable) distinguishes confidence about an answer’s meaning from token-level confidence about its wording, and the Discussion describes semantic entropy as measuring uncertainty over meanings rather than sequences. But the paper does not establish that verbalized confidence is intrinsically tied to meaning; its semantic-entropy method is designed to account for that distinction. (LLM, conf=0.84)

      Experimental — accuracy not yet measured
  • Premisecitation

    The variance observed among multiple responses for a given question offers another valuable perspective on model confidence.

    “the variance observed among multiple responses for a given question offers another valuable perspective on model confidence.”
    Quote in the paperSubstantial claim1 of 3 sources checked
    • cites Gawlikowski et al., 2021 source
    • cites Gal & Ghahramani, 2016

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites Lakshminarayanan et al., 2017

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Premisecitation

    Alternative approaches like semantic uncertainty (Kuhn et al., 2023) come with significant computational demands.

    “Although several alternative approaches like semantic uncertainty (Kuhn et al., 2023) have been proposed, they come with significant computational demands.”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites Kuhn et al., 2023 source

      The Acknowledgements describe experimentation with foundation models as computationally costly, and Step 2: Clustering by semantic equivalence says the method can require a quadratic number of comparisons. But that same Step 2 passage says the clustering cost is small in practice compared with generating sequences. The paper therefore supports a computational-cost concern, but not an unqualified claim that semantic uncertainty itself has significant computational demands. (LLM, conf=0.82)

      Experimental — accuracy not yet measured
  • Premisecitation

    Zero-shot Chain-of-Thought (CoT) has proven efficacy in inducing reasoning processes and improving model accuracy across diverse datasets.

    “we use zeroshot Chain-of-Thought, CoT (Kojima et al., 2022) for its proven efficacy in inducing reasoning processes and improving model accuracy across diverse datasets.”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites Kojima et al., 2022 source
  • Findingown data

    None of the investigated confidence elicitation techniques consistently outperform others.

    “Despite these advancements, none of these techniques consistently outperform others”
    • Also stated in: § 5.2 HUMAN-INSPIRED PROMPTING STRATEGIES PARTIALLY REDUCE OVERCONFIDENCE · plain
      “No single prompting strategy consistently outperforms the others.”
    Quote in the paperSubstantial claim
  • Classified as probably not a substantial claim. 18
  • May combine two claims. 28

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Pretraining improves model calibration.

    “pretraining improves model calibration.”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites Chen et al. (2022) source

      The Introduction identifies “pretraining improves model calibration” as an established conclusion that this paper challenges. In Section 4.2 (page unavailable), the authors report that pretraining consistently increases confidence in wrong predictions and conclude that it does not enable models to become calibrated; its effect on other calibration measures depends on the evaluation dataset. (LLM, conf=0.90)

      Checked by a reasoning model — may be wrong.

      Experimental — accuracy not yet measured

  • Classified as probably not a substantial claim.

    Premisecitation

    Chen et al. (2022) found that PLMs are not well calibrated.

    “Chen et al. (2022) finds that PLMs are not well calibrated”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites Chen et al. (2022) source

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Kadavath et al. (2022) found that larger models appear to be well-calibrated on multiple choice and true/false questions when provided in the right format.

    “Kadavath et al. (2022) studies the calibration of LLMs (parameter size ranging 800M to 50B), finding that larger models appear to be well-calibrated on multiple choice and true/false questions when provided in the right format.”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites Kadavath et al. (2022) source

  • Classified as probably not a substantial claim.

    Premisecitation

    Jiang et al. (2021) discovered that generative language models' probabilities on question-answering tasks are poorly calibrated.

    “Jiang et al. (2021) investigates the calibration of generative language models (T5, BART, and GPT-2) and discovers that these models' probabilities on question-answering tasks are poorly calibrated.”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites Jiang et al. (2021) source

  • May combine two claims.

    Interpretationcitation

    Issues making sampling consistency ineffective can be the model's poor calibration or computational constraints limiting the number of sampled queries.

    “Issues making this method ineffective can be: 1) the model's poor calibration (Kuhn et al., 2023), i.e., P model (x t |x 1:t-1 ) does not align well with P data (x t |x 1:t-1 ); or 2) the computational constraints limiting the number of sampled queries, leading to inaccurate estimates.”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites Kuhn et al., 2023 source

      In the passage headed “Jiang et al. (2021) do examine calibration in generative question answering” (page unavailable), the paper reports a weak relationship between answer likelihood and correctness and says semantic equivalence makes calibration difficult to evaluate. Its “Step 1: Generating a set of answers from the model” section says sampling choices affect uncertainty-estimation performance. Neither passage establishes the cited claim that a mismatch between model and data token probabilities makes sampling consistency ineffective. The computational-constraints point is not attached to this citation. (LLM, conf=0.74)

      Experimental — accuracy not yet measured

  • May combine two claims.

    Findingown data

    Vanilla verbalized confidence exhibits significant overconfidence and poor failure prediction, casting doubts on its reliability.

    “Vanilla verbalized confidence exhibits significant overconfidence and poor failure prediction, casting doubts on its reliability.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    The sampling strategy with 5 sampled responses paired with consistency aggregation consistently outperforms verbalized confidence in calibration and failure prediction, particularly on arithmetic tasks, e.g., GSM8K AUROC improves from 54.8% to 92.7%.

    “Table 3 demonstrates that the sampling strategy with 5 sampled responses paired with consistency aggregation consistently outperform verbalized confidence in calibration and failure prediction, particularly on arithmetic tasks, e.g., GSM8K showcases a remarkable improvement in AUROC from 54.8% (akin to random guessing) to 92.7%, effectively distinguishing between incorrect and correct answers.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingreasoning

    Pair-Rank achieves better performance in calibration while Avg-Conf boosts more in failure prediction.

    “Pair-Rank achieves better performance in calibration while Avg-Conf boosts more in failure prediction.”
    Quote in the paperSubstantial claim
    • The authors state this as an assumption: Accurate samples often produce consistent outcomes, while incorrect ones yield various responses, leading to low consistency.
    • The authors state this as an assumption: The model's ranking between two options may be more accurate than the verbalized confidence it provides.

  • May combine two claims.

    Findingown data

    White-box methods perform better than black-box methods, but the gap is narrow (e.g., 0.522 to 0.605 in AUROC).

    “Comparisons with white-box methods indicate that while white-box methods perform better, the gap is narrow, e.g., 0.522 to 0.605 in AUROC.”
    • Also stated in: § Comparative analysis of white-box and black-box methods. · plain
      “We conducted comparisons on five datasets with three white-box methods (see §B.1) and observed that although white-box methods indeed perform better, the gap is narrow, e.g., 0.522 to 0.605 in AUROC.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    GPT-4 displays lower ECE, but its AUROC and AUPRC-Negative scores remain suboptimal, with an average AUROC of merely 62.7%.

    “While GPT-4 displays lower ECE, its AUROC and AUPRC-Negative scores remain suboptimal, with an average AUROC of merely 62.7%-close to the 50% random guess threshold-highlighting challenges in distinguishing correct from incorrect predictions.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Multi-step and Top-K prompting strategies effectively reduce ECE and enhance AUROC.

    “both Multi-step and Top-K prompting strategies effectively reduce ECE and enhance AUROC.”
    • Also stated in: § B.4 DETAILED PERFORMANCE OF DIFFERENT PROMPTING STRATEGIES · boosted
      “Multi-step and Top-K prompting strategies demonstrate promising results in reducing ECE and improving AUROC”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    The ECE and AUROC could be improved by sampling more responses, but the improvement becomes marginal as the number gets larger.

    “The result suggests that the ECE and AUROC could be improved by sampling more responses, but the improvement becomes marginal as the number gets larger.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    While ECE can be effectively reduced using suitable prompting strategies, failure prediction still remains a challenge.

    “While ECE can be effectively reduced using suitable prompting strategies, failure prediction still remains a challenge.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Self-Probing maintains the most consistent advantage over the baseline on GPT-4, while Top-K emerges as the top performer on GPT-3.5.

    “we find that Self-Probing maintains the most consistent advantage over the baseline on GPT-4, while Top-K emerges as the top performer on GPT-3.5.”
    Quote in the paperSubstantial claim
    • The authors state this as an assumption: Humans often find it easier to identify errors in others' answers than in their own, as they can become fixated on a particular line of thinking, potentially overlooking mistakes.

  • May combine two claims.

    Findingown data

    Models tend to have high confidence for all samples, appearing as multiples of 5 and with most values ranging between 80% and 100%.

    “Notably, the models tend to have high confidence for all samples, appearing as multiples of 5 and with most values ranging between the 80% to 100% range”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Human-inspired prompting strategies improve model accuracy and calibration over the vanilla baseline, with modest advancements in failure prediction.

    “Human-inspired prompting strategies improve model accuracy and calibration, albeit with diminishing returns in advanced models like GPT-4.”
    • Also stated in: § 5.2 HUMAN-INSPIRED PROMPTING STRATEGIES PARTIALLY REDUCE OVERCONFIDENCE · plain
      “human-inspired strategies offer consistent improvements in accuracy and calibration over the vanilla baseline, with modest advancements in failure prediction.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Consistency among multiple responses is more effective in improving failure prediction and calibration compared to verbalized confidence (M=1), with particularly notable improvements on arithmetic tasks.

    “Consistency among multiple responses is more effective in improving failure prediction and calibration compared to verbalized confidence (M = 1), with particularly notable improvements on the arithmetic task.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Proposed prompt strategies such as CoT significantly increased accuracy, while the confidence output distribution still remains at the range of 80%-100%.

    “the proposed prompt strategies such as CoT have significantly increased the accuracy (see Table 8), while the confidence output distribution still remains at the range of 80% -100%”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    The best-performing algorithms can reduce ECE to a quite low value like 0.028, but they still face challenges in predicting incorrect predictions, especially in tasks requiring professional knowledge, such as professional law.

    “while the best-performing algorithms can reduce ECE to a quite low value like 0.028, they still face challenges in predicting incorrect predictions, especially in those tasks requiring professional knowledge, such as professional law.”
    Quote in the paperSubstantial claim
    • The authors state this as an assumption: The model's ranking between two options may be more accurate than the verbalized confidence it provides.

  • May combine two claims.

    Findingown data

    As model capability scales up, both calibration and failure prediction performance improve.

    “As model capability scales up, both calibration and failure prediction performance improve, yet still far from ideal performance.”
    • Also stated in: § 5.1 LLMS TEND TO BE OVERCONFIDENT WHEN VERBALIZING THEIR CONFIDENCE · plain
      “Calibration and failure prediction performance improve as model capacity scales.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    In terms of AUROC, a higher candidate set size K contributes to superior performance and reduced variance.

    “The results indicate that, in terms of AUROC, a higher candidate set size K contributes to superior performance and reduced variance.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Accuracy within each confidence bin is much lower than its corresponding confidence, indicating significant overconfidence.

    “the accuracy within each bin is much lower than its corresponding confidence, indicating significant overconfidence.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingreasoning

    Sampling consistency among multiple responses is more effective compared to verbalized confidence (M = 1), with particularly notable improvements on the arithmetic task.

    “Sampling Consistency among multiple responses is more effective compared to verbalized confidence (M = 1), with particularly notable improvements on the arithmetic task.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Top-K outperforms all other methods on GPT-3.5 and is comparable to the top-performing method Self-Probing on GPT4.

    “Top-K outperforms all other methods on GPT-3.5 and is comparable to the top-performing method Self-Probing on GPT4.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Pair-Rank aggregation achieves the lowest ECE in half of the datasets and maintains the lowest average ECE in calibration.

    “Pair-Rank aggregation achieves the lowest ECE in half of the datasets and maintains the lowest average ECE in calibration”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationown data

    Verbalized confidence, albeit imprecise, reflects the model's uncertainty tendency and can enhance results when combined with ensemble methods.

    “Verbalized confidence, albeit imprecise, reflects the model's uncertainty tendency and can enhance results when combined with ensemble methods.”
    Quote in the paperSubstantial claim
    • The authors state this as an assumption: Accurate samples often produce consistent outcomes, while incorrect ones yield various responses, leading to low consistency.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    This behavior resembles patterns identified in the training corpus for GPT-like models as discussed by Zhou et al. (2023).

    “This behavior resembles the patterns identified in the training corpus for GPT-like models as discussed by Zhou et al. (2023).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Zhou et al. (2023)

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • May combine two claims.

    Interpretationreasoning

    Pair-Rank learns the categorical distribution of potential answers through K observations, which aligns well with calibration and is therefore more likely to lead to a lower ECE.

    “Pair-Rank learns the categorical distribution of potential answers through our K observations, which aligns well with the notion of calibration and is therefore more likely to lead to a lower ECE.”
    Quote in the paperSubstantial claim
    • The authors state this as an assumption: The model's ranking between two options may be more accurate than the verbalized confidence it provides.

  • Probably not a substantial claim: may not pass S4 (based).

    Premisereasoning

    Another way to alleviate overconfidence is to realize the existence of multiple possible solutions or answers, which acts as a normalization for the confidence distribution.

    “Another way to alleviate overconfidence is to realize the existence of multiple possible solutions or answers, which acts as a normalization for the confidence distribution.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Interpretationreasoning

    Even weak misleading information causes the model to analyze and reassess its answers.

    “even providing weak misleading information, the model will analyze and reassess their answers.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisereasoning

    The optimization objective to minimize the expected loss is given by equation 4.

    “The optimization objective to minimize the expected loss is then: min P - N i=1 Su∈A Sv∈A I S u (i) ≻ S v • log P (S u ) P (S u ) + P (S v ) s.t. Su∈A P (S u ) = 1(4)”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisereasoning

    The challenges in adapting ensemble-based methods lie in two critical components: the sampling strategy and the aggregation strategy.

    “The challenges in adapting ensemble-based methods lie in two critical components: 1) the sampling strategy, i.e., how to sample multiple responses from the model's answer distribution, and 2) the aggregation strategy, i.e., how to aggregate these responses to yield the final answer and its associated confidence.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationreasoning

    Confidently providing randomly generated misleading answers can sometimes lead to negative effects.

    “since the misleading answers are generated randomly, confidently providing this information can sometimes lead to negative effects.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationown data

    Even as model capability scales up, calibration and failure prediction performance remain far from ideal.

    “As model capability scales up, both calibration and failure prediction performance improve, yet still far from ideal performance.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationreasoning

    Models might be imitating human expressions when verbalizing confidence.

    “Such behavior suggests that models might be imitating human expressions when verbalizing confidence.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Interpretationcitation

    The suboptimal performance of verbalized confidence might be explained by the inherent inaccuracy of the training data, particularly human expressions of uncertainty.

    “this might be explained by the inherent inaccuracy of the training data, particularly human expressions of uncertainty.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Interpretationown data

    The current suboptimal performance of verbalized confidence points to an important research gap.

    “The current suboptimal performance of verbalized confidence points to an important research gap”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Vicuna is a smaller model fine-tuned from LLaMA.

    “Vicuna is a smaller model fine-tuned from LLaMA (Touvron et al., 2023a).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Touvron et al., 2023a

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • May combine two claims.

    Interpretationreasoning

    The Business Ethics dataset's better performance with a moderate K can be attributed to the limited variability of misleading information (restricted to 4 types) used in the experiments, implying that introducing a large number of additional queries does not significantly enhance the information pool.

    “This observation can be attributed to the limited variability of misleading information (restricted to 4 types) used in our experiments for the Business Ethics dataset, implying that the introduction of a large number of more queries does not significantly enhance the information pool.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Humans sometimes tend to exaggerate their a priori probability for an event that has occurred.

    “humans sometimes tend to exaggerate their a priori probability for an event that has occurred.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Garthwaite et al. (2005a)

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • May combine two claims.

    Premisecitation

    Descriptions of uncertainty in the training corpus are mostly human expressions, which are often considered inaccurate.

    “descriptions of uncertainty in the training corpus are mostly human expressions, which are often considered inaccurate (Garthwaite et al., 2005b).”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Garthwaite et al., 2005b

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • May combine two claims.

    Interpretationreasoning

    Verbalized confidence, despite being less precise, still captures the model's uncertainty tendency and allows for finer-grained output values, and hence can be combined to enhance calibration performance.

    “The verbalized confidence, despite being less precise, still captures the model's uncertainty tendency and allows for finer-grained output values, and hence can be combined to enhance calibration performance.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Premisenone given

    GSM8K is the more representative dataset because it has wider usage.

    “We select GSM9K as the more representative dataset because it has a wider usage.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Premisenone given

    Date Understanding is the more representative dataset because it is more difficult than Object Counting.

    “We select Date Understanding as the more representative dataset since it is more difficult than Object Counting.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premiseown data

    StrategyQA is the more representative dataset because it contains more data.

    “We select StrategyQA as the more representative dataset since it contains more data.”
    Quote in the paperNot a substantial claim
45 of 45

  • The authors state this as an assumption.

    Accurate samples often produce consistent outcomes, while incorrect ones yield various responses, leading to low consistency.

    “This approach is grounded in the observation that accurate samples often produce consistent outcomes, while incorrect ones yield various responses, leading to a low consistency.”

  • The paper relies on this without support.

    When confident, individuals tend to stick to their initial answers despite contrary suggestions; conversely, when uncertain, they are more likely to waver or adjust their responses based on misleading hints.

    “This method draws inspiration from human behaviors: when confident, individuals tend to stick to their initial answers despite contrary suggestions; conversely, when uncertain, they are more likely to waver or adjust their responses based on misleading hints.”

  • The authors state this as an assumption.

    The model's ranking between two options may be more accurate than the verbalized confidence it provides.

    “The underlying assumption is that the model's ranking between two options may be more accurate than the verbalized confidence it provides, especially given our observation that the latter tends to exhibit overconfidence.”

  • The authors state this as an assumption.

    Humans often find it easier to identify errors in others' answers than in their own, as they can become fixated on a particular line of thinking, potentially overlooking mistakes.

    “A common observation of humans is that they often find it easier to identify errors in others' answers than in their own, as they can become fixated on a particular line of thinking, potentially overlooking mistakes.”

  • The paper relies on this without support.

    Previous confidence elicitation methods relying on white-box access or fine-tuning have become less suitable for LLMs, especially closed-source commercial APIs.

    “Previous confidence elicitation methods, which primarily rely on white-box access to internal model information or model fine-tuning, have become less suitable for LLMs, especially closed-source commercial APIs.”

  • The authors state this as an assumption.

    The Top-K answers are drawn from a categorical distribution P without replacement.

    “Suppose the Top-K answers are drawn from a categorical distribution P without replacement.”

  • The paper relies on this without support.

    LLMs are not explicitly modeled to express uncertainty in textual outputs.

    “LLMs are not explicitly modeled to express uncertainty in textual outputs”

  • The paper relies on this without support.

    The model's internal distribution P_model(x_t|x_1:t-1) is trained to mirror the ground truth data distribution.

    “which is trained to mirror the ground truth data distribution.”

  • The paper relies on this without support.

    Humans are able to verbalize their uncertainty.

    “humans are able to verbalize their uncertainty”

  • The paper relies on this without support.

    It is reasonable to expect LLMs to have also learned the ability to verbalize uncertainty, or to learn it at some point in the future.

    “So it is reasonable to expect LLMs to have also learned this ability, or to learn it at some point in the future.”

  • The authors state this as an assumption.

    Top-K answers are drawn from a categorical distribution P without replacement.

    “Suppose the Top-K answers are drawn from a categorical distribution P without replacement.”
11 of 11

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Uncertainty estimation has been a valuable tool in assessing the reliability of machine learning models, but the quality of uncertainty estimation within LLMs and VLMs remains relatively underexplored.

    “Uncertainty estimation has been a valuable tool in assessing the reliability of machine learning models, but the quality of uncertainty estimation within LLMs and VLMs remains relatively underexplored (Xiong et al., 2023; Huang et al., 2023b; Kostumov et al., 2024).”

    From Overconfidence is Key: Verbalized Uncertainty Evaluation in Large Language and Vision-Language Models (2024) · cites this paper as Xiong et al., 2023

    The Abstract (page unavailable) describes reliable confidence estimation as important and black-box approaches to LLM uncertainty estimation as an untapped area. The comparative analysis (page unavailable) calls LLM uncertainty estimation challenging and unresolved. The paper does not address VLMs, and its underexploration claim is specifically about black-box LLM approaches rather than all LLM uncertainty estimation. (LLM, conf=0.86)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • Top-K ECE values disagree across tables

    Major

    Table 13 gives Top-K ECEs of 39.8 for GSM8K and 40.1 for DateUND, whereas Table 3 gives 19.6 and 26.1 for the same datasets and M=1 method. The conflicting inputs also produce different reported averages.

  • Avg-Conf mean ECE in Table 4 is miscalculated

    Minor

    The six listed ECE values average 11.47, not the reported 14.8. The table states that metrics are given by ×102, so both figures are on the same scale.

    Reported 14.8 · recomputed 11.466667

  • Five-response sampling does not consistently improve calibration

    Minor

    The claim of consistent improvement conflicts with Table 3: on GSM8K, prompting with M=5 has ECE 35.2, worse than CoT with M=1 at 10.1.

3 of 3

Cites

Flagged first, then by citations

Research works printed but not linked: 8

  1. Yarin Gal and Zoubin Ghahramani. Dropout as a bayesian approximation: Representing model uncertainty in deep learning. In international conference on machine learning, pp. 1050-1059.
  2. PMLR, 2016.
  3. Paul H Garthwaite, Joseph B Kadane, and Anthony O'Hagan. Statistical methods for eliciting probability distributions. Journal of the American statistical Association, 100(470):680-701, 2005b.
  4. Volodymyr Kuleshov and Shachi Deshpande. Calibrated and sharp uncertainties in deep learning via density estimation. In International Conference on Machine Learning, pp. 11683-11693.
  5. PMLR, 2022.
  6. Volodymyr Kuleshov, Nathan Fenner, and Stefano Ermon. Accurate uncertainties for deep learning using calibrated regression. In International conference on machine learning, pp. 2796-2804.
  7. PMLR, 2018.
  8. Bianca Zadrozny and Charles Elkan. Obtaining calibrated probability estimates from decision trees and naive bayesian classifiers. In Icml, volume 1, pp. 609-616, 2001.

Cited by

Sorted by citations

Authors
Xiong, M., Hu, Z., Lu, X., Li, Y., Fu, J., He, J., & Hooi, B.
Affiliations
  • National University of Singapore (SG)
  • École Polytechnique Fédérale de Lausanne (CH)
  • Hong Kong University of Science and Technology (HK)
Venue
arXiv
Publisher
arXiv
Published
2023
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2306.13063
arXiv
2306.13063
OpenAlex
W7055216282
Cited by
52
References
37
Citations by year
2026: 9 · 2025: 25 · 2024: 17 · 2023: 1
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • Rare & Special e-Zone (The Hong Kong University of Science and Technology) (submittedVersion)
  • arXiv (Cornell University)
Funders
Ministry of Education - Singapore (Tier 1, FY2023)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
Keywords
large language models, uncertainty estimation, confidence calibration, failure prediction, overconfidence, prompting strategies, commonsense reasoning, arithmetic reasoning, GPT-4

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-10.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-10.

Cited sources are checked by gpt-6-sol.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review