CoChat
Sign Up
Cited by 22Adversarial Robustness in Machine LearningTopic ModelingDomain Adaptation and Few-Shot Learning

A Close Look into the Calibration of Pre-trained Language Models

Open access
Chen, Yangyi, Yuan, Lifan, Cui, Ganqu, Liu, Zhiyuan, Ji, Heng·2023·Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)
DOI 10.18653/v1/2023.acl-long.75
Open this paper in CoChat — free accountOpen PDF

Abstract

Pre-trained language models (PLMs) may fail in giving reliable estimates of their predictive uncertainty.We take a close look into this problem, aiming to answer two questions: (1) Do PLMs learn to become calibrated in the training process?(2) How effective are existing calibration methods?For the first question, we conduct fine-grained control experiments to study the dynamic change in PLMs' calibration performance in training.We consider six factors as control variables, including dataset difficulty, available training samples, training steps, the number of tunable parameters, model scale, and pretraining.We observe a consistent change in calibration performance across six factors.We find that PLMs don't learn to become calibrated in training, evidenced by the continual increase in confidence, no matter whether the predictions are correct or not.We highlight that our finding somewhat contradicts two established conclusions: (a) Larger PLMs are more calibrated; (b) Pretraining improves model calibration.Next, we study the effectiveness of existing calibration methods in mitigating the overconfidence issue.Besides unlearnable calibration methods (e.g., label smoothing), we adapt and extend two recently proposed learnable methods that directly collect data to train models to have reasonable confidence estimations.Experimental results show that learnable methods significantly reduce PLMs' confidence in wrong predictions.The code is available at https://github. com/lifan-yuan/PLMCalibration.

AI snapshot

Written from the paper’s full text.

TL;DR

This paper empirically examines whether pre-trained language models (PLMs) become calibrated during training and how well existing calibration methods mitigate overconfidence. It finds that PLMs do not learn to become calibrated—confidence keeps increasing even for wrong predictions—and that learnable calibration methods substantially reduce confidence on wrong predictions compared with unlearnable methods.

Key findings

  • PLMs do not learn to become calibrated in training: confidence continually increases whether predictions are correct or not, producing increasing ECE in the over-fitted state and a consistent increase in CErr_neg throughout training.
  • The paper reports findings that contradict two established conclusions: larger PLMs are more calibrated, and pretraining improves calibration. It attributes the inconsistency to evaluation dataset difficulty (performance not saturating) and to prior metrics not measuring confidence in wrong predictions.
  • Learnable calibration methods significantly reduce confidence in wrong predictions. For T5, E-T5 lowers CErr_neg from 84.12 to 45.84 on MNLI, from 82.05 to 3.40 on Amazon, and from 82.12 to 0.99 on Civil, compared with Vanilla.
  • More data and larger model scales for the calibration task both improve calibration performance; an extrinsic predictive model can achieve comparable results to intrinsic methods given the same calibration training data, suggesting the success lies in the learnable attribute of the calibration task rather than PLMs’ self-checking.

Methods

  • Fine-grained control experiments with RoBERTa-base and T5-base under prompt-based learning, using SST-2, MNLI, AG-News, and Yahoo; six factors are manipulated: dataset difficulty, available training samples, training steps, number of tunable parameters, model scale, and pretraining.
  • Calibration is evaluated with accuracy, average confidence, ECE using equal-mass binning and 100 bins, and application-driven errors CErr_pos = 1 − Conf_pos and CErr_neg = Conf_neg.
  • Calibration methods are grouped into unlearnable (Vanilla, temperature scaling, label smoothing, EDA, deep ensemble) and learnable (E-MLP, E-T5, I-Vanilla, I-Iter, I-Simul), tested in in-distribution and out-of-distribution settings using MNLI/HANS/ANLI, Amazon/SST-5/SemEval, and Civil/Hate Speech/Implicit Hate.

Limitations

  • The paper states that only empirical results are presented and a theoretical understanding of PLMs calibration is still lacking.
  • The paper states an assumption of a large held-out validation set for training the calibration task; in practice, data must be allocated between the main task and the calibration task given limited training samples.
  • The text makes evident that the effect of scaling the calibration-task model on OOD calibration is not consistent and is left for future exploration.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.18653/v1/2023.acl-long.75

Cites

67 of 67 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

0 of 20 cited sources checked · 0 supported · 0 flagged

  • Findingown data

    In the under-fitted state, PLMs' performance and confidence increase at different speeds as they become more fitted on the training distribution.

    “In the under-fitted state, PLMs' performance and confidence increase at different speeds when more fitted on the training distribution.”
    • Also stated in: § 4.2 Experimental Results · plain
      “In this state, PLMs' performance and confidence increase at different speeds when more fitted on the training distribution.”
    Quote in the paperSubstantial claim
  • Findingown data

    PLMs do not learn to become calibrated during training; their confidence continually increases regardless of whether predictions are correct.

    “We find that PLMs don't learn to become calibrated in training, evidenced by the continual increase in confidence, no matter whether the predictions are correct or not.”
    • Also stated in: § 4.2 Experimental Results · boosted
      “we conclude that PLMs don't learn to become calibrated in training, evidenced by the continually increasing confidence in predictions, no matter correct or not, in the fitting process.”
    Quote in the paperSubstantial claim
  • Findingcitation

    PLMs can be trained to give their uncertainty.

    “PLMs can be trained to give their uncertainty.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Lin et al., 2022

      Not checked in the free run.

  • Findingown data

    PLMs' calibration performance under distribution shifts depends on the evaluation datasets chosen.

    “PLMs' calibration performance under distribution shifts depends on the evaluation datasets chosen.”
    Quote in the paperSubstantial claim
  • Findingreasoning

    Using an extrinsic predictive model can achieve comparable results, given the same calibration training data.

    “We find that using an extrinsic predictive model can achieve comparable results, given the same calibration training data.”
    Quote in the paperSubstantial claim
  • May combine two claims. 29
  • Classified as probably not a substantial claim. 14
  • Borderline: one of the tests for a substantial claim is unclear. 1

  • May combine two claims.

    Findingnone given

    More data and larger model scales for the calibration task both play significant positive roles in PLMs' calibration performance.

    “More data and larger model scales for the calibration task both play significant positive roles in PLMs' calibration performance”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    All six factors (dataset difficulty, available training samples, training steps, number of tunable parameters, model scale, pretraining) influence PLMs' fitness on the training distribution.

    “All six factors influence PLMs' fitness on the training distribution.”
    • Also stated in: § 4.2 Experimental Results · plain
      “We note that all six factors dynamically influence PLMs' fitness on the training distribution”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingreasoning

    Unlearnable methods take effect by roughly imposing confidence regularization on models' predictions, resulting in almost the same amount of increase in CErr pos as the decrease in CErr neg.

    “which take effect by roughly imposing confidence regularization on models' predictions (e.g., label smoothing), resulting in almost the same amount of increase in CErr pos with the decrease in CErr neg .”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    The conclusion that PLMs exhibit degraded calibration performance under distribution shifts is reversed when the ID datasets are harder and PLMs achieve better performance on OOD datasets.

    “We find that this conclusion is reversed when the ID datasets are harder and PLMs achieve better performance on OOD datasets.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Learnable calibration methods are superior to unlearnable methods in both ID and OOD settings.

    “In our experiments, we identify the superiority of learnable methods compared to unlearnable ones, considering both in-distribution (ID) and out-of-distribution (OOD) settings.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Learnable methods maintain a reasonable increase in CErr pos, with consistent correlations between confidence drop and performance under distribution shifts.

    “Moreover, learnable methods can maintain a reasonable increase in CErr pos , holding consistent correlations between the drop in confidence and performance under distribution shifts.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingcitation

    Temperature scaling (TS) achieves overall better ECE, maintaining a strong baseline, with label smoothing (LS) being the second effective method in the unlearnable category.

    “TS achieves overall better ECE, maintaining a strong baseline method, with LS being the second effective method for the unlearnable category.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingreasoning

    For ID evaluation, we observe lower ECE, consistent with Desai and Durrett (2020).

    “For ID evaluation, we observe lower ECE, consistent with Desai and Durrett (2020).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Desai and Durrett, 2020

      Not checked in the free run.

  • May combine two claims.

    Findingnone given

    Our methods (I-Iter and I-Simul) address the degraded performance issue of I-Vanilla and make the main task performance match with vanilla fine-tuning.

    “our methods (I-Iter and I-Simul) address the degraded performance issue of I-Vanilla and make the main task performance match with the vanilla fine-tuning”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    T5 has an obvious confidence drop in the early stage of training.

    “T5 has an obvious confidence drop in the early stage”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingnone given

    The confidence of RoBERTa seems to continually increase along with the number of available training samples.

    “the confidence of RoBERTa seems to con-tinually increase along with the number of available training samples.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    For Adapter tuning, overall confidence continues to increase given more capacity, resulting in increasing ECE and CErr_neg, while performance stays constant.

    “However, the overall confidence continues to increase given more capacity, resulting in increasing ECE and CErr neg , while the performance stays constant.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    For intrinsic methods, the three training paradigms do not show substantial differences in calibration performance, and none consistently achieves the best performance on all datasets.

    “For intrinsic methods, the three different training paradigms don't show substantial differences considering the calibration performance, and none of them consistently achieves the best performance on all datasets.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    The vanilla T5 achieves only 30.53% accuracy on ANLI, but its average confidence is up to 93.77%.

    “For example, the vanilla T5 achieves only 30.53% accuracy on ANLI, but its average confidence is up to 93.77%.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    For Soft-prompt tuning, the increasing rates of performance and confidence are nearly matched, thus decreasing ECE continually.

    “We can observe that the increasing rates are nearly matched, thus decreasing ECE continually.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingreasoning

    Scaling larger and employing pretraining consistently bring CErr neg higher.

    “Scaling larger and employing pretraining consistently bring CErr neg higher.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    A larger increasing rate in performance reduces ECE; a smaller increasing rate increases ECE.

    “We observe that a larger (smaller) increasing rate in performance reduces (increases) ECE.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Learnable methods lower confidence in correct predictions, resulting in increasing CErr pos and ECE.

    “But we also observe that learnable methods lower the confidence in correct predictions, resulting in increasing CErr pos and ECE.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Learnable methods give obviously lower confidence in OOD samples, and the average confidence drop is highly relevant to the performance drop under distribution shifts.

    “Learnable methods give obviously lower confidence in OOD samples, and the average confidence drop is highly relevant to the performance drop under distribution shifts.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Pretraining only reduces ECE on relatively simpler datasets, like SST-2 and AG-News, but brings negligible benefits on MNLI and Yahoo.

    “We find that pretraining only reduces ECE on relative simpler datasets, like SST-2 and AG-News, but bring negligible benefits on MNLI and Yahoo.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Across all methods, ECE and CErr_pos decrease under hard-to-easy distribution shifts, contradictory to the previous conclusion that PLMs are less calibrated on OOD samples.

    “Across all methods, the ECE and CErr pos decrease under the hard-to-easy distribution shifts, contradictory to the previous conclusion that PLMs are less calibrated on OOD samples.”
    Quote in the paperSubstantial claim

  • Borderline on S3 (load-bearing): unclear from the text.

    Findingnone given

    The trends in average confidence are different in the two model architectures.

    “In this dimension, we additionally find that the trends in average confidence are different in the two model architectures.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationreasoning

    The main issue of PLMs calibration is overconfidence in wrong predictions, which cannot be trivially solved by increasing model scale.

    “Thus, we find that the main issue of PLMs calibration lies in their overconfidence in wrong predictions, which cannot be trivially solved by increasing the model scale.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationown data

    The finding that PLMs don't learn to become calibrated contradicts the established conclusions that larger PLMs are more calibrated and pretraining improves model calibration.

    “We highlight that our finding somewhat contradicts two established conclusions: (a) Larger PLMs are more calibrated; (b) Pretraining improves model calibration.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    Calibration is essential for out-of-distribution detection, selective prediction, robustness, and pseudolabeling.

    “Calibration is essential for out-ofdistribution detection (Hendrycks et al., 2019a), selective prediction (Varshney et al., 2022), robustness (Kumar et al., 2022), and pseudolabeling (Rizve et al., 2021).”
    Quote in the paperNot a substantial claim0 of 3 sources checked
    • cites Hendrycks et al., 2019a

      Not checked in the free run.

    • cites Varshney et al., 2022

      Not checked in the free run.

    • cites Kumar et al., 2022

      Not checked in the free run.

  • May combine two claims.

    Interpretationreasoning

    Data augmentation and model ensemble fail to relieve overconfidence, resulting in the same CErr neg as vanilla fine-tuning.

    “The reason lies in their inability to relieve the overconfident issue, resulting in the same Cerr neg with the vanilla fine-tuning”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    PLMs can be directly trained to output their uncertainty by words.

    “PLMs can also be directly trained to output their uncertainty by words (Lin et al., 2022).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Lin et al., 2022

      Not checked in the free run.

  • Classified as probably not a substantial claim.

    Interpretationreasoning

    Whether scaling the calibration-task model larger improves calibration performance under distribution shifts is determined by many factors (e.g., the dataset difficulty, the overconfidence issue in the calibration task).

    “Whether scaling the calibration-task model larger improves calibration performance under distribution shifts is determined by many factors (e.g., the dataset difficulty, the overconfidence issue in the calibration task).”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Premisecitation

    Previous work shows that PLMs exhibit degraded calibration performance under distribution shifts.

    “Previous work shows that PLMs exhibit degraded calibration performance under distribution shifts (Desai and Durrett, 2020).”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Desai and Durrett, 2020

      Not checked in the free run.

  • May combine two claims.

    Interpretationown data

    The conclusion that PLMs calibration degrades under distribution shifts is one-sided and heavily depends on the evaluation datasets chosen.

    “We further show the conclusion that PLMs calibration degrades under distribution shifts is one-sided and heavily depends on the evaluation datasets chosen in Appendix E.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationreasoning

    The inconsistency between our findings and prior conclusions lies in evaluation metrics: previous work doesn't measure confidence in wrong predictions, overlooking that models become more confident in wrong predictions when scaling larger and employing pretraining.

    “Evaluation metrics: previous work doesn't measure the confidence in wrong predictions, overlooking the fact that models are becoming more confident in wrong predictions when scaling larger and employing pretraining.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S1 (asserted).

    Premisecitation

    Larger PLMs show better calibration.

    “Larger PLMs show better calibration (Srivastava et al., 2022)”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Srivastava et al., 2022

      Not checked in the free run.

  • Classified as probably not a substantial claim.

    Premisecitation

    Pretraining improves model calibration.

    “Pretraining improves model calibration (Hendrycks et al., 2019b).”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Hendrycks et al., 2019b

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S4 (based). May combine two claims.

    Interpretationnone given

    Scaling larger and employing pretraining are both strategies to increase PLMs capacity, making them more fitted on the training distribution.

    “Actually, scaling larger and employing pretraining are both strategies to increase PLMs capacity, making them more fitted on the training distribution.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    Data augmentation and model ensemble have been empirically proven to improve model calibration.

    “Data augmentation (Hendrycks et al., 2020;Wang et al., 2021) and model ensemble (Gal and Ghahramani, 2016;Lakshminarayanan et al., 2017) have also been empirically proven to improve model calibration.”
    Quote in the paperNot a substantial claim0 of 3 sources checked
    • cites Hendrycks et al., 2020

      Not checked in the free run.

    • cites Wang et al., 2021

      Not checked in the free run.

    • cites Gal and Ghahramani, 2016

      Not checked in the free run.

  • May combine two claims.

    Interpretationreasoning

    TS and LS directly impose confidence regularization on predictions, which does not make PLMs have clear confidence estimations.

    “The reason is that these two methods directly impose confidence regularization on predictions, which don't actually make PLMs have clear confidence estimations.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S1 (asserted), S4 (based).

    Interpretationnone given

    PLMs' calibration performance degrades under distribution shifts.

    “This evaluation setting is expected to test the conclusion that PLMs' calibration performance degrades under distribution shifts.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Premisenone given

    Well-calibrated PLMs should give very low confidence and high probability entropy on task-irrelevant inputs.

    “Well-calibrated PLMs should give very low confidence and high probability entropy in the task-irrelevant inputs.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationown data

    PLMs' relative calibration performance on ID and OOD samples relies on the dataset difficulty, and the conclusion that PLMs are less calibrated under distribution shifts is one-sided.

    “The indication is that PLMs' relative calibration performance on ID and OOD samples relies on the dataset difficulty, and the conclusion that PLMs are less calibrated under distribution shifts is onesided.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Interpretationnone given

    The influence of the ID training dataset on PLMs calibration can be attributed to dataset difficulty.

    “Still, this can be attributed to the dataset difficulty.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductreasoning

    Each K was experimented on five times per dataset and the average performance was reported to account for potential variance in the few-shot setting.

    “We experiment with each K five times on each dataset and report the average performance due to the potential variance in the fewshot setting.”
    Quote in the paperNot a substantial claim
41 of 41

  • The paper relies on this without support.

    A deep understanding of PLMs calibration is lacking.

    “and a deep understanding of PLMs calibration is lacking.”

  • The paper relies on this without support.

    Prompt-based learning has superior performance compared to traditional fine-tuning, especially in the few-shot setting.

    “We employ the prompt-based learning paradigm (Liu et al., 2021) since its superior performance compared to traditional fine-tuning, especially in the few-shot setting.”

  • The authors state this as an assumption.

    We have a large held-out validation set that can be employed as the training dataset for the calibration task.

    “We make an assumption that we have a large held-out validation set that can be employed as the training dataset for the calibration task.”
3 of 3

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • ANLI accuracy and confidence do not match the reported T5 results

    Major

    The narrative reports 30.53% accuracy and 93.77% confidence for vanilla T5 on ANLI. Table 1 reports 31.31% accuracy and 85.58% confidence for that model and dataset.

1 of 1

Cites

Flagged first, then by citations

Research works printed but not linked: 8

  1. Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam Mc- Candlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In Advances in Neural Information Pro- cessing Systems 33: Annual Conference on Neu- ral Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual.
  2. PMLR.
  3. Kartik Gupta, Amir Rahimi, Thalaiyasingam Ajan- than, Thomas Mensink, Cristian Sminchisescu, and Richard Hartley. 2021. Calibration of neural net- works using splines. In 9th International Con- ference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenRe- view.net.
  4. Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. 2017. Simple and scalable predic- tive uncertainty estimation using deep ensembles. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Pro- cessing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pages 6402-6413.
  5. Matthias Minderer, Josip Djolonga, Rob Romijnders, Frances Hubis, Xiaohua Zhai, Neil Houlsby, Dustin Tran, and Mario Lucic. 2021. Revisiting the cal- ibration of modern neural networks. In Advances in Neural Information Processing Systems 34: An- nual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual, pages 15682-15694.
  6. Preslav Nakov, Sara Rosenthal, Zornitsa Kozareva, Veselin Stoyanov, Alan Ritter, and Theresa Wilson. 2013. SemEval-2013 task 2: Sentiment analysis in Twitter. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Pro- ceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013), pages 312- 320, Atlanta, Georgia, USA. Association for Com- putational Linguistics.
  7. Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. 2013b. Recursive deep models for semantic compositionality over a sentiment tree- bank. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Process- ing, pages 1631-1642, Seattle, Washington, USA. Association for Computational Linguistics.
  8. Adina Williams, Nikita Nangia, and Samuel Bowman. 2018b. A broad-coverage challenge corpus for sen- tence understanding through inference. In Proceed- ings of the 2018 Conference of the North American Chapter of the Association for Computational Lin- guistics: Human Language Technologies, Volume 1 (Long Papers), pages 1112-1122, New Orleans, Louisiana. Association for Computational Linguis- tics. · DOI · Paper page

Cited by

Sorted by citations

Related

Authors
Chen, Y., Yuan, L., Cui, G., Liu, Z., & Ji, H.
Affiliations
  • University of Illinois Urbana-Champaign (US)
  • Huazhong University of Science and Technology (CN)
  • Tsinghua University (CN)
Corresponding author
Chen, Yangyi
ORCID
Venue
Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)
Publisher
Association for Computational Linguistics
Published
2023
Pages
1343-1367
ISSN
0736-587X (ISSN-L)
DOI
10.18653/v1/2023.acl-long.75
OpenAlex
W4385571092
Cited by
22
References
62
Citations by year
2026: 6 · 2025: 7 · 2024: 5 · 2023: 4
FWCI
3.49
Type
conference paper
Language
en
Open access
Licence
cc-by
Locations
  • Proceedings of the conference - Association for Computational Linguistics. Meeting (publishedVersion, cc-by)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Funders
Tsinghua University
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
Keywords
pre-trained language models, model calibration, forecast uncertainty, overconfidence, label smoothing, confidence estimation

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-07.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-07.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review