The DOI matches its Crossref record; 38 of the 38 references it cites were found in a registry or an index.
On Mixup Training: Improved Calibration and Predictive Uncertainty for Deep Neural Networks
Open accessAbstract
Mixup~\cite{zhang2017mixup} is a recently proposed method for training deep neural networks where additional samples are generated during training by convexly combining random pairs of images and their associated labels. While simple to implement, it has been shown to be a surprisingly effective method of data augmentation for image classification: DNNs trained with mixup show noticeable gains in classification performance on a number of image classification benchmarks. In this work, we discuss a hitherto untouched aspect of mixup training -- the calibration and predictive uncertainty of models trained with mixup. We find that DNNs trained with mixup are significantly better calibrated -- i.e., the predicted softmax scores are much better indicators of the actual likelihood of a correct prediction -- than DNNs trained in the regular fashion. We conduct experiments on a number of image classification architectures and datasets -- including large-scale datasets like ImageNet -- and find this to be the case. Additionally, we find that merely mixing features does not result in the same calibration benefit and that the label smoothing in mixup training plays a significant role in improving calibration. Finally, we also observe that mixup-trained DNNs are less prone to over-confident predictions on out-of-distribution and random-noise data. We conclude that the typical overconfidence seen in neural networks, even on in-distribution data is likely a consequence of training with hard labels, suggesting that mixup be employed for classification tasks where predictive uncertainty is a significant concern.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
Very Deep Convolutional Networks for Large-Scale Image Recognition
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1409.1556 · Cited by 114,148
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333
Glove: Global Vectors for Word Representation
Pennington, J., Socher, R., & Manning, C. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1162 · Cited by 34,071
Rethinking the Inception Architecture for Computer Vision
Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., & Wojna, Z. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.308 · Cited by 32,138
Learning Multiple Layers of Features from Tiny Images
Krizhevsky, A. (2024) · doi:10.57702/zp44cu3g · Cited by 23,395
Convolutional Neural Networks for Sentence Classification
Kim, Y. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1181 · Cited by 14,831
Aggregated Residual Transformations for Deep Neural Networks
Xie, S., Girshick, R., Dollar, P., Tu, Z., & He, K. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.634 · Cited by 12,465
Dropout as a bayesian approximation: Representing model uncertainty in deep learning
Gal, Y., & Ghahramani, Z. (2015) · arXiv · doi:10.48550/arxiv.1506.02142 · Cited by 6,471
Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
Xiao, H., Rasul, K., & Vollgraf, R. (2017) · arXiv · doi:10.48550/arxiv.1708.07747 · Cited by 6,108
mixup: Beyond Empirical Risk Minimization
Zhang, H., Cisse, M., Dauphin, Y. N., & Lopez-Paz, D. (2017) · arXiv · doi:10.48550/arxiv.1710.09412 · Cited by 5,831
Universal Language Model Fine-tuning for Text Classification
Howard, J., & Ruder, S. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1031 · Cited by 4,100
Learning Word Vectors for Sentiment Analysis
Maas, A. L., Daly, R. E., Pham, P. T., Huang, D., Ng, A. Y., & Potts, C. (2011) · Cited by 3,751
On calibration of modern neural networks
Guo, C., Pleiss, G., Sun, Y., & Weinberger, K. Q. (2017) · arXiv · doi:10.48550/arxiv.1706.04599 · Cited by 3,435
On the Uniform Convergence of Relative Frequencies of Events to Their Probabilities
Vapnik, V. N., & Chervonenkis, A. Y. (2015) · Measures of Complexity · doi:10.1007/978-3-319-21852-6_3 · Cited by 3,105
What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?
Kendall, A., & Gal, Y. (2017) · arXiv · doi:10.48550/arxiv.1703.04977 · Cited by 3,088
An analysis of single-layer networks in unsupervised feature learning
Coates, A., Ng, A. Y., & Lee, H. (2011) · Cited by 2,758
Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
Goyal, P., Dollár, P., Girshick, R., Noordhuis, P., Wesolowski, L., Kyrola, A., Tulloch, A., Jia, Y., & He, K. (2017) · arXiv · doi:10.48550/arxiv.1706.02677 · Cited by 2,294
Miotto, R., Li, L., Kidd, B. A., & Dudley, J. T. (2016) · Scientific Reports · doi:10.1038/srep26094 · Cited by 1,710
Towards fully autonomous driving: Systems and algorithms
Levinson, J., Askeland, J., Becker, J., Dolson, J., Held, D., Kammel, S., Kolter, J. Z., Langer, D., Pink, O., Pratt, V., et al. (2011) · 2011 IEEE Intelligent Vehicles Symposium (IV) · doi:10.1109/ivs.2011.5940562 · Cited by 1,374
Cer, D., Yang, Y., Kong, S.-Y., Hua, N., Limtiaco, N., John, R. S., Constant, N., Guajardo-Cespedes, M., Yuan, S., Tar, C., et al. (2018) · arXiv · doi:10.48550/arxiv.1803.11175 · Cited by 1,267
Li, X., & Roth, D. (2002) · Proceedings of the 19th international conference on Computational linguistics - · doi:10.3115/1072228.1072378 · Cited by 1,234
When does label smoothing help?
Müller, R., Kornblith, S., & Hinton, G. (2019) · arXiv · doi:10.48550/arxiv.1906.02629 · Cited by 859
Evidential Deep Learning to Quantify Classification Uncertainty
Sensoy, M., Kaplan, L., & Kandemir, M. (2018) · arXiv · doi:10.48550/arxiv.1806.01768 · Cited by 694
Pang, B., & Lee, L. (2005) · arXiv (Cornell University) · doi:10.48550/arxiv.cs/0506075 · Cited by 678
Training Confidence-calibrated Classifiers for Detecting Out-of-Distribution SamplesPreprint
Lee, K., Lee, H., Lee, K., & Shin, J. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1711.09325 · Cited by 664
Text Classification Improved by Integrating Bidirectional LSTM with Two-dimensional Max Pooling
Zhou, P., Qi, Z., Zheng, S., Xu, J., Bao, H., & Xu, B. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1611.06639 · Cited by 434
Fine-tuned Language Models for Text Classification.Preprint
Howard, J., & Ruder, S. (2018) · arXiv (Cornell University) · Cited by 357
Data Augmentation by Pairing Samples for Images Classification
Inoue, H. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1801.02929 · Cited by 319
MixUp as Locally Linear Out-of-Manifold Regularization
Guo, H., Mao, Y., & Zhang, R. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.33013714 · Cited by 257
Chapelle, O., Weston, J., Bottou, L., & Vapnik, V. N. (2000) · Cited by 244
Selective Classification for Deep Neural Networks
Geifman, Y., & El-Yaniv, R. (2017) · arXiv · doi:10.48550/arxiv.1705.08500 · Cited by 220
Combating Label Noise in Deep Learning Using AbstentionPreprint
Thulasidasan, S., Bhattacharya, T., Bilmes, J. A., Chennupati, G., & Mohd-Yusof, J. (2019) · International Conference on Machine Learning · doi:10.48550/arxiv.1905.10964 · Cited by 201
Self-citationA baseline for detecting misclassified and out-of-distribution examples in neural networks
Hendrycks, D. (2025) · arXiv (Cornell University) · doi:10.57702/9gs720u1 · Cited by 196
An impact assessment of machine learning risk forecasts on parole board decisions and recidivism
Berk, R. (2017) · Journal of Experimental Criminology · doi:10.1007/s11292-017-9286-2 · Cited by 169
Augmenting Data with Mixup for Sentence Classification: An Empirical StudyPreprint
Guo, H., Mao, Y., & Zhang, R. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1905.08941 · Cited by 140
Understanding Mixup Training Methods
Liang, D., Yang, F., Zhang, T., & Yang, P. C. (2018) · IEEE Access · doi:10.1109/access.2018.2872698 · Cited by 122
Manifold Mixup: Encouraging Meaningful On-Manifold Interpolation as a Regularizer.Preprint
Verma, V., Lamb, A., Beckham, C., Courville, A. C., Mitliagkas, I., & Bengio, Y. (2018) · arXiv (Cornell University) · Cited by 41
Cited by
A review of uncertainty quantification in deep learning: Techniques, applications and challenges
Abdar, M., Pourpanah, F., Hussain, S., Rezazadegan, D., Liu, L., Ghavamzadeh, M., Fieguth, P., Cao, X., Khosravi, A., Acharya, U. R., et al. (2021) · Information Fusion · doi:10.1016/j.inffus.2021.05.008 · Cited by 3,111
“[236] investigated the predictive uncertainty and calibration of models trained with”
BackgroundA survey of uncertainty in deep neural networks
Gawlikowski, J., Tassi, C. R. N., Ali, M., Lee, J., Humt, M., Feng, J., Kruspe, A., Triebel, R., Jung, P., Roscher, R., et al. (2023) · Artificial Intelligence Review · doi:10.1007/s10462-023-10562-9 · Cited by 1,406
Generalized Out-of-Distribution Detection: A Survey
Yang, J., Zhou, K., Li, Y., & Liu, Z. (2024) · International Journal of Computer Vision · doi:10.1007/s11263-024-02117-4 · Cited by 884
DyTox: Transformers for Continual Learning with DYnamic TOken eXpansion
Douillard, A., Rame, A., Couairon, G., & Cord, M. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.00907 · Cited by 357
Improving Calibration for Long-Tailed Recognition
Zhong, Z., Cui, J., Liu, S., & Jia, J. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.01622 · Cited by 282
Metadata
- Authors
- Thulasidasan, S., Chennupati, G., Bilmes, J., Bhattacharya, T., & Michalak, S.
- Published
- 2019-12-04
- OpenAlex
- W2945125794
- Cited by
- 410
- References
- 38
- Type
- report
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.