Symmetric Cross Entropy for Robust Learning With Noisy Labels
Open accessAbstract
Training accurate deep neural networks (DNNs) in the presence of noisy labels is an important and challenging task. Though a number of approaches have been proposed for learning with noisy labels, many open issues remain. In this paper, we show that DNN learning with Cross Entropy (CE) exhibits overfitting to noisy labels on some classes ("easy" classes), but more surprisingly, it also suffers from significant under learning on some other classes ("hard" classes). Intuitively, CE requires an extra term to facilitate learning of hard classes, and more importantly, this term should be noise tolerant, so as to avoid overfitting to noisy labels. Inspired by the symmetric KL-divergence, we propose the approach of Symmetric cross entropy Learning (SL), boosting CE symmetrically with a noise robust counterpart Reverse Cross Entropy (RCE). Our proposed SL approach simultaneously addresses both the under learning and overfitting problem of CE in the presence of noisy labels. We provide a theoretical analysis of SL and also empirically show, on a range of benchmark and real-world datasets, that SL outperforms state-of-the-art methods. We also show that SL can be easily incorporated into existing methods in order to further enhance their performance.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The symmetric cross-entropy loss function reduces the influence of noise and prevents overfitting in medical image datasets.
“The symmetric cross-entropy loss function can reduce the influence of noise and prevent overfitting [31].”
From Method for diagnosis of acute lymphoblastic leukemia based on vit-cnn ensemble model (2021) · cites this paper as [31]
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · see citing works
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333 · see citing works
Gradient-based learning applied to document recognition
Lecun, Y., Bottou, L., Bengio, Y., & Haffner, P. (1998) · Proceedings of the IEEE · doi:10.1109/5.726791 · Cited by 58,805 · see citing works
(2008) · Journal of Machine Learning Research · Cited by 33,051 · see citing works
Rethinking the Inception Architecture for Computer Vision
Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., & Wojna, Z. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.308 · Cited by 32,138 · see citing works
Learning Multiple Layers of Features from Tiny Images
Krizhevsky, A. (2024) · doi:10.57702/zp44cu3g · Cited by 23,890 · see citing works
Generalized Cross Entropy Loss for Training Deep Neural Networks with Noisy Labels
Zhang, Z., & Sabuncu, M. R. (2018) · arXiv · doi:10.48550/arxiv.1805.07836 · Cited by 1,907 · cited 13× · see citing works
“Whilst for asymmetric noisy labels, flipping labels only occurs within a specific set of classes [16, 29], for example, for MNIST, flipping 2 → 7, 3 → 8, 5 ↔ 6 and 7 → 1; for CIFAR-10, flipping TRUCK → AUTOMOBILE, BIRD → AIRPLANE, DEER → HORSE, CAT ↔ DOG; for CIFAR-100, the 100 classes are grouped into 20 super-classe…”
BackgroundMethodInfluentialOpening the Black Box of Deep Neural Networks via InformationPreprint
Shwartz-Ziv, R., & Tishby, N. (2017) · arXiv.org · doi:10.48550/arxiv.1703.00810 · Cited by 1,674 · see citing works
Co-teaching: Robust Training of Deep Neural Networks with Extremely Noisy LabelsPreprint
Han, B., Yao, Q., Yu, X., Niu, G., Xu, M., Hu, W., Tsang, I. W.-H., & Sugiyama, M. (2018) · arXiv (Cornell University) · doi:10.5555/3327757.3327944 · Cited by 1,674 · see citing works
Making Deep Neural Networks Robust to Label Noise: A Loss Correction Approach
Patrini, G., Rozza, A., Menon, A. K., Nock, R., & Qu, L. (2017) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2017.240 · Cited by 1,464 · see citing works
A Closer Look at Memorization in Deep Networks
Arpit, D., Jastrzębski, S., Ballas, N., Krueger, D., Bengio, E., Kanwal, M. S., Maharaj, T., Fischer, A., Courville, A., Bengio, Y., et al. (2017) · arXiv · doi:10.48550/arxiv.1706.05394 · Cited by 1,307 · see citing works
Learning from massive noisy labeled data for image classification
Xiao, T., Xia, T., Yang, Y., Huang, C., & Wang, X. (2015) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2015.7298885 · Cited by 971 · see citing works
Robust Loss Functions under Label Noise for Deep Neural Networks
Ghosh, A., Kumar, H., & SASTRY, P. S. (2017) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v31i1.10894 · Cited by 932 · see citing works
MentorNet: Learning Data-Driven Curriculum for Very Deep Neural Networks on Corrupted LabelsPreprint
Jiang, L., Zhou, Z., Leung, T., Li, L.-J., & Fei-Fei, L. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1712.05055 · Cited by 834 · see citing works
Training Deep Neural Networks on Noisy Labels with BootstrappingPreprint
Reed, S., Lee, H., Anguelov, D., et al. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1412.6596 · Cited by 762 · see citing works
Joint Optimization Framework for Learning with Noisy Labels
Tanaka, D., Ikami, D., Yamasaki, T., & Aizawa, K. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00582 · Cited by 737 · see citing works
Regularizing Neural Networks by Penalizing Confident Output Distributions
Pereyra, G., Tucker, G., Chorowski, J., Kaiser, Ł., & Hinton, G. (2017) · arXiv · doi:10.48550/arxiv.1701.06548 · Cited by 655 · see citing works
Learning from Noisy Labels with Distillation
Li, Y., Yang, J., Song, Y., Cao, L., Luo, J., & Li, L.-J. (2017) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.211 · Cited by 557 · see citing works
Training Convolutional Networks with Noisy LabelsPreprint
Sukhbaatar, S., Bruna, J., Paluri, M., Bourdev, L. D., & Fergus, R. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1406.2080 · Cited by 532 · see citing works
Training deep neural-networks using a noise adaptation layer
Goldberger, J., & Ben-Reuven, E. (2017) · International Conference on Learning Representations · Cited by 511 · see citing works
CleanNet: Transfer Learning for Scalable Image Classifier Training with Label Noise
Lee, K.-H., He, X., Zhang, L., & Yang, L. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00571 · Cited by 506 · see citing works
Learning from Noisy Large-Scale Datasets with Minimal Supervision
Veit, A., Alldrin, N., Chechik, G., Krasin, I., Gupta, A., & Belongie, S. (2017) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2017.696 · Cited by 480 · see citing works
Generalized cross entropy loss for training deep neural networks with noisy labelsPreprint
Zhang, Z., & Sabuncu, M. R. (2018) · arXiv (Cornell University) · Cited by 450 · see citing works
MentorNet: Learning Data-Driven Curriculum for Very Deep Neural Networks on Corrupted Labels
Jiang, L., Zhou, Z., Leung, T., Li, L.-J., & Fei-Fei, L. (2018) · International Conference on Machine Learning · Cited by 371 · see citing works
Iterative Learning with Open-set Noisy Labels
Wang, Y., Liu, W., Ma, X., Bailey, J., Hongyuan Zha, B., Song, L., & Xia, S. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00906 · Cited by 339 · see citing works
Self-citationDimensionality-Driven Learning with Noisy LabelsPreprint
Ma, X., Wang, Y., Houle, M. E., Zhou, S., Erfani, S., Xia, S., Wijewickrema, S. N. R., & Bailey, J. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1806.02612 · Cited by 296 · see citing works
Self-citationMasking: A New Perspective of Noisy SupervisionPreprint
Han, B., Yao, J., Niu, G., Zhou, M., Tsang, I. W.-H., Zhang, Y., & Sugiyama, M. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1805.08193 · Cited by 207 · see citing works
Toward Robustness against Label Noise in Training Deep Discriminative Neural Networks
Vahdat, A. (2017) · neural information processing systems · Cited by 179 · see citing works
Toward Robustness against Label Noise in Training Deep Discriminative Neural NetworksPreprint
Vahdat, A. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1706.00038 · Cited by 119 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Method for diagnosis of acute lymphoblastic leukemia based on vit-cnn ensemble model
Jiang, Z., Dong, Z., Wang, L., & Jiang, W. (2021) · Computational Intelligence and Neuroscience · doi:10.1155/2021/7529893 · Cited by 124 · see citing works
Model Generalization on COVID-19 Fake News Detection
Bang, Y., Ishii, E., Cahyawijaya, S., Ji, Z., & Fung, P. (2021) · Communications in Computer and Information Science · doi:10.1007/978-3-030-73696-5_13 · Cited by 52 · see citing works
Hallaji, E., Razavi-Far, R., Saif, M., & Herrera-Viedma, E. (2023) · Knowledge-Based Systems · doi:10.1016/j.knosys.2023.110384 · Cited by 41 · see citing works
Exploring the Robustness of In-Context Learning with Noisy Labels
Cheng, C., Yu, X., Wen, H., Sun, J., Yue, G., Zhang, Y., & Wei, Z. (2025) · ICASSP 2025 - 2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp49660.2025.10889281 · Cited by 11 · see citing works
- Authors
- Wang, Y., Ma, X., Chen, Z., Luo, Y., Yi, J., & Bailey, J.
- Affiliations
- Shanghai Jiao Tong University (CN)
- The University of Melbourne (AU)
- Venue
- 2019 IEEE/CVF International Conference on Computer Vision (ICCV)
- Publisher
- IEEE
- Published
- 2019-10
- Pages
- 322-330
- OpenAlex
- W2981873476
- Cited by
- 1,046
- References
- 29
- Citations by year
- 2026: 92 · 2025: 173 · 2024: 229 · 2023: 216 · 2022: 145 · 2021: 150 · 2020: 37 · 2019: 4
- FWCI
- 32.14
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- IEEE/CVF International Conference on Computer Vision (ICCV) (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- Figshare (submittedVersion, cc-by)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Data Classification
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
- Keywords
- label noise, noise-robust learning, deep neural network
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
