CoChat
Sign Up
PreprintCited by 100Machine Learning and Data ClassificationDomain Adaptation and Few-Shot LearningMachine Learning and Algorithms

A Second-Order Approach to Learning with Instance-Dependent Label Noise

Open access
Zhu, Zhaowei, Liu, Tongliang, Liu, Yang·2020·arXiv
DOI 10.48550/arxiv.2012.11854arXiv 2012.11854
Open this paper in CoChat — free accountOpen PDF

Abstract

The presence of label noise often misleads the training of deep neural networks. Departing from the recent literature which largely assumes the label noise rate is only determined by the true label class, the errors in human-annotated labels are more likely to be dependent on the difficulty levels of tasks, resulting in settings with instance-dependent label noise. We first provide evidences that the heterogeneous instance-dependent label noise is effectively down-weighting the examples with higher noise rates in a non-uniform way and thus causes imbalances, rendering the strategy of directly applying methods for class-dependent label noise questionable. Built on a recent work peer loss [24], we then propose and study the potentials of a second-order approach that leverages the estimation of several covariance terms defined between the instance-dependent noise rates and the Bayes optimal label. We show that this set of second-order statistics successfully captures the induced imbalances. We further proceed to show that with the help of the estimated second-order statistics, we identify a new loss function whose expected risk of a classifier under instance-dependent label noise is equivalent to a new problem with only class-dependent label noise. This fact allows us to apply existing solutions to handle this better-studied setting. We provide an efficient procedure to estimate these second-order statistics without accessing either ground truth labels or prior knowledge of the noise rates. Experiments on CIFAR10 and CIFAR100 with synthetic instance-dependent label noise and Clothing1M with real-world human label noise verify our approach. Our implementation is available at https://github.com/UCSC-REAL/CAL.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2012.11854

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Instance-dependent label noise is more realistic than class-dependent label noise, as human error is likely to depend on each instance's features.

    “In line with previous work on learning with noisy labels17,18, we argue that instance-dependent label noise (IDN) is more realistic, as human error is likely to depend on each instance’s features.”

    From A benchmarking framework and dataset for learning to defer in human-AI decision-making (2025) · cites this paper as [17]

  • Because of the use of a feature-dependent label-noise approach, the framework is not applicable to decision-making tasks where no ground truth labels are present.

    “Due to the use of a feature-dependent label-noise approach17,18, our framework is not applicable to decision-making tasks where no ground truth labels are present.”

    From A benchmarking framework and dataset for learning to defer in human-AI decision-making (2025) · cites this paper as [17]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

No references could be resolved yet.

Research works printed but not linked: 49

  1. Ehsan Amid, Manfred KK Warmuth, Rohan Anil, and Tomer Koren. Robust bi-tempered logistic loss based on bregman divergences. In Advances in Neural In- formation Processing Systems, pages 14987-14996, 2019.
  2. Ehsan Amid, Manfred K Warmuth, and Sriram Srini- vasan. Two-temperature logistic regression based on the tsallis divergence. In The 22nd International Con- ference on Artificial Intelligence and Statistics, pages 2388-2396. PMLR, 2019. 2
  3. Peter L Bartlett, Michael I Jordan, and Jon D McAuliffe. Convexity, classification, and risk bounds. Journal of the American Statistical Association, 101(473):138-156, 2006.
  4. Antonin Berthon, Bo Han, Gang Niu, Tongliang Liu, and Masashi Sugiyama. Confidence scores make instance-dependent label-noise learning possi- ble. arXiv preprint arXiv:2001.03772, 2020. · arXiv
  5. Hao Cheng, Zhaowei Zhu, Xingyu Li, Yifei Gong, Xing Sun, and Yang Liu. Learning with instance- dependent label noise: A sample sieve approach. In International Conference on Learning Representa- tions, 2021. 1, 2, 3, 4, 5, 6, 7, 8, 18
  6. Jiacheng Cheng, Tongliang Liu, Kotagiri Ramamo- hanarao, and Dacheng Tao. Learning with bounded instance-and label-dependent label noise. In Proceed- ings of the 37th International Conference on Machine Learning, ICML '20, 2020. 2, 5, 6, 17
  7. Tongtong Fang, Nan Lu, Gang Niu, and Masashi Sugiyama. Rethinking importance weighting for deep learning under distribution shift. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 11996-12007.
  8. Aritra Ghosh, Himanshu Kumar, and PS Sastry. Ro- bust loss functions under label noise for deep neural networks. In Thirty-First AAAI Conference on Artifi- cial Intelligence, 2017. 2
  9. Maoguo Gong, Hao Li, Deyu Meng, Qiguang Miao, and Jia Liu. Decomposition-based evolutionary multiobjective optimization to self-paced learning. IEEE Transactions on Evolutionary Computation, 23(2):288-302, 2018.
  10. Bo Han, Gang Niu, Xingrui Yu, Quanming Yao, Miao Xu, Ivor Tsang, and Masashi Sugiyama. Sigua: For- getting may make learning with noisy labels more ro- bust. In International Conference on Machine Learn- ing, pages 4006-4016. PMLR, 2020. 1
  11. Bo Han, Quanming Yao, Tongliang Liu, Gang Niu, Ivor W Tsang, James T Kwok, and Masashi Sugiyama. A survey of label-noise representation learning: Past, present and future. arXiv preprint arXiv:2011.04406, 2020. 1 · arXiv
  12. Bo Han, Quanming Yao, Xingrui Yu, Gang Niu, Miao Xu, Weihua Hu, Ivor Tsang, and Masashi Sugiyama. Co-teaching: Robust training of deep neural networks with extremely noisy labels. In Advances in neu- ral information processing systems, pages 8527-8537, 2018. 2, 7, 8
  13. Jiangfan Han, Ping Luo, and Xiaogang Wang. Deep self-learning from noisy labels. In Proceedings of the IEEE International Conference on Computer Vision, pages 5138-5147, 2019. 2, 5
  14. Jiayuan Huang, Arthur Gretton, Karsten Borgwardt, Bernhard Schölkopf, and Alex J Smola. Correcting sample selection bias by unlabeled data. In Advances in neural information processing systems, pages 601- 608, 2007. 6, 17
  15. Lu Jiang, Di Huang, Mason Liu, and Weilong Yang. Beyond synthetic noise: Deep learning on controlled noisy labels. In Proceedings of the 37th International Conference on Machine Learning, volume 119, pages 4804-4815. PMLR, 13-18 Jul 2020. 1
  16. Lu Jiang, Zhengyuan Zhou, Thomas Leung, Li-Jia Li, and Li Fei-Fei. Mentornet: Learning data-driven cur- riculum for very deep neural networks on corrupted la- bels. In International Conference on Machine Learn- ing, pages 2304-2313. PMLR, 2018. 2
  17. Kuang-Huei Lee, Xiaodong He, Lei Zhang, and Lin- jun Yang. Cleannet: Transfer learning for scalable im- age classifier training with label noise. In Proceedings of the IEEE Conference on Computer Vision and Pat- tern Recognition, pages 5447-5456, 2018. 2
  18. Junnan Li, Richard Socher, and Steven C.H. Hoi. Dividemix: Learning with noisy labels as semi- supervised learning. In International Conference on Learning Representations, 2020. 2
  19. Xuefeng Li, Tongliang Liu, Bo Han, Gang Niu, and Masashi Sugiyama. Provably end-to-end label- noise learning without anchor points. arXiv preprint arXiv:2102.02400, 2021. 2 · arXiv
  20. Yuncheng Li, Jianchao Yang, Yale Song, Liangliang Cao, Jiebo Luo, and Li-Jia Li. Learning from noisy labels with distillation. In Proceedings of the IEEE International Conference on Computer Vision, pages 1910-1918, 2017. 2
  21. Tongliang Liu and Dacheng Tao. Classification with noisy labels by importance reweighting. IEEE Trans- actions on pattern analysis and machine intelligence, 38(3):447-461, 2015. 2
  22. Yang Liu and Hongyi Guo. Peer loss functions: Learn- ing from noisy labels without knowing noise rates. In Proceedings of the 37th International Conference on Machine Learning, ICML '20, 2020. 1, 2, 3, 4, 5, 7, 8, 12, 16
  23. Philip M Long and Rocco A Servedio. Random clas- sification noise defeats all convex potential boosters. Machine learning, 78(3):287-304, 2010. 2
  24. Naresh Manwani and PS Sastry. Noise tolerance under risk minimization. IEEE transactions on cybernetics, 43(3):1146-1151, 2013. 2
  25. Nagarajan Natarajan, Inderjit S Dhillon, Pradeep K Ravikumar, and Ambuj Tewari. Learning with noisy labels. In Advances in neural information processing systems, pages 1196-1204, 2013. 1, 2, 3
  26. Duc Tam Nguyen, Chaithanya Kumar Mummadi, Thi Phuong Nhung Ngo, Thi Hoai Phuong Nguyen, Laura Beggel, and Thomas Brox. Self: Learning to filter noisy labels with self-ensembling. In International Conference on Learning Representations, 2020. 2
  27. Giorgio Patrini, Alessandro Rozza, Aditya Kr- ishna Menon, Richard Nock, and Lizhen Qu. Mak- ing deep neural networks robust to label noise: A loss correction approach. In Proceedings of the IEEE Con- ference on Computer Vision and Pattern Recognition, pages 1944-1952, 2017. 1, 2, 7, 8
  28. Giorgio Patrini, Alessandro Rozza, Aditya Kr- ishna Menon, Richard Nock, and Lizhen Qu. Making deep neural networks robust to label noise: A loss cor- rection approach. In The IEEE Conference on Com- puter Vision and Pattern Recognition (CVPR), July 2017. 2, 5
  29. Jun Shu, Qian Zhao, Keyu Chen, Zongben Xu, and Deyu Meng. Learning adaptive loss for robust learning with noisy labels. arXiv preprint arXiv:2002.06482, 2020. 2 · arXiv
  30. Arash Vahdat. Toward robustness against label noise in training deep discriminative neural networks. In Advances in Neural Information Processing Systems, pages 5596-5605, 2017. 1
  31. Andreas Veit, Neil Alldrin, Gal Chechik, Ivan Krasin, Abhinav Gupta, and Serge Belongie. Learning from noisy large-scale datasets with minimal supervision. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 839-847, 2017. 2
  32. Qizhou Wang, Bo Han, Tongliang Liu, Gang Niu, Jian Yang, and Chen Gong. Tackling instance-dependent label noise via a universal probabilistic model. arXiv preprint arXiv:2101.05467, 2021. 1, 4 · arXiv
  33. Qizhou Wang, Jiangchao Yao, Chen Gong, Tongliang Liu, Mingming Gong, Hongxia Yang, and Bo Han. Learning with group noise. arXiv preprint arXiv:2103.09468, 2021. 1 · arXiv
  34. Yisen Wang, Xingjun Ma, Zaiyi Chen, Yuan Luo, Jin- feng Yi, and James Bailey. Symmetric cross entropy for robust learning with noisy labels. In Proceedings of the IEEE International Conference on Computer Vi- sion, pages 322-330, 2019. 2
  35. Hongxin Wei, Lei Feng, Xiangyu Chen, and Bo An. Combating noisy labels by agreement: A joint training method with co-regularization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pat- tern Recognition, pages 13726-13735, 2020. 2, 7, 8
  36. Xiaobo Xia, Tongliang Liu, Bo Han, Chen Gong, Nan- nan Wang, Zongyuan Ge, and Yi Chang. Robust early- learning: Hindering the memorization of noisy labels. In International Conference on Learning Representa- tions, 2021. 1
  37. Xiaobo Xia, Tongliang Liu, Bo Han, Nannan Wang, Jiankang Deng, Jiatong Li, and Yinian Mao. Extended T: Learning with mixed closed-set and open-set noisy labels. arXiv preprint arXiv:2012.00932, 2020. 2 · arXiv
  38. Xiaobo Xia, Tongliang Liu, Bo Han, Nannan Wang, Mingming Gong, Haifeng Liu, Gang Niu, Dacheng Tao, and Masashi Sugiyama. Part-dependent label noise: Towards instance-dependent label noise. In Advances in Neural Information Processing Systems, volume 33, pages 7597-7610, 2020. 1, 2, 4, 5, 7, 8, 17, 18
  39. Xiaobo Xia, Tongliang Liu, Nannan Wang, Bo Han, Chen Gong, Gang Niu, and Masashi Sugiyama. Are anchor points really indispensable in label-noise learn- ing? In Advances in Neural Information Processing Systems, pages 6838-6849, 2019. 2, 5, 7, 8
  40. Tong Xiao, Tian Xia, Yi Yang, Chang Huang, and Xiaogang Wang. Learning from massive noisy la- beled data for image classification. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2691-2699, 2015. 1, 7, 8
  41. Yilun Xu, Peng Cao, Yuqing Kong, and Yizhou Wang. L_dmi: A novel information-theoretic loss function for training deep nets robust to label noise. In Ad- vances in Neural Information Processing Systems, volume 32, 2019. 2, 3, 7, 8
  42. Quanming Yao, Hansi Yang, Bo Han, Gang Niu, and James T Kwok. Searching to exploit memorization effect in learning with noisy labels. In Proceedings of the 37th International Conference on Machine Learn- ing, ICML '20, 2020. 2, 5
  43. Yu Yao, Tongliang Liu, Bo Han, Mingming Gong, Jiankang Deng, Gang Niu, and Masashi Sugiyama. Dual T: Reducing estimation error for transition ma- trix in label-noise learning. In Advances in Neural Information Processing Systems, volume 33, pages 7260-7271, 2020. 2
  44. Xingrui Yu, Bo Han, Jiangchao Yao, Gang Niu, Ivor W Tsang, and Masashi Sugiyama. How does disagreement help generalization against label corrup- tion? arXiv preprint arXiv:1901.04215, 2019. 2, 7, 8 · arXiv
  45. Chiyuan Zhang, Samy Bengio, Moritz Hardt, Ben- jamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. arXiv preprint arXiv:1611.03530, 2016. 1 · arXiv
  46. Hongyi Zhang, Moustapha Cisse, Yann N. Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. In International Conference on Learn- ing Representations, 2018. 18
  47. Yivan Zhang, Gang Niu, and Masashi Sugiyama. Learning noise transition matrix from only noisy la- bels via total variation regularization. arXiv preprint arXiv:2102.02414, 2021. 2 · arXiv
  48. Zhilu Zhang and Mert Sabuncu. Generalized cross en- tropy loss for training deep neural networks with noisy labels. In Advances in neural information processing systems, pages 8778-8788, 2018. 2, 7, 8
  49. Zhaowei Zhu, Yiwen Song, and Yang Liu. Cluster- ability as an alternative to anchor points when learning with noisy labels. arXiv preprint arXiv:2102.05291, 2021. · arXiv

Cited by

Sorted by citations

Authors
Zhu, Z., Liu, T., & Liu, Y.
Affiliations
  • University of California, Santa Cruz (US)
  • The University of Sydney (AU)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2020
Pages
10108-10118
ISSN
1063-6919 (ISSN-L)
DOI
10.48550/arxiv.2012.11854
arXiv
2012.11854
OpenAlex
W3170414933
Cited by
100
References
0
Citations by year
2026: 6 · 2025: 16 · 2024: 26 · 2023: 21 · 2022: 14 · 2021: 14 · 2020: 3
FWCI
8.59
Type
conference paper
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Funders
  • National Science Foundation (2007951)
  • Australian Research Council
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Data Classification
  • Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Algorithms
Keywords
second-order statistics, covariance matrix estimation, class imbalance, deep neural network training

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review