CoChat
Sign Up
PreprintCited by 57Machine Learning and Data ClassificationMachine Learning and AlgorithmsDomain Adaptation and Few-Shot Learning

Confidence Scores Make Instance-dependent Label-noise Learning Possible

Open access
Berthon, Antonin, Han, Bo, Niu, Gang, Liu, Tongliang, Sugiyama, Masashi·2020·arXiv
DOI 10.48550/arxiv.2001.03772arXiv 2001.03772
Open this paper in CoChat — free accountOpen PDF

Abstract

In learning with noisy labels, for every instance, its label can randomly walk to other classes following a transition distribution which is named a noise model. Well-studied noise models are all instance-independent, namely, the transition depends only on the original label but not the instance itself, and thus they are less practical in the wild. Fortunately, methods based on instance-dependent noise have been studied, but most of them have to rely on strong assumptions on the noise models. To alleviate this issue, we introduce confidence-scored instance-dependent noise (CSIDN), where each instance-label pair is equipped with a confidence score. We find with the help of confidence scores, the transition distribution of each instance can be approximately estimated. Similarly to the powerful forward correction for instance-independent noise, we propose a novel instance-level forward correction for CSIDN. We demonstrate the utility and effectiveness of our method through multiple experiments under synthetic label noise and real-world unknown noise.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2001.03772

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Instance-dependent label noise is more realistic than class-dependent label noise, as human error is likely to depend on each instance's features.

    “In line with previous work on learning with noisy labels17,18, we argue that instance-dependent label noise (IDN) is more realistic, as human error is likely to depend on each instance’s features.”

    From A benchmarking framework and dataset for learning to defer in human-AI decision-making (2025) · cites this paper as [18]

  • Because of the use of a feature-dependent label-noise approach, the framework is not applicable to decision-making tasks where no ground truth labels are present.

    “Due to the use of a feature-dependent label-noise approach17,18, our framework is not applicable to decision-making tasks where no ground truth labels are present.”

    From A benchmarking framework and dataset for learning to defer in human-AI decision-making (2025) · cites this paper as [18]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

No references could be resolved yet.

Research works printed but not linked: 54

  1. Angluin, D. and Laird, P. Learning from noisy examples. Machine Learning, 2(4):343–370, 1988.
  2. Arachie, C. and Huang, B. Adversarial label learning. In AAAI, 2019.
  3. Arpit, D., Jastrzębski, S., Ballas, N., Krueger, D., Bengio, E., Kanwal, M. S., Maharaj, T., Fischer, A., Courville, A., Bengio, Y., et al. A closer look at memorization in deep networks. In ICML, 2017.
  4. Bartlett, P. L., Jordan, M. I., and McAuliffe, J. D. Convexity, classification, and risk bounds. Journal of the American Statistical Association, 101(473):138–156, 2006.
  5. Bootkrajang, J. and Chaijaruwanich, J. Towards instance-dependent label noise-tolerant classification: a probabilistic approach. Pattern Analysis and Applications, pp. 1–17, 2018.
  6. Branson, S., Van Horn, G., and Perona, P. Lean crowdsourcing: Combining humans and machines in an online system. In CVPR, 2017.
  7. Charoenphakdee, N., Lee, J., and Sugiyama, M. On Symmetric Losses for Learning from Corrupted Labels. ICML, 2019.
  8. Cheng, H., Zhu, Z., Li, X., Gong, Y., Sun, X., and Liu, Y. Learning with instance-dependent label noise: A sample sieve approach. arXiv preprint arXiv:2010.02347, 2020a. · arXiv
  9. Cheng, J., Liu, T., Ramamohanarao, K., and Tao, D. Learning with bounded instance and label-dependent label noise. In ICML, 2020b.
  10. Du, J. and Cai, Z. Modelling class noise with symmetric and asymmetric distributions. In AAAI, 2015.
  11. Ghosh, A., Kumar, H., and Sastry, P. Robust loss functions under label noise for deep neural networks. In AAAI, 2017.
  12. Gneiting, T. and Raftery, A. E. Strictly proper scoring rules, prediction, and estimation. Journal of the American Statistical Association, 102(477):359–378, 2007.
  13. Goldberger, J. and Ben-Reuven, E. Training deep neural-networks using a noise adaptation layer. In ICLR, 2017.
  14. Guan, M. Y., Gulshan, V., Dai, A. M., and Hinton, G. E. Who said what: Modeling individual labelers improves classification. In AAAI, 2018.
  15. Han, B., Yao, Q., Yu, X., Niu, G., Xu, M., Hu, W., Tsang, I., and Sugiyama, M. Co-teaching: Robust training of deep neural networks with extremely noisy labels. In NeurIPS, 2018.
  16. Ishida, T., Niu, G., and Sugiyama, M. Binary classification from positive-confidence data. In NeurIPS, 2018.
  17. Jiang, L., Zhou, Z., Leung, T., Li, L.-J., and Fei-Fei, L. Mentornet: Learning data-driven curriculum for very deep neural networks on corrupted labels. In ICML, 2018.
  18. Kearns, M. Efficient noise-tolerant learning from statistical queries. Proceedings of the twenty-fifth annual ACM symposium on Theory of computing - STOC 93, 1993.
  19. Khetan, A., Lipton, Z. C., and Anandkumar, A. Learning from noisy singly-labeled data. In ICLR, 2018.
  20. Laine, S. and Aila, T. Temporal Ensembling for Semi-Supervised Learning. ICLR, 2017.
  21. Liu, T. and Tao, D. Classification with noisy labels by importance reweighting. IEEE Transactions on pattern analysis and machine intelligence, 38(3):447–461, 2015.
  22. Ma, X., Wang, Y., Houle, M. E., Zhou, S., Erfani, S., Xia, S., Wijewickrema, S., and Bailey, J. Dimensionality- driven learning with noisy labels. In ICML, 2018.
  23. Manwani, N. and Sastry, P. S. Noise tolerance under risk minimization. IEEE Transactions on Cybernetics, 43:1146–1151, 2013.
  24. Masnadi-shirazi, H. and Vasconcelos, N. On the Design of Loss Functions for Classification: theory, robustness to outliers, and SavageBoost. In NeurIPS. 2009.
  25. Menon, A., Van Rooyen, B., Ong, C. S., and Williamson, B. Learning from corrupted binary labels via class-probability estimation. In ICML, pp. 125–134, 2015.
  26. Menon, A. K., Van Rooyen, B., and Natarajan, N. Learning from binary labels with instance-dependent corruption. arXiv preprint arXiv:1605.00751, 2016. · arXiv
  27. Menon, A. K., van Rooyen, B., and Natarajan, N. Learning from binary labels with instance-dependent noise. Machine Learning, 107(8-10):1561–1595, September 2018.
  28. Miyato, T., Maeda, S.-i., Koyama, M., and Ishii, S. Virtual adversarial training: a regularization method for supervised and semi-supervised learning. IEEE transactions on pattern analysis and machine intelligence, 41(8):1979–1993, 2018.
  29. Natarajan, N., Dhillon, I. S., Ravikumar, P. K., and Tewari, A. Learning with Noisy Labels. In NeurIPS. 2013.
  30. Nock, R. and Nielsen, F. On the efficient minimization of classification calibrated surrogates. In NeurIPS, 2009.
  31. Patrini, G., Nielsen, F., Nock, R., and Carioni, M. Loss factorization, weakly supervised learning and label noise robustness. In ICML, 2016.
  32. Patrini, G., Rozza, A., Krishna Menon, A., Nock, R., and Qu, L. Making deep neural networks robust to label noise: A loss correction approach. In CVPR, 2017.
  33. Ratner, A., Bach, S. H., Ehrenberg, H., Fries, J., Wu, S., and Ré, C. Snorkel: Rapid training data creation with weak supervision. The VLDB Journal, 29(2):709–730, 2020.
  34. Ratner, A. J., De Sa, C. M., Wu, S., Selsam, D., and Ré, C. Data programming: Creating large training sets, quickly. In NeurIPS, 2016.
  35. Raykar, V. C., Yu, S., Zhao, L. H., Jerebko, A., Florin, C., Valadez, G. H., Bogoni, L., and Moy, L. Supervised learning from multiple experts: whom to trust when everyone lies a bit. In ICML, 2009.
  36. Reed, S., Lee, H., Anguelov, D., Szegedy, C., Erhan, D., and Rabinovich, A. Training deep neural networks on noisy labels with bootstrapping. ICLR, 2015.
  37. Reid, M. D. and Williamson, R. C. Composite binary losses. Journal of Machine Learning Research, 11(Sep): 2387–2422, 2010.
  38. Scott, C., Blanchard, G., and Handy, G. Classification with asymmetric label noise: Consistency and maximal denoising. In COLT, pp. 489–511, 2013.
  39. Shen, Y. and Sanghavi, S. Learning with bad training data via iterative trimmed loss minimization. In ICML, 2019.
  40. Shimodaira, H. Improving predictive inference under covariate shift by weighting the log-likelihood function. Journal of statistical planning and inference, 90(2):227–244, 2000.
  41. Snow, R., O'Connor, B., Jurafsky, D., and Ng, A. Cheap and fast – but is it good? evaluating non-expert annotations for natural language tasks. In EMNLP, 2008.
  42. Stempfel, G. and Ralaivola, L. Learning SVMs from sloppily labeled data. In International Conference on Artificial Neural Networks, pp. 884–893, 2009.
  43. Sugiyama, M. and Kawanabe, M. Machine Learning in Non-Stationary Environments: Introduction to Covariate Shift Adaptation. MIT Press, Cambridge, Massachusetts, USA, 2012.
  44. Sugiyama, M., Krauledat, M., and Müller, K.-R. Covariate shift adaptation by importance weighted cross validation. Journal of Machine Learning Research, 8(May):985–1005, 2007.
  45. Sukhbaatar, S., Bruna, J., Paluri, M., Bourdev, L., and Fergus, R. Training convolutional networks with noisy labels. ICLR workshop, 2015.
  46. Tanaka, D., Ikami, D., Yamasaki, T., and Aizawa, K. Joint optimization framework for learning with noisy labels. In CVPR, 2018.
  47. Tarvainen, A. and Valpola, H. Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results. In NeurIPS, 2017.
  48. Tong Xiao, Tian Xia, Yi Yang, Chang Huang, and Xiaogang Wang. Learning from massive noisy labeled data for image classification. In CVPR, 2015.
  49. van Rooyen, B. and Williamson, R. C. Learning in the Presence of Corruption. arXiv e-prints, art. arXiv:1504.00091, Mar 2015. · arXiv
  50. Yan, Y., Rosales, R., Fung, G., Schmidt, M., Hermosillo, G., Bogoni, L., Moy, L., and Dy, J. Modeling annotator expertise: Learning when everybody knows a bit of something. In AISTATS, 2010.
  51. Zhang, C., Bengio, S., Hardt, M., Recht, B., and Vinyals, O. Understanding deep learning requires rethinking generalization. In ICLR, 2017.
  52. Zhang, H., Cisse, M., Dauphin, Y. N., and Lopez-Paz, D. mixup: Beyond empirical risk minimization. ICLR, 2018.
  53. Zhang, T. et al. Statistical behavior and consistency of classification methods based on convex risk minimization. The Annals of Statistics, 32(1):56–85, 2004.
  54. Zhang, Z. and Sabuncu, M. Generalized Cross Entropy Loss for Training Deep Neural Networks with Noisy Labels. In NeurIPS. 2018.

Cited by

Sorted by citations

Authors
Berthon, A., Han, B., Niu, G., Liu, T., & Sugiyama, M.
Affiliations
  • RIKEN Center for Advanced Intelligence Project (JP)
  • Hong Kong Baptist University (HK)
  • The University of Sydney (AU)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2020
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2001.03772
arXiv
2001.03772
OpenAlex
W2998375045
Cited by
57
References
0
Citations by year
2025: 7 · 2024: 18 · 2023: 10 · 2022: 10 · 2021: 7 · 2020: 5
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Funders
  • National Natural Science Foundation of China (62006202)
  • Young Scientists Fund
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Data Classification
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Algorithms
  • Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
Keywords
label noise, confidence scores, label noise correction, semi-supervised learning

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review