CoChat
Sign Up
Cited by 34Machine Learning and Data ClassificationAdversarial Robustness in Machine LearningText and Document Classification Technologies

Identifying Mislabeled Instances in Classification Datasets

Open access
Muller, Nicolas M., Markert, Karla·2019·2019 International Joint Conference on Neural Networks (IJCNN)
DOI 10.1109/ijcnn.2019.8851920
Open this paper in CoChat — free accountOpen PDF

Abstract

A key requirement for supervised machine learning is labeled training data, which is created by annotating unlabeled data with the appropriate class. Because this process can in many cases not be done by machines, labeling needs to be performed by human domain experts. This process tends to be expensive both in time and money, and is prone to errors. Additionally, reviewing an entire labeled dataset manually is often prohibitively costly, so many real world datasets contain mislabeled instances.To address this issue, we present in this paper a non-parametric end-to-end pipeline to find mislabeled instances in numerical, image and natural language datasets. We evaluate our system quantitatively by adding a small number of label noise to 29 datasets, and show that we find mislabeled instances with an average precision of more than 0.84 when reviewing our system's top 1% recommendation. We then apply our system to publicly available datasets and find mislabeled instances in CIFAR-100, Fashion-MNIST, and others. Finally, we publish the code and an applicable implementation of our approach.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1109/ijcnn.2019.8851920

Cites

1 of 1 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Even in the most common non-medical ML datasets there is a significant percentage of wrong labels.

    “Even in the most common (non-medical) datasets of ML (e.g., MNIST208, CIFAR-100209, Fashion-MNIST210) there is a significant percentage of wrong labels211,212.”

    From The METRIC-framework for assessing data quality for trustworthy AI in medicine: A systematic review (2024) · cites this paper as [211]

    The Introduction (p. 1) says the authors identified mislabeled instances in CIFAR-100, MNIST, and Fashion-MNIST, and the Qualitative evaluation (p. 5) gives examples from CIFAR-100 and Fashion-MNIST. The paper does not establish that a significant percentage of labels in those datasets is wrong: its 3% rate was deliberately introduced for testing, not measured in the original datasets (Quantitative and Qualitative evaluation, pp. 4–5). (LLM, conf=0.88)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 19

  1. [1][1] Carla E Brodley and Mark A Friedl. Identifying mislabeled training data. Journal of artificial intelligence research , 11:131-167, 1999. -
  2. [2][2] Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms. arXiv preprint arXiv:1708.07747 , 2017. - · arXiv
  3. [4][4] Rajmadhan Ekambaram, Dmitry B Goldgof, and Lawrence O Hall. Finding label noise examples in large scale datasets. In Systems, Man, and Cybernetics (SMC), 2017 IEEE International Conference on , pages 2420-2424. IEEE, 2017. -
  4. [5][5] Mohammed Al-Rawi and Dimosthenis Karatzas. On the labeling correctness in computer vision datasets. In Proceedings of the Workshop on Interactive Adaptive Learning , pages 1-23, 2018. -
  5. [7][7] Benoˆ ıt Fr´ enay and Michel Verleysen. Classification in the presence of label noise: a survey. IEEE transactions on neural networks and learning systems , 25(5):845-869, 2014. -
  6. [8][8] Yuncheng Li, Jianchao Yang, Yale Song, Liangliang Cao, Jiebo Luo, and Li-Jia Li. Learning from noisy labels with distillation. In ICCV , pages 1928-1936, 2017. -
  7. [10][10] Dan Hendrycks, Mantas Mazeika, Duncan Wilson, and Kevin Gimpel. Using trusted data to train deep networks on labels corrupted by severe noise. arXiv preprint arXiv:1802.05300 , 2018. - · arXiv
  8. [12][12] Andrea Esuli and Fabrizio Sebastiani. Improving text classification accuracy by training label cleaning. ACM Transactions on Information Systems (TOIS) , 31(4):19, 2013. -
  9. [13][13] Aleksander Kolcz and Gordon V Cormack. Genre-based decomposition of email class noise. In Proceedings of the 15th ACM SIGKDD international conference on Knowledge discovery and data mining , pages 427-436. ACM, 2009. -
  10. [14][14] Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Corrado, and Jeffrey Dean. Distributed representations of words and phrases and their compositionality. CoRR , abs/1310.4546, 2013. - · arXiv
  11. [17][17] Jure Leskovec, Anand Rajaraman, and Jeffrey David Ullman. Mining of massive datasets , pages 1-19. Cambridge university press, 2014. -
  12. [18][18] Jason Yosinski, Jeff Clune, Yoshua Bengio, and Hod Lipson. How transferable are features in deep neural networks? In Advances in neural information processing systems , pages 3320-3328, 2014. -
  13. [19][19] F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research , 12:2825-2830, 2011. -
  14. [21][21] I-Cheng Yeh and Che-hui Lien. The comparisons of data mining techniques for the predictive accuracy of probability of default of credit card clients. Expert Systems with Applications , 36(2):2473-2480, 2009. -
  15. [22][22] Tiago A Almeida, Jos´ e Mar´ ıa G Hidalgo, and Akebo Yamakami. Contributions to the study of sms spam filtering: new collection and results. In Proceedings of the 11th ACM symposium on Document engineering , pages 259-262. ACM, 2011. -
  16. [24][24] Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts. Learning word vectors for sentiment analysis. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies , pages 142-150, Portland, Oregon, USA, June 2011. Association for Computational Linguistics. -
  17. [25][25] Li Deng. The mnist database of handwritten digit images for machine learning research [best of the web]. IEEE Signal Processing Magazine , 29(6):141-142, 2012. -
  18. [26][26] Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng. Reading digits in natural images with unsupervised feature learning. In NIPS workshop on deep learning and unsupervised feature learning , volume 2/2011, page 5, 2011. -
  19. [29][29] Ekin D Cubuk, Barret Zoph, Dandelion Mane, Vijay Vasudevan, and Quoc V Le. Autoaugment: Learning augmentation policies from data. arXiv preprint arXiv:1805.09501 , 2018. - · arXiv

Cited by

Sorted by citations

Authors
Muller, N. M., & Markert, K.
Affiliations
Fraunhofer Institute for Applied and Integrated Security (DE)
Venue
2019 International Joint Conference on Neural Networks (IJCNN)
Publisher
IEEE
Published
2019-07
Pages
1-8
ISSN
2161-4407 (ISSN-L)
DOI
10.1109/ijcnn.2019.8851920
OpenAlex
W2978012698
Cited by
34
References
1
Citations by year
2026: 1 · 2025: 7 · 2024: 7 · 2023: 5 · 2022: 4 · 2021: 8 · 2020: 2
FWCI
1.34
Type
conference paper
Language
en
Open access
Licence
Locations
  • International Joint Conference on Neural Networks (IJCNN) (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • Publikationsdatenbank der Fraunhofer-Gesellschaft (Fraunhofer-Gesellschaft) (submittedVersion)
  • Fraunhofer-Publica (Fraunhofer-Gesellschaft) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Data Classification
  • Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
  • Physical Sciences › Computer Science › Artificial Intelligence › Text and Document Classification Technologies
Keywords
label noise, training data quality, image classification dataset

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review