CoChat
Sign Up
PreprintCited by 8Machine Learning and Data ClassificationText and Document Classification TechnologiesData Quality and Management

The DOI matches its DataCite record.

AQuA: A Benchmarking Tool for Label Quality Assessment

Open access
Goswami, Mononito, Sanil, Vedant, Choudhry, Arjun, Srinivasan, Arvind, Udompanyawit, Chalisa, Dubrawski, Artur·2023·arXiv
DOI 10.48550/arxiv.2306.09467arXiv 2306.09467
Open this paper in CoChat — free accountOpen PDF

Abstract

Machine learning (ML) models are only as good as the data they are trained on. But recent studies have found datasets widely used to train and evaluate ML models, e.g. ImageNet, to have pervasive labeling errors. Erroneous labels on the train set hurt ML models' ability to generalize, and they impact evaluation and model selection using the test set. Consequently, learning in the presence of labeling errors is an active area of research, yet this field lacks a comprehensive benchmark to evaluate these methods. Most of these methods are evaluated on a few computer vision datasets with significant variance in the experimental protocols. With such a large pool of methods and inconsistent evaluation, it is also unclear how ML practitioners can choose the right models to assess label quality in their data. To this end, we propose a benchmarking environment AQuA to rigorously evaluate methods that enable machine learning in the presence of label noise. We also introduce a design space to delineate concrete design choices of label error detection models. We hope that our proposed design space and benchmark enable practitioners to choose the right tools to improve their label quality and that our benchmark enables objective and rigorous evaluation of machine learning tools facing mislabeled data.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2306.09467

Cites

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

No references could be resolved yet.

Research works printed but not linked: 80

  1. Benoît Frénay and Michel Verleysen. Classification in the presence of label noise: a survey. IEEE transactions on neural networks and learning systems, 25(5):845-869, 2013.
  2. Xingquan Zhu and Xindong Wu. Class noise vs. attribute noise: A quantitative study. The Artificial Intelligence Review, 22(3):177, 2004.
  3. Curtis G. Northcutt, Anish Athalye, and Jonas Mueller. Pervasive label errors in test sets destabilize machine learning benchmarks. In Proceedings of the 35th Conference on Neural Information Processing Systems Track on Datasets and Benchmarks, December 2021.
  4. Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJCV), 115(3):211-252, 2015. doi: 10.1007/s11263-015-0816-y. · DOI · Paper page
  5. Zhengqi Gao, Fan-Keng Sun, Mingran Yang, Sucheng Ren, Zikai Xiong, Marc Engeler, Antonio Burazer, Linda Wildling, Luca Daniel, and Duane S Boning. Learning from multiple annotator noisy labels via sample-wise label fusion. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXIV, pages 407-422. Springer, 2022. ISBN 978-3-031-20053-3.
  6. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
  7. Andrew G. Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. CoRR, abs/1704.04861, 2017. URL http://arxiv. org/abs/1704.04861. · arXiv
  8. Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. Aggregated residual transformations for deep neural networks. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 5987-5995, 2017. doi: 10.1109/CVPR.2017.634. · DOI · Paper page
  9. Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. Un- supervised cross-lingual representation learning at scale. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8440-8451, Online, July 2020. Association for Computational Linguistics. doi: 10.18653/v1/2020.acl-main.747. URL https://aclanthology.org/2020.acl-main.747. · DOI · Paper page
  10. Man-Ching Yuen, Irwin King, and Kwong-Sak Leung. A survey of crowdsourcing systems. In 2011 IEEE third international conference on privacy, security, risk and trust and 2011 IEEE third international conference on social computing, pages 766-773. IEEE, 2011.
  11. Jieyu Zhang, Cheng-Yu Hsieh, Yue Yu, Chao Zhang, and Alexander Ratner. A survey on programmatic weak supervision. arXiv preprint arXiv:2202.05433, 2022. · arXiv
  12. Mononito Goswami, Benedikt Boecking, and Artur Dubrawski. Weak supervision for affordable modeling of electrocardiogram data. In AMIA Annual Symposium Proceedings, volume 2021, page 536. American Medical Informatics Association, 2021.
  13. Joshua C Peterson, Ruairidh M Battleday, Thomas L Griffiths, and Olga Russakovsky. Hu- man uncertainty makes classification more robust. In Proceedings of the IEEE/CVF In- ternational Conference on Computer Vision, pages 9617-9626, Los Alamitos, CA, USA, nov 2019. IEEE Computer Society. doi: 10.1109/ICCV.2019.00971. URL https://doi. ieeecomputersociety.org/10.1109/ICCV.2019.00971. · DOI · Paper page
  14. Lucas Beyer, Olivier J Hénaff, Alexander Kolesnikov, Xiaohua Zhai, and Aäron van den Oord. Are we done with imagenet? arXiv preprint arXiv:2006.07159, 2020. · arXiv
  15. Devansh Arpit, Stanisław Jastrzębski, Nicolas Ballas, David Krueger, Emmanuel Bengio, Maxinder S Kanwal, Tegan Maharaj, Asja Fischer, Aaron Courville, Yoshua Bengio, et al. A closer look at memorization in deep networks. In International conference on machine learning, pages 233-242. PMLR, 2017.
  16. Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning (still) requires rethinking generalization. Communications of the ACM, 64(3): 107-115, 2021.
  17. Jialu Wang, Yang Liu, and Caleb Levy. Fair classification with group-dependent label noise. In Proceedings of the 2021 ACM conference on fairness, accountability, and transparency, pages 526-536, 2021.
  18. Songhua Wu, Mingming Gong, Bo Han, Yang Liu, and Tongliang Liu. Fair classification with instance-dependent label noise. In Conference on Causal Learning and Reasoning, pages 927-943. PMLR, 2022.
  19. Curtis Northcutt, Lu Jiang, and Isaac Chuang. Confident learning: Estimating uncertainty in dataset labels. Journal of Artificial Intelligence Research, 70:1373-1411, 2021.
  20. Geoff Pleiss, Tianyi Zhang, Ethan Elenberg, and Kilian Q. Weinberger. Identifying mislabeled data using the area under the margin ranking. In Proceedings of the 34th International Confer- ence on Neural Information Processing Systems, NIPS'20, Red Hook, NY, USA, 2020.
  21. Curtis G. Northcutt, Tailin Wu, and Isaac L. Chuang. Learning with confident examples: Rank pruning for robust classification with noisy labels. In Proceedings of the Thirty-Third Conference on Uncertainty in Artificial Intelligence, UAI'17. AUAI Press, 2017. URL http: //auai.org/uai2017/proceedings/papers/35.pdf.
  22. Scott E. Reed, Honglak Lee, Dragomir Anguelov, Christian Szegedy, Dumitru Erhan, and An- drew Rabinovich. Training deep neural networks on noisy labels with bootstrapping. In Yoshua Bengio and Yann LeCun, editors, 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Workshop Track Proceedings, 2015. URL http://arxiv.org/abs/1412.6596. · arXiv
  23. Giorgio Patrini, Alessandro Rozza, Aditya Krishna Menon, Richard Nock, and Lizhen Qu. Making deep neural networks robust to label noise: A loss correction approach. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1944-1952, 2017. doi: 10.1109/CVPR.2017.240. · DOI
  24. Xiaobo Xia, Tongliang Liu, Nannan Wang, Bo Han, Chen Gong, Gang Niu, and Masashi Sugiyama. Are anchor points really indispensable in label-noise learning? Advances in neural information processing systems, 32, 2019.
  25. Melanie Bernhardt, Daniel Coelho de Castro, Ryutaro Tanno, Anton Schwaighofer, Kerem Tezcan, Miguel Monteiro, Shruthi Bannur, Matthew Lungren, Aditya Nori, Ben Glocker, Javier Alvarez-Valle, and Ozan Oktay. Active label cleaning for improved dataset quality under resource constraints. Nature Communications, 13, 03 2022. doi: 10.1038/s41467-022-28818-3. · DOI
  26. Stefano Teso, Andrea Bontempelli, Fausto Giunchiglia, and Andrea Passerini. Interactive label cleaning with example-based explanations. In Neural Information Processing Systems, 2021.
  27. T. Liu and D. Tao. Classification with noisy labels by importance reweighting. IEEE Transac- tions on Pattern Analysis &; Machine Intelligence, 38(03):447-461, mar 2016. ISSN 1939-3539. doi: 10.1109/TPAMI.2015.2456899. · DOI
  28. Clayton Scott. A Rate of Convergence for Mixture Proportion Estimation, with Application to Learning from Noisy Labels. In Guy Lebanon and S. V. N. Vishwanathan, editors, Proceedings of the Eighteenth International Conference on Artificial Intelligence and Statistics, volume 38 of Proceedings of Machine Learning Research, pages 838-846, San Diego, California, USA, 09-12 May 2015. PMLR. URL https://proceedings.mlr.press/v38/scott15.html.
  29. Zhaowei Zhu, Yiwen Song, and Yang Liu. Clusterability as an alternative to anchor points when learning with noisy labels. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 12912-12923. PMLR, 18-24 Jul 2021. URL https: //proceedings.mlr.press/v139/zhu21e.html.
  30. Zhaowei Zhu, Zihao Dong, and Yang Liu. Detecting corrupted labels without training a model to predict. In International Conference on Machine Learning, pages 27412-27427. PMLR, 2022.
  31. Lu Jiang, Zhengyuan Zhou, Thomas Leung, Li-Jia Li, and Li Fei-Fei. MentorNet: Learning data-driven curriculum for very deep neural networks on corrupted labels. In Jennifer Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 2304-2313. PMLR, 10-15 Jul 2018. URL https://proceedings.mlr.press/v80/jiang18c.html.
  32. Bo Han, Quanming Yao, Xingrui Yu, Gang Niu, Miao Xu, Weihua Hu, Ivor W. Tsang, and Masashi Sugiyama. Co-teaching: Robust training of deep neural networks with extremely noisy labels. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, NIPS'18, page 8536-8546, Red Hook, NY, USA, 2018.
  33. Junnan Li, Yongkang Wong, Qi Zhao, and Mohan S. Kankanhalli. Learning to learn from noisy labeled data. In 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 5046-5054, 2019. doi: 10.1109/CVPR.2019.00519. · DOI
  34. Guoqing Zheng, Ahmed Hassan Awadallah, and Susan Dumais. Meta label correction for noisy label learning. Proceedings of the AAAI Conference on Artificial Intelligence, 35(12): 11053-11061, May 2021. doi: 10.1609/aaai.v35i12.17319. URL https://ojs.aaai.org/ index.php/AAAI/article/view/17319. · DOI
  35. Derek Chong, Jenny Hong, and Christopher Manning. Detecting label errors by using pre- trained language models. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 9074-9091, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics. URL https://aclanthology.org/2022. emnlp-main.618.
  36. Arash Vahdat. Toward robustness against label noise in training deep discriminative neural networks. In Proceedings of the 31st International Conference on Neural Information Processing Systems, NIPS'17, page 5601-5610, Red Hook, NY, USA, 2017.
  37. Xiaobo Xia, Tongliang Liu, Bo Han, Nannan Wang, Mingming Gong, Haifeng Liu, Gang Niu, Dacheng Tao, and Masashi Sugiyama. Part-dependent label noise: Towards instance-dependent label noise. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 7597-7610.
  38. Curran Associates, Inc., 2020. URL https://proceedings.neurips.cc/paper_files/paper/ 2020/file/5607fe8879e4fd269e88387e8cb30b7e-Paper.pdf.
  39. Hao Cheng, Zhaowei Zhu, Xingyu Li, Yifei Gong, Xing Sun, and Yang Liu. Learning with instance-dependent label noise: A sample sieve approach. In International Conference on Learning Representations, 2021.
  40. Golara Javadi, Samareh Samadi, Sharareh Bayat, Samira Sojoudi, Antonio Hurtado, Silvia Chang, Peter Black, Parvin Mousavi, and Purang Abolmaesumi. Characterizing the uncertainty of label noise in systematic ultrasound-guided prostate biopsy. In 2021 IEEE 18th International Symposium on Biomedical Imaging (ISBI), pages 424-428, 2021. doi: 10.1109/ISBI48211. 2021.9433765. · DOI
  41. Xingrui Yu, Bo Han, Jiangchao Yao, Gang Niu, Ivor Tsang, and Masashi Sugiyama. How does disagreement help generalization against label corruption? In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pages 7164-7173. PMLR, 09-15 Jun 2019. URL https://proceedings.mlr.press/v97/yu19b.html.
  42. Dara Bahri, Heinrich Jiang, and Maya Gupta. Deep k-NN for noisy labels. In Hal Daumé III and Aarti Singh, editors, Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 540-550. PMLR, 13-18 Jul 2020. URL https://proceedings.mlr.press/v119/bahri20a.html.
  43. Michael Desmond, Catherine Finegan-Dollak, Jeff Boston, and Matt Arnold. Label noise in context. In Proceedings of the 58th Annual Meeting of the Association for Computa- tional Linguistics: System Demonstrations, pages 157-186, Online, July 2020. Associa- tion for Computational Linguistics. doi: 10.18653/v1/2020.acl-demos.21. URL https: //aclanthology.org/2020.acl-demos.21. · DOI
  44. Aritra Ghosh, Himanshu Kumar, and P. S. Sastry. Robust loss functions under label noise for deep neural networks. Proceedings of the AAAI Conference on Artificial Intelligence, 31 (1), Feb. 2017. doi: 10.1609/aaai.v31i1.10894. URL https://ojs.aaai.org/index.php/ AAAI/article/view/10894. · DOI
  45. Sunil Thulasidasan, Tanmoy Bhattacharya, Jeff Bilmes, Gopinath Chennupati, and Jamal Mohd- Yusof. Combating label noise in deep learning using abstention. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pages 6234-6243. PMLR, 09-15 Jun 2019. URL https://proceedings.mlr.press/v97/thulasidasan19a.html.
  46. Jiaheng Wei, Zhaowei Zhu, Hao Cheng, Tongliang Liu, Gang Niu, and Yang Liu. Learning with noisy labels revisited: A study using real-world human annotations. In International Conference on Learning Representations, 2022. URL https://openreview.net/forum? id=TBWA6PLJZQm.
  47. Johnson Kuan and Jonas Mueller. Model-agnostic label quality scoring to detect real-world label errors. In ICML DataPerf Workshop, 2022.
  48. Tong Xiao, Tian Xia, Yi Yang, Chang Huang, and Xiaogang Wang. Learning from massive noisy labeled data for image classification. In 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 2691-2699, 2015. doi: 10.1109/CVPR.2015.7298885. · DOI
  49. George Shih, Carol C. Wu, Safwan S. Halabi, Marc D. Kohli, Luciano M. Prevedello, Tessa S. Cook, Arjun Sharma, Judith K. Amorosa, Veronica Arteaga, Maya Galperin-Aizenberg, Ritu R. Gill, Myrna C.B. Godoy, Stephen Hobbs, Jean Jeudy, Archana Laroia, Palmi N. Shah, Dharshan Vummidi, Kavitha Yaddanapudi, and Anouk Stein. Augmenting the national institutes of health chest radiograph dataset with expert annotations of possible pneumonia. Radiology: Artificial Intelligence, 1(1):e180041, 2019. doi: 10.1148/ryai.2019180041. URL https: //doi.org/10.1148/ryai.2019180041. PMID: 33937785. · DOI
  50. Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts. Learning word vectors for sentiment analysis. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 142- 150, Portland, Oregon, USA, June 2011. Association for Computational Linguistics. URL http://www.aclweb.org/anthology/P11-1015.
  51. Francesco Barbieri, Jose Camacho-Collados, Luis Espinosa Anke, and Leonardo Neves. Tweet- Eval: Unified benchmark and comparative evaluation for tweet classification. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 1644-1650, Online, November 2020. Association for Computational Linguistics. doi: 10.18653/v1/2020.findings-emnlp.148. URL https://aclanthology.org/2020.findings-emnlp.148. · DOI
  52. Andrea Dal Pozzolo, Olivier Caelen, Reid A. Johnson, and Gianluca Bontempi. Calibrating probability with undersampling for unbalanced classification. In 2015 IEEE Symposium Series on Computational Intelligence, pages 159-166, 2015. doi: 10.1109/SSCI.2015.33. · DOI · Paper page
  53. Zahra Salekshahrezaee, Joffrey Leevy, and Taghi Khoshgoftaar. A reconstruction error- based framework for label noise detection. Journal of Big Data, 8, 04 2021. doi: 10.1186/s40537-021-00447-5. · DOI
  54. Murat Koklu and Ilker Ali Ozkan. Multiclass classification of dry beans using computer vision and machine learning techniques. Computers and Electronics in Agriculture, 174: 105507, 2020. ISSN 0168-1699. doi: https://doi.org/10.1016/j.compag.2020.105507. URL https://www.sciencedirect.com/science/article/pii/S0168169919311573. · DOI
  55. Marko Bohanec and Vladislav Rajkovi. Knowledge acquisition and explanation for multi- attribute decision making. In 8th Intl Workshop on Expert Systems and their Applications, 1988.
  56. Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H. Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Daniel Guo, Mohammad Gheshlaghi Azar, Bilal Piot, Koray Kavukcuoglu, Rémi Munos, and Michal Valko. Bootstrap your own latent a new approach to self-supervised learning. In Proceedings of the 34th International Conference on Neural Information Processing Systems, NIPS'20, Red Hook, NY, USA, 2020.
  57. J.S. Schlimmer. Concept Acquisition Through Representational Adjustment (Technical Report 87-19). PhD thesis, Department of Information and Computer Science, University of California, Irvine, 1987.
  58. Chang Wei Tan, Geoffrey I. Webb, and Francois Petitjean. Indexing and classifying gigabytes of time series under time warping. In SIAM International Conference on Data Mining, pages 1-10, 2017.
  59. Jason Lines, Anthony Bagnall, Patrick Caiger-Smith, and Simon Anderson. Classification of household devices by electricity usage profiles. In Hujun Yin, Wenjia Wang, and Victor Rayward-Smith, editors, Intelligent Data Engineering and Automated Learning -IDEAL 2011, pages 403-412, Berlin, Heidelberg, 2011. Springer Berlin Heidelberg. ISBN 978-3-642-23878- 9.
  60. George B. Moody and Roger G. Mark. The impact of the MIT-BIH arrhythmia database. IEEE Engineering in Medicine and Biology Magazine, 20:45-50, 2001.
  61. F. Alimoglu and E. Alpaydin. Combining multiple representations and classifiers for pen- based handwritten digit recognition. In Proceedings of the Fourth International Conference on Document Analysis and Recognition, volume 2, pages 637-640 vol.2, 1997. doi: 10.1109/ ICDAR.1997.620583. · DOI
  62. Liu Zhuang, Lin Wayne, Shi Ya, and Zhao Jun. A robustly optimized BERT pre-training approach with post-training. In Proceedings of the 20th Chinese National Conference on Com- putational Linguistics, pages 1218-1227, Huhhot, China, August 2021. Chinese Information Processing Society of China. URL https://aclanthology.org/2021.ccl-1.108.
  63. Wenhui Wang, Furu Wei, Li Dong, Hangbo Bao, Nan Yang, and Ming Zhou. Minilm: Deep self- attention distillation for task-agnostic compression of pre-trained transformers. In Proceedings of the 34th International Conference on Neural Information Processing Systems, NIPS'20, Red Hook, NY, USA, 2020.
  64. Yuqi Nie, Nam H. Nguyen, Phanwadee Sinthong, and Jayant Kalagnanam. A time series is worth 64 words: Long-term forecasting with transformers. In International Conference on Learning Representations, 2023.
  65. Fazle Karim, Somshubra Majumdar, Houshang Darabi, and Shun Chen. LSTM fully con- volutional networks for time series classification. IEEE Access, 6:1662-1669, 2018. doi: 10.1109/ACCESS.2017.2779939. · DOI
  66. Görkem Algan and Ilkay Ulusoy. Label noise types and their effects on deep learning. arXiv preprint arXiv:2003.10471, 2020. · arXiv
  67. Lu Jiang, Di Huang, Mason Liu, and Weilong Yang. Beyond synthetic noise: Deep learning on controlled noisy labels. In International conference on machine learning, pages 4804-4815. PMLR, 2020.
  68. Yiding Jiang, Behnam Neyshabur, Hossein Mobahi, Dilip Krishnan, and Samy Bengio. Fantastic generalization measures and where to find them. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020. URL https://openreview.net/forum?id=SJgIPJBFvH.
  69. Nicholas Gisolfi. Model-Centric Verification of Artificial Intelligence. PhD thesis, Carnegie Mellon University, 2021.
  70. Janez Demšar. Statistical comparisons of classifiers over multiple data sets. The Journal of Machine learning research, 7:1-30, 2006.
  71. Alessio Benavoli, Giorgio Corani, and Francesca Mangili. Should we really use post-hoc tests based on mean-ranks? The Journal of Machine Learning Research, 17(1):152-161, 2016.
  72. Anthony Bagnall, Jason Lines, Aaron Bostrom, James Large, and Eamonn Keogh. The great time series classification bake off: a review and experimental evaluation of recent algorithmic advances. Data mining and knowledge discovery, 31:606-660, 2017.
  73. Michael A. Hedderich, D. Zhu, and Dietrich Klakow. Analysing the noise model error for realistic noisy label data. In AAAI Conference on Artificial Intelligence, 2021.
  74. Dawei Zhu, Michael A. Hedderich, Fangzhou Zhai, David Adelani, and Dietrich Klakow. Is BERT robust to label noise? a study on learning with noisy labels in text classification. In Proceedings of the Third Workshop on Insights from Negative Results in NLP, pages 62-67, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10.18653/v1/2022. insights-1.8. URL https://aclanthology.org/2022.insights-1.8. · DOI
  75. Y. Wang, X. Ma, Z. Chen, Y. Luo, J. Yi, and J. Bailey. Symmetric cross entropy for robust learning with noisy labels. In 2019 IEEE/CVF International Conference on Computer Vision (ICCV), pages 322-330, Los Alamitos, CA, USA, nov 2019. IEEE Computer Society. doi: 10. 1109/ICCV.2019.00041. URL https://doi.ieeecomputersociety.org/10.1109/ICCV. 2019.00041. · DOI · Paper page
  76. Varun Gulshan, Lily Peng, Marc Coram, Martin C. Stumpe, Derek Wu, Arunachalam Narayanaswamy, Subhashini Venugopalan, Kasumi Widner, Tom Madams, Jorge Cuadros, Ramasamy Kim, Rajiv Raman, Philip C. Nelson, Jessica L. Mega, and Dale R. Webster. Devel- opment and Validation of a Deep Learning Algorithm for Detection of Diabetic Retinopathy in Retinal Fundus Photographs. JAMA, 316(22):2402-2410, 12 2016. ISSN 0098-7484. doi: 10.1001/jama.2016.17216. URL https://doi.org/10.1001/jama.2016.17216. · DOI · Paper page
  77. Hassan Ismail Fawaz, Germain Forestier, Jonathan Weber, Lhassane Idoumghar, and Pierre- Alain Muller. Deep learning for time series classification: a review. Data mining and knowledge discovery, 33(4):917-963, 2019.
  78. Milton Friedman. A comparison of alternative tests of significance for the problem of m rankings. The Annals of Mathematical Statistics, 11(1):86-92, 1940. ISSN 00034851. URL http://www.jstor.org/stable/2235971.
  79. Frank Wilcoxon. Individual comparisons by ranking methods. Biometrics Bulletin, 1(6):80-83, 1945. ISSN 00994987. URL http://www.jstor.org/stable/3001968.
  80. Sture Holm. A simple sequentially rejective multiple test procedure. Scandinavian Journal of Statistics, 6(2):65-70, 1979. ISSN 03036898, 14679469. URL http://www.jstor.org/ stable/4615733.

Cited by

Sorted by citations

Metadata

Authors
Goswami, M., Sanil, V., Choudhry, A., Srinivasan, A., Udompanyawit, C., & Dubrawski, A.
ORCID
Venue
arXiv
Publisher
arXiv
Published
2023
Pages
79792-79807
ISSN
1049-5258 (ISSN-L)
DOI
10.48550/arxiv.2306.09467
arXiv
2306.09467
OpenAlex
W7133187838
Cited by
8
References
0
Citations by year
2026: 2 · 2025: 4 · 2024: 2
FWCI
1.42
Type
conference paper
Language
en
Open access
Licence
Locations
  • neural information processing systems (publishedVersion)
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • arXiv (Cornell University) (cc-by)
Funders
  • National Institutes of Health
  • National Institute of Neurological Disorders and Stroke (R01NS124642)
  • Army Research Office (W911NF, W911NF20D0002)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Data Classification
  • Physical Sciences › Computer Science › Artificial Intelligence › Text and Document Classification Technologies
  • Physical Sciences › Computer Science › Information Systems › Data Quality and Management
Keywords
benchmarking tool, annotation quality, data annotation, Aqua

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review