CoChat
Sign Up
Cited by 573Data Quality and ManagementMachine Learning and Data ClassificationEthics and Social Impacts of AI

Data collection and quality challenges in deep learning: a data-centric AI perspective

Open access
Whang, Steven Euijong, Roh, Yuji, Song, Hwanjun, Lee, Jae-Gil·2023·The VLDB Journal
DOI 10.1007/s00778-022-00775-9
Open this paper in CoChat — free accountOpen PDF

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1007/s00778-022-00775-9

Cites

10 of 10 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Another dominant topic in the DL papers is completeness, which the ML community almost exclusively refers to as missing data.

    “Another dominant topic is completeness to which the ML community almost exclusively refers to as missing data119,125–128,133–135,182.”

    From The METRIC-framework for assessing data quality for trustworthy AI in medicine: A systematic review (2024) · cites this paper as [182]

    Section 4.2 (p. 10) says missing-data imputation is an active research topic in statistics and machine learning, supporting the claim that missing data is a significant topic. The paper does not establish that completeness is a dominant topic in deep-learning papers or that the ML community almost exclusively uses “missing data” to refer to completeness. (LLM, conf=0.78)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 145

  1. [18]18. Alekh Agarwal, Alina Beygelzimer, Miroslav Dud´ ık, John Langford, and Hanna M. Wallach. A reductions approach to fair classification. In ICML , pages 60-69, 2018.
  2. [19]19. Pulkit Agrawal, Rajat Arya, Aanchal Bindal, Sandeep Bhatia, Anupriya Gagneja, Joseph Godlewski, Yucheng Low, Timothy Muss, Mudit Manu Paliwal, Sethu Raman, Vishrut Shah, Bochao Shen, Laura Sugden, Kaiyu Zhao, and Ming-Chuan Wu. Data platform for machine learning. In SIGMOD , pages 1803-1816, 2019.
  3. [20]20. Saleema Amershi, Andrew Begel, Christian Bird, Robert DeLine, Harald C. Gall, Ece Kamar, Nachiappan Nagappan, Besmira Nushi, and Thomas Zimmermann. Software engineering for machine learning: a case study. In ICSE , pages 291-300, 2019.
  4. [22]22. Saeed Anwar and Nick Barnes. Real image denoising with feature attention. In CVPR , pages 3155-3164, 2019.
  5. [23]23. Abolfazl Asudeh, Zhongjun Jin, and H. V. Jagadish. Assessing and remedying coverage for a given dataset. In ICDE , pages 554-565, 2019.
  6. [24]24. Stephen H. Bach, Daniel Rodriguez, Yintao Liu, Chong Luo, Haidong Shao, Cassandra Xia, Souvik Sen, Alexander Ratner, Braden Hancock, Houman Alborzi, Rahul Kuchhal, Christopher R´ e, and Rob Malkin. Snorkel drybell: A case study in deploying weak supervision at industrial scale. In SIGMOD , pages 362-375, 2019.
  7. [25]25. Tadas Baltrusaitis, Chaitanya Ahuja, and LouisPhilippe Morency. Multimodal machine learning: A survey and taxonomy. IEEE Trans. Pattern Anal. Mach. Intell. , 41(2):423-443, 2019.
  8. [27]27. Denis Baylor, Eric Breck, Heng-Tze Cheng, Noah Fiedel, Chuan Yu Foo, Zakaria Haque, Salem Haykal, Mustafa Ispir, Vihan Jain, Levent Koc, Chiu Yuen Koo, Lukasz Lew, Clemens Mewald, Akshay Naresh Modi, Neoklis Polyzotis, Sukriti Ramesh, Sudip Roy, Steven Euijong Whang, Martin Wicke, Jarek Wilkiewicz, Xin Zhang, and Martin Zinkevich. TFX: A tensorflow-based production-scale machine learning platform. In KDD , pages 1387-1395, 2017.
  9. [28]28. Rachel K. E. Bellamy, Kuntal Dey, Michael Hind, et al. AI fairness 360: An extensible toolkit for detecting and mitigating algorithmic bias. IBM Journal of Research and Development , 2019.
  10. [30]30. David Berthelot, Nicholas Carlini, Ekin D. Cubuk, Alex Kurakin, Kihyuk Sohn, Han Zhang, and Colin Raffel. Remixmatch: Semi-supervised learning with distribution matching and augmentation anchoring. In ICLR , 2020.
  11. [31]31. David Berthelot, Nicholas Carlini, Ian J. Goodfellow, Nicolas Papernot, Avital Oliver, and Colin Raffel. Mixmatch: A holistic approach to semi-supervised learning. In NeurIPS , pages 5050-5060, 2019.
  12. [32]32. Felix Biessmann, Jacek Golebiowski, Tammo Rukat, Dustin Lange, and Philipp Schmidt. Automated data validation in machine learning systems. IEEE Data Eng. Bull. , 44(1):51-65, 2021.
  13. [34]34. Avrim Blum and Tom Mitchell. Combining labeled and unlabeled data with co-training. In COLT , pages 92100, New York, NY, USA, 1998. ACM.
  14. [37]37. Dan Brickley, Matthew Burgess, and Natasha F. Noy. Google dataset search: Building a search engine for datasets in an open web ecosystem. In WWW , pages 1365-1375, 2019.
  15. [38]38. Michael J. Cafarella, Alon Y. Halevy, Hongrae Lee, Jayant Madhavan, Cong Yu, Daisy Zhe Wang, and Eugene Wu. Ten years of webtables. PVLDB , 11(12):21402149, 2018.
  16. [39]39. Jos´ e Cambronero, John K. Feser, Micah J. Smith, and Samuel Madden. Query optimization for dynamic imputation. Proc. VLDB Endow. , 10(11):1310-1321, 2017.
  17. [40]40. Anirban Chakraborty, Manaar Alam, Vishal Dey, Anupam Chattopadhyay, and Debdeep Mukhopadhyay. Adversarial attacks and defences: A survey. CoRR , abs/1810.00069, 2018. · arXiv
  18. [41]41. Haw-Shiuan Chang, Erik G. Learned-Miller, and Andrew McCallum. Active bias: Training more accurate neural networks by emphasizing high variance samples. In NeurIPS , pages 1002-1012, 2017.
  19. [42]42. Zhengping Che, Sanjay Purushotham, Kyunghyun Cho, David Sontag, and Yan Liu. Recurrent neural networks for multivariate time series with missing values. Nature Scientific Reports , 8(1):6085, 2018.
  20. [44]44. Irene Y. Chen, Fredrik D. Johansson, and David A. Sontag. Why is my classifier discriminatory? In NeurIPS , pages 3543-3554, 2018.
  21. [46]46. Yu Cheng, Ilias Diakonikolas, and Rong Ge. Highdimensional robust mean estimation in nearly-linear time. In SIAM , pages 2755-2771, 2019.
  22. [47]47. Kristy Choi, Aditya Grover, Trisha Singh, Rui Shu, and Stefano Ermon. Fair generative modeling via weak supervision. In ICML , pages 1887-1898, 2020.
  23. [49]49. Alexandra Chouldechova and Aaron Roth. A snapshot of the frontiers of fairness in machine learning. Commun. ACM , 63(5):82-89, 2020.
  24. [50]50. Evgenii Chzhen, Christophe Denis, Mohamed Hebiri, Luca Oneto, and Massimiliano Pontil. Leveraging labeled and unlabeled data for consistent fair binary classification. In NeurIPS , pages 12739-12750, 2019.
  25. [51]51. Andrew Cotter, Heinrich Jiang, and Karthik Sridharan. Two-player games for efficient non-convex constrained optimization. In ALT , pages 300-332, 2019.
  26. [53]53. Ekin D. Cubuk, Barret Zoph, Dandelion Man´ e, Vijay Vasudevan, and Quoc V. Le. Autoaugment: Learning augmentation strategies from data. In CVPR , pages 113-123, 2019.
  27. [54]54. Ilias Diakonikolas, Gautam Kamath, Daniel Kane, Jerry Li, Ankur Moitra, and Alistair Stewart. Robust estimators in high-dimensions without the computational intractability. SIAM Journal on Computing , 48(2):742-864, 2019.
  28. [56]56. AnHai Doan, Alon Y. Halevy, and Zachary G. Ives. Principles of Data Integration . Morgan Kaufmann, 2012.
  29. [57]57. Mohamad Dolatshah, Mathew Teoh, Jiannan Wang, and Jian Pei. Cleaning crowdsourced labels using oracles for statistical classification. PVLDB , 12(4):376-389, 2018.
  30. [58]58. Xin Luna Dong and Theodoros Rekatsinas. Data integration and machine learning: A natural synergy. In KDD , pages 3193-3194, 2019.
  31. [59]59. Mike Dreves, Gene Huang, Zhuo Peng, Neoklis Polyzotis, Evan Rosen, and Paul Suganthan G. C. Validating data and models in continuous ML pipelines. IEEE Data Eng. Bull. , 44(1):42-50, 2021.
  32. [61]61. Cynthia Dwork, Moritz Hardt, Toniann Pitassi, Omer Reingold, and Richard S. Zemel. Fairness through awareness. In ITCS , pages 214-226, 2012.
  33. [62]62. Michael Feldman, Sorelle A. Friedler, John Moeller, Carlos Scheidegger, and Suresh Venkatasubramanian. Certifying and removing disparate impact. In KDD , pages 259-268, 2015.
  34. [63]63. Raul Castro Fernandez, Ziawasch Abedjan, Famien Koko, Gina Yuan, Samuel Madden, and Michael Stonebraker. Aurum: A data discovery system. In ICDE , pages 1001-1012, 2018.
  35. [64]64. Dean P. Foster and Robert A. Stine. Alpha-investing: a procedure for sequential control of expected false discoveries. Journal of the Royal Statistical Society: Series B (Statistical Methodology) , 70(2):429-444, 2008.
  36. [65]65. Karan Goel, Albert Gu, Yixuan Li, and Christopher R´ e. Model patching: Closing the subgroup performance gap with data augmentation. In ICLR , 2021.
  37. [66]66. Ian J. Goodfellow. NIPS 2016 tutorial: Generative adversarial networks. CoRR , abs/1701.00160, 2017. · arXiv
  38. [67]67. Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron C. Courville, and Yoshua Bengio. Generative adversarial nets. In NIPS , pages 2672-2680, 2014.
  39. [68]68. Ian J. Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. In ICLR , 2015.
  40. [71]71. Alon Y. Halevy, Flip Korn, Natalya Fridman Noy, Christopher Olston, Neoklis Polyzotis, Sudip Roy, and Steven Euijong Whang. Goods: Organizing Google's datasets. In SIGMOD , pages 795-806, 2016.
  41. [72]72. Bo Han, Quanming Yao, Xingrui Yu, Gang Niu, Miao Xu, Weihua Hu, Ivor W. Tsang, and Masashi Sugiyama. Co-teaching: Robust training of deep neural networks with extremely noisy labels. In NeurIPS , pages 85368546, 2018.
  42. [73]73. Moritz Hardt, Eric Price, and Nati Srebro. Equality of opportunity in supervised learning. In NIPS , pages 3315-3323, 2016.
  43. [74]74. Tatsunori B. Hashimoto, Megha Srivastava, Hongseok Namkoong, and Percy Liang. Fairness without demographics in repeated loss minimization. In Jennifer G. Dy and Andreas Krause, editors, ICML , volume 80, pages 1934-1943. PMLR, 2018.
  44. [75]75. Kim M. Hazelwood, Sarah Bird, David M. Brooks, Soumith Chintala, Utku Diril, Dmytro Dzhulgakov, Mohamed Fawzy, Bill Jia, Yangqing Jia, Aditya Kalro, James Law, Kevin Lee, Jason Lu, Pieter Noordhuis, Misha Smelyanskiy, Liang Xiong, and Xiaodong Wang. Applied machine learning at facebook: A datacenter infrastructure perspective. In HPCA , pages 620-629, 2018.
  45. [76]76. Dan Hendrycks, Norman Mu, Ekin Dogus Cubuk, Barret Zoph, Justin Gilmer, and Balaji Lakshminarayanan. Augmix: A simple data processing method to improve robustness and uncertainty. In ICLR , 2020.
  46. [79]79. Victoria J. Hodge and Jim Austin. A survey of outlier detection methodologies. Artif. Intell. Rev. , 22(2):85126, 2004.
  47. [80]80. Peter J Huber. Robust estimation of a location parameter. In Breakthroughs in statistics , pages 492-518. Springer, 1992.
  48. [81]81. Ihab F. Ilyas and Xu Chu. Data Cleaning . ACM, 2019.
  49. [83]83. Vasileios Iosifidis and Eirini Ntoutsi. Adafair: Cumulative fairness adaptive boosting. In CIKM , pages 781790, 2019.
  50. [84]84. Heinrich Jiang and Ofir Nachum. Identifying and correcting label bias in machine learning. In AISTATS , pages 702-712, 2020.
  51. [85]85. Lu Jiang, Zhengyuan Zhou, Thomas Leung, Li-Jia Li, and Li Fei-Fei. Mentornet: Learning data-driven curriculum for very deep neural networks on corrupted labels. In ICML , pages 2309-2318, 2018.
  52. [87]87. Toshihiro Kamishima, Shotaro Akaho, Hideki Asoh, and Jun Sakuma. Fairness-aware classifier with prejudice remover regularizer. In ECML PKDD , pages 35-50, 2012.
  53. [88]88. Bojan Karlas, Peng Li, Renzhi Wu, Nezihe Merve G¨ urel, Xu Chu, Wentao Wu, and Ce Zhang. Nearest neighbor classifiers over incomplete information: From certain answers to certain predictions. Proc. VLDB Endow. , 14(3):255-267, 2020.
  54. [89]89. Aria Khademi, Sanghack Lee, David Foley, and Vasant Honavar. Fairness in algorithmic decision making: An excursion through the lens of causality. In WWW , pages 2907-2914, 2019.
  55. [90]90. Fereshte Khani and Percy Liang. Removing spurious features can hurt accuracy and affect groups disproportionately. In FAccT , pages 196-205. ACM, 2021.
  56. [91]91. Niki Kilbertus, Mateo Rojas-Carulla, Giambattista Parascandolo, Moritz Hardt, Dominik Janzing, and Bernhard Sch¨ olkopf. Avoiding discrimination through causal reasoning. In NeurIPS , pages 656-666, 2017.
  57. [92]92. Hoon Kim, Kangwook Lee, Gyeongjo Hwang, and Changho Suh. Crash to not crash: Learn to identify dangerous vehicles using a simulator. In AAAI , pages 978-985, 2019.
  58. [93]93. Pang Wei Koh, Jacob Steinhardt, and Percy Liang. Stronger data poisoning attacks break data sanitization defenses. CoRR , abs/1811.00741, 2018. · arXiv
  59. [94]94. Sanjay Krishnan, Jiannan Wang, Eugene Wu, Michael J. Franklin, and Ken Goldberg. Activeclean: Interactive data cleaning for statistical modeling. PVLDB , 9(12):948-959, 2016.
  60. [95]95. Karol Kurach, Mario Lucic, Xiaohua Zhai, Marcin Michalski, and Sylvain Gelly. The GAN landscape: Losses, architectures, regularization, and normalization. CoRR , abs/1807.04720, 2018. · arXiv
  61. [96]96. Matt J Kusner, Joshua Loftus, Chris Russell, and Ricardo Silva. Counterfactual fairness. In NeurIPS , pages 4066-4076. 2017.
  62. [97]97. Preethi Lahoti, Alex Beutel, Jilin Chen, Kang Lee, Flavien Prost, Nithum Thain, Xuezhi Wang, and Ed Chi. Fairness without demographics through adversarially reweighted learning. In NeurIPS , 2020.
  63. [98]98. Alexandre Louis Lamy and Ziyuan Zhong. Noisetolerant fair classification. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d'Alch´ e-Buc, Emily B. Fox, and Roman Garnett, editors, NeurIPS , pages 294-305, 2019.
  64. [99]99. Doris Jung Lin Lee and Aditya G. Parameswaran. The case for a visual discovery assistant: A holistic solution for accelerating visual data exploration. IEEE Data Eng. Bull. , 41(3):3-14, 2018.
  65. [100]100. Jae-Gil Lee, Yuji Roh, Hwanjun Song, and Steven Euijong Whang. Machine learning robustness, fairness, and their convergence. In KDD , pages 4046-4047, 2021.
  66. [101]101. Junnan Li, Richard Socher, and Steven C. H. Hoi. Dividemix: Learning with noisy labels as semi-supervised learning. In ICLR , 2020.
  67. [102]102. Peng Li, Xi Rao, Jennifer Blase, Yue Zhang, Xu Chu, and Ce Zhang. CleanML: A benchmark for joint data cleaning and machine learning [experiments and analysis]. In ICDE , 2021.
  68. [103]103. Zifan Liu, Jong Ho Park, Theodoros Rekatsinas, and Christos Tzamos. On robust mean estimation under coordinate-level corruption. In ICML , pages 6914-6924. PMLR, 2021.
  69. [104]104. Zifan Liu, Jongho Park, Theodoros Rekatsinas, and Christos Tzamos. On robust mean estimation under coordinate-level corruption. In ICML , pages 6914-6924, 2021.
  70. [105]105. Eran Malach and Shai Shalev-Shwartz. Decoupling 'when to update' from 'how to update'. In NIPS , pages 960-970, 2017.
  71. [106]106. Ninareh Mehrabi, Fred Morstatter, Nripsuta Saxena, Kristina Lerman, and Aram Galstyan. A survey on bias and fairness in machine learning. CoRR , abs/1908.09635, 2019. · arXiv
  72. [108]108. Dongyu Meng and Hao Chen. Magnet: A two-pronged defense against adversarial examples. In Bhavani M. Thuraisingham, David Evans, Tal Malkin, and Dongyan Xu, editors, ACM SIGSAC , pages 135-147, 2017.
  73. [109]109. Jan Hendrik Metzen, Tim Genewein, Volker Fischer, and Bastian Bischoff. On detecting adversarial perturbations. In ICLR , 2017.
  74. [110]110. Ren´ ee J. Miller, Fatemeh Nargesian, Erkang Zhu, Christina Christodoulakis, Ken Q. Pu, and Periklis Andritsos. Making open data transparent: Data discovery on open data. IEEE Data Eng. Bull. , 41(2):59-70, 2018.
  75. [111]111. Mike Mintz, Steven Bills, Rion Snow, and Daniel Jurafsky. Distant supervision for relation extraction without labeled data. In Keh-Yih Su, Jian Su, and Janyce Wiebe, editors, ACL , pages 1003-1011, 2009.
  76. [112]112. Razieh Nabi and Ilya Shpitser. Fair inference on outcomes. In AAAI , pages 1931-1940, 2018.
  77. [113]113. Felix Neutatz, Binger Chen, Ziawasch Abedjan, and Eugene Wu. From cleaning before ML to cleaning for ML. IEEE Data Eng. Bull. , 44(1):24-41, 2021.
  78. [114]114. Mehdi Noroozi and Paolo Favaro. Unsupervised learning of visual representations by solving jigsaw puzzles. In ECCV , pages 69-84, 2016.
  79. [115]115. Sinno Jialin Pan and Qiang Yang. A survey on transfer learning. IEEE TKDE , 22(10):1345-1359, 2010.
  80. [116]116. Nicolas Papernot, Patrick D. McDaniel, Xi Wu, Somesh Jha, and Ananthram Swami. Distillation as a defense to adversarial perturbations against deep neural networks. In IEEE SP , pages 582-597, 2016.
  81. [117]117. Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d ' Alch´ eBuc, E. Fox, and R. Garnett, editors, NeurIPS , pages 8024-8035. Curran Associates, Inc., 2019.
  82. [118]118. Giorgio Patrini, Alessandro Rozza, Aditya Krishna Menon, Richard Nock, and Lizhen Qu. Making deep neural networks robust to label noise: A loss correction approach. In CVPR , pages 2233-2241, 2017.
  83. [119]119. Andrea Paudice, Luis Mu˜ noz-Gonz´ alez, Andr´ as Gy¨ orgy, and Emil C. Lupu. Detection of adversarial training examples in poisoning attacks through anomaly detection. CoRR , abs/1802.03041, 2018. · arXiv
  84. [120]120. Nikos Pelekis, Christos Ntrigkogias, Panagiotis Tampakis, Stylianos Sideridis, and Yannis Theodoridis. Hermoupolis: A trajectory generator for simulating generalized mobility patterns. In ECML PKDD , pages 659-662, 2013.
  85. [121]121. Geoff Pleiss, Manish Raghavan, Felix Wu, Jon M. Kleinberg, and Kilian Q. Weinberger. On fairness and calibration. In NIPS , pages 5680-5689, 2017.
  86. [122]122. Neoklis Polyzotis, Sudip Roy, Steven Euijong Whang, and Martin Zinkevich. Data management challenges in production machine learning. In SIGMOD , pages 17231726, 2017.
  87. [123]123. Neoklis Polyzotis, Sudip Roy, Steven Euijong Whang, and Martin Zinkevich. Data lifecycle challenges in production machine learning: A survey. SIGMOD Rec. , 47(2):17-28, 2018.
  88. [124]124. Adnan Qayyum, Junaid Qadir, Muhammad Bilal, and Ala Al-Fuqaha. Secure and robust machine learning for healthcare: A survey. IEEE Reviews in Biomedical Engineering , 14:156-180, 2020.
  89. [125]125. Alexander Ratner, Stephen H. Bach, Henry Ehrenberg, Jason Fries, Sen Wu, and Christopher R´ e. Snorkel: Rapid training data creation with weak supervision. PVLDB , 11(3):269-282, November 2017.
  90. [126]126. Alexander Ratner, Stephen H. Bach, Henry R. Ehrenberg, Jason A. Fries, Sen Wu, and Christopher R´ e. Snorkel: rapid training data creation with weak supervision. VLDB J. , 29(2-3):709-730, 2020.
  91. [127]127. Alexander J. Ratner, Henry R. Ehrenberg, Zeshan Hussain, Jared Dunnmon, and Christopher R´ e. Learning to compose domain-specific transformations for data augmentation. In NIPS , pages 3239-3249, 2017.
  92. [128]128. Sergey Redyuk, Zoi Kaoudi, Volker Markl, and Sebastian Schelter. Automating data quality validation for dynamic data ingestion. In EDBT , pages 61-72, 2021.
  93. [129]129. Scott E. Reed, Honglak Lee, Dragomir Anguelov, Christian Szegedy, Dumitru Erhan, and Andrew Rabinovich. Training deep neural networks on noisy labels with bootstrapping. In ICLR , 2015.
  94. [130]130. Theodoros Rekatsinas, Xu Chu, Ihab F. Ilyas, and Christopher R´ e. Holoclean: Holistic data repairs with probabilistic inference. PVLDB , 10(11):1190-1201, 2017.
  95. [131]131. C´ edric Renggli, Luka Rimanic, Nezihe Merve G¨ urel, Bojan Karlas, Wentao Wu, and Ce Zhang. A data qualitydriven view of mlops. IEEE Data Eng. Bull. , 44(1):11-23, 2021.
  96. [132]132. Francesco Ricci, Lior Rokach, and Bracha Shapira, editors. Recommender Systems Handbook . Springer, 2015.
  97. [133]133. Yuji Roh, Geon Heo, and Steven Euijong Whang. A survey on data collection for machine learning: a big data - AI integration perspective. IEEE TKDE , 2019.
  98. [134]134. Yuji Roh, Kangwook Lee, Steven Euijong Whang, and Changho Suh. FR-Train: A mutual information-based approach to fair and robust training. In ICML , 2020.
  99. [135]135. Yuji Roh, Kangwook Lee, Steven Euijong Whang, and Changho Suh. Fairbatch: Batch selection for model fairness. In ICLR . OpenReview.net, 2021.
  100. [136]136. Yuji Roh, Kangwook Lee, Steven Euijong Whang, and Changho Suh. Sample selection for fair and robust training. In NeurIPS , 2021.
  101. [138]138. S. Schelter, Joos-Hendrik B¨ ose, Johannes Kirschnick, T. Klein, and Stephan Seufert. Automatically tracking metadata and provenance of machine learning experiments. In Workshop on ML Systems at NIPS , 2017.
  102. [139]139. Sebastian Schelter, Stefan Grafberger, Philipp Schmidt, Tammo Rukat, Mario Kießling, Andrey Taptunov, Felix Bießmann, and Dustin Lange. Differential data quality verification on partitioned data. In ICDE , pages 19401945, 2019.
  103. [140]140. Sebastian Schelter, Dustin Lange, Philipp Schmidt, Meltem Celikel, Felix Bießmann, and Andreas Grafberger. Automating large-scale data quality verification. Proc. VLDB Endow. , 11(12):1781-1794, 2018.
  104. [141]141. Sebastian Schelter, Tammo Rukat, and Felix Biessmann. JENGA - A framework to study the impact of data errors on the predictions of machine learning models. In EDBT , pages 529-534, 2021.
  105. [142]142. D. Sculley, Gary Holt, Daniel Golovin, Eugene Davydov, Todd Phillips, Dietmar Ebner, Vinay Chaudhary, Michael Young, Jean-Fran¸ cois Crespo, and Dan Dennison. Hidden technical debt in machine learning systems. In NIPS , pages 2503-2511, 2015.
  106. [144]144. Ali Shafahi, W. Ronny Huang, Mahyar Najibi, Octavian Suciu, Christoph Studer, Tudor Dumitras, and Tom Goldstein. Poison frogs! targeted clean-label poisoning attacks on neural networks. In NeurIPS , pages 61066116, 2018.
  107. [145]145. Li Shang. Denoising natural images based on a modified sparse coding algorithm. Applied mathematics and computation , 205(2):883-889, 2008.
  108. [146]146. Zheyan Shen, Jiashuo Liu, Yue He, Xingxuan Zhang, Renzhe Xu, Han Yu, and Peng Cui. Towards out-ofdistribution generalization: A survey. arXiv preprint arXiv:2108.13624 , 2021. · arXiv
  109. [147]147. Victor S. Sheng, Foster J. Provost, and Panagiotis G. Ipeirotis. Get another label? improving data quality and data mining using multiple, noisy labelers. In KDD , pages 614-622, 2008.
  110. [148]148. Aman Sinha, Hongseok Namkoong, and John C. Duchi. Certifying some distributional robustness with principled adversarial training. In ICLR , 2018.
  111. [149]149. David Solans, Battista Biggio, and Carlos Castillo. Poisoning attacks on algorithmic fairness. In Frank Hutter, Kristian Kersting, Jefrey Lijffijt, and Isabel Valera, editors, ECML PKDD , volume 12457, pages 162-177. Springer, 2020.
  112. [150]150. Hwanjun Song, Minseok Kim, and Jae-Gil Lee. SELFIE: refurbishing unclean samples for robust deep learning. In ICML , pages 5907-5915, 2019.
  113. [151]151. Hwanjun Song, Minseok Kim, Dongmin Park, and JaeGil Lee. Learning from noisy labels with deep neural networks: A survey. CoRR , abs/2007.08199, 2020. · arXiv
  114. [152]152. Hwanjun Song, Minseok Kim, Dongmin Park, Yooju Shin, and Jae-Gil Lee. Robust learning by self-transition for handling noisy labels. In KDD , pages 1490-1500, 2021.
  115. [153]153. Michael Stonebraker and Ihab F. Ilyas. Data integration: The current status and the way forward. IEEE Data Eng. Bull. , 41(2):3-9, 2018.
  116. [154]154. Michael Stonebraker and El Kindi Rezig. Machine learning and big data: What is important? IEEE Data Eng. Bull. , 2019.
  117. [155]155. Ki Hyun Tae and Steven Euijong Whang. Slice tuner: A selective data acquisition framework for accurate and fair machine learning models. In SIGMOD , pages 17711783. ACM, 2021.
  118. [156]156. Antti Tarvainen and Harri Valpola. Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results. In NIPS , pages 1195-1204, 2017.
  119. [159]159. Jonathan Tremblay, Aayush Prakash, David Acuna, Mark Brophy, Varun Jampani, Cem Anil, Thang To, Eric Cameracci, Shaad Boochoon, and Stan Birchfield. Training deep networks with synthetic data: Bridging the reality gap by domain randomization. In CVPR Workshops , pages 969-977, 2018.
  120. [160]160. Isaac Triguero, Salvador Garc´ ıa, and Francisco Herrera. Self-labeled techniques for semi-supervised learning: taxonomy, software and empirical study. Knowl. Inf. Syst. , 42(2):245-284, 2015.
  121. [161]161. John W Tukey. A survey of sampling from contaminated distributions. Contributions to probability and statistics , pages 448-485, 1960.
  122. [163]163. Paroma Varma and Christopher R´ e. Snuba: Automating weak supervision to label training data. Proc. VLDB Endow. , 12(3):223-236, 2018.
  123. [164]164. Manasi Vartak, Sajjadur Rahman, Samuel Madden, Aditya G. Parameswaran, and Neoklis Polyzotis. SEEDB: Efficient data-driven visualization recommendations to support visual analytics. PVLDB , 8(13):2182-2193, 2015.
  124. [166]166. Hao Wang, Bing Liu, Chaozhuo Li, Yan Yang, and Tianrui Li. Learning with noisy labels for sentence-level sentiment classification. In EMNLP , 2019.
  125. [167]167. Jialu Wang, Yang Liu, and Caleb Levy. Fair classification with group-dependent label noise. In Madeleine Clare Elish, William Isaac, and Richard S. Zemel, editors, FAccT , pages 526-536. ACM, 2021.
  126. [168]168. Serena Wang, Wenshuo Guo, Harikrishna Narasimhan, Andrew Cotter, Maya R. Gupta, and Michael I. Jordan. Robust optimization for fairness with noisy protected groups. In Hugo Larochelle, Marc'Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and HsuanTien Lin, editors, NeurIPS , 2020.
  127. [169]169. Steven Euijong Whang and Jae-Gil Lee. Data collection and quality challenges for deep learning. Proc. VLDB Endow. , 13(12):3429-3432, 2020.
  128. [18]18. interactivity of dataframe queries by leveraging think time. IEEE Data Eng. Bull. , 44(1):66-78, 2021.
  129. [171]171. Depeng Xu, Shuhan Yuan, Lu Zhang, and Xintao Wu. Fairgan: Fairness-aware generative adversarial networks. In IEEE Big Data , pages 570-575, 2018.
  130. [172]172. Han Xu, Xiaorui Liu, Yaxin Li, Anil K. Jain, and Jiliang Tang. To be robust or to be fair: Towards fairness in adversarial training. In Marina Meila and Tong Zhang, editors, ICML , volume 139, pages 11492-11501. PMLR, 2021.
  131. [173]173. Weilin Xu, David Evans, and Yanjun Qi. Feature squeezing: Detecting adversarial examples in deep neural networks. In NDSS , 2018.
  132. [174]174. David Yarowsky. Unsupervised word sense disambiguation rivaling supervised methods. In ACL , pages 189196, Stroudsburg, PA, USA, 1995. Association for Computational Linguistics.
  133. [175]175. Sangdoo Yun, Dongyoon Han, Sanghyuk Chun, Seong Joon Oh, Youngjoon Yoo, and Junsuk Choe. Cutmix: Regularization strategy to train strong classifiers with localizable features. In ICCV , pages 6022-6031, 2019.
  134. [176]176. Muhammad Bilal Zafar, Isabel Valera, Manuel GomezRodriguez, and Krishna P. Gummadi. Fairness beyond disparate treatment & disparate impact: Learning classification without disparate mistreatment. In WWW , pages 1171-1180. ACM, 2017.
  135. [177]177. Muhammad Bilal Zafar, Isabel Valera, Manuel GomezRodriguez, and Krishna P. Gummadi. Fairness constraints: Mechanisms for fair classification. In AISTATS , pages 962-970, 2017.
  136. [178]178. Brian Hu Zhang, Blake Lemoine, and Margaret Mitchell. Mitigating unwanted biases with adversarial learning. In AIES , pages 335-340, 2018.
  137. [179]179. Hantian Zhang, Xu Chu, Abolfazl Asudeh, and Shamkant B. Navathe. Omnifair: A declarative system for model-agnostic group fairness in machine learning. In SIGMOD , pages 2076-2088, 2021.
  138. [180]180. Hongjing Zhang and Ian Davidson. Facct. pages 138148. ACM, 2021.
  139. [181]181. Hongyi Zhang, Moustapha Ciss´ e, Yann N. Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. In ICLR , 2018.
  140. [182]182. Junzhe Zhang and Elias Bareinboim. Fairness in decision-making the causal explanation formula. In AAAI , 2018.
  141. [183]183. Yi Zhang and Zachary G. Ives. Finding related tables in data lakes for interactive data science. In SIGMOD , pages 1951-1966, 2020.
  142. [184]184. Zheguang Zhao, Lorenzo De Stefani, Emanuel Zgraggen, Carsten Binnig, Eli Upfal, and Tim Kraska. Controlling false discoveries during interactive data exploration. In SIGMOD , pages 527-540, 2017.
  143. [185]185. Yan Zhou and Sally A. Goldman. Democratic colearning. In IEEE ICTAI , pages 594-602, 2004.
  144. [186]186. Zhi-Hua Zhou and Ming Li. Tri-training: Exploiting unlabeled data using three classifiers. IEEE TKDE , 17(11):1529-1541, November 2005.
  145. [187]187. Chen Zhu, W. Ronny Huang, Hengduo Li, Gavin Taylor, Christoph Studer, and Tom Goldstein. Transferable clean-label poisoning attacks on deep neural nets. In ICML , pages 7614-7623, 2019.

Cited by

Sorted by citations

Authors
Whang, S. E., Roh, Y., Song, H., & Lee, J.-G.
Affiliations
  • Korea Advanced Institute of Science and Technology (KR)
  • NAVER Cloud (South Korea) (KR)
Corresponding author
Whang, Steven Euijong
ORCID
Venue
The VLDB Journal
Publisher
Springer Science and Business Media LLC
Published
2023-01-03
Volume
32
Issue
4
Pages
791-813
ISSN
  • 0949-877X (ISSN-L)
  • 1066-8888
DOI
10.1007/s00778-022-00775-9
OpenAlex
W4226251359
Cited by
573
References
10
Citations by year
2026: 134 · 2025: 214 · 2024: 172 · 2023: 47 · 2022: 5 · 2017: 1
FWCI
104.48
Type
journal article
Language
en
Open access
Licence
Locations
  • The VLDB Journal (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University) (cc-by)
Funders
  • National Research Foundation of Korea (NRF-2018R1A5A1059921, NRF-2022R1A2C2004382)
  • Institute of Information and Communications Technology Planning and Evaluation (2020-0-00862, 2022-0-00157)
Topics
  • Social Sciences › Decision Sciences › Management Science and Operations Research › Data Quality and Management
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Data Classification
  • Social Sciences › Safety Research › Ethics and Social Impacts of AI
Keywords
data-centric AI, data collection, data quality, deep learning, data preparation, data cleaning, data integration, data validation, data poisoning, data bias, fairness, data management

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review