Data collection and quality challenges for deep learning
The DOI matches its Crossref record; 2 of the 2 references it cites were found in a registry or an index.
Abstract
Software 2.0 refers to the fundamental shift in software engineering where using machine learning becomes the new norm in software with the availability of big data and computing infrastructure. As a result, many software engineering practices need to be rethought from scratch where data becomes a first-class citizen, on par with code. It is well known that 80--90% of the time for machine learning development is spent on data preparation. Also, even the best machine learning algorithms cannot perform well without good data or at least handling biased and dirty data during model training. In this tutorial, we focus on data collection and quality challenges that frequently occur in deep learning applications. Compared to traditional machine learning, there is less need for feature engineering, but more need for significant amounts of data. We thus go through state-of-the-art data collection techniques for machine learning. Then, we cover data validation and cleaning techniques for improving data quality. Even if the data is still problematic, hope is not lost, and we cover fair and robust training techniques for handling data bias and errors. We believe that the data management community is well poised to lead the research in these directions. The presenters have extensive experience in developing machine learning platforms and publishing papers in top-tier database, data mining, and machine learning venues.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
TFX: A TensorFlow-Based Production-Scale Machine Learning Platform
Baylor, D., Breck, E., Cheng, H.-T., Fiedel, N., Foo, C. Y., Haque, Z., Haykal, S., Ispir, M., Jain, V., Koc, L., et al. (2017) · Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/3097983.3098021 · Cited by 350
Self-citationCasting out Demons: Sanitizing Training Data for Anomaly Sensors
Cretu, G. F., Stavrou, A., Locasto, M. E., Stolfo, S. J., & Keromytis, A. D. (2008) · 2008 IEEE Symposium on Security and Privacy (sp 2008) · doi:10.1109/sp.2008.11 · Cited by 226 · cited 1×
“The notion of data sanitization was introduced in 2008 [4] where attacks were assumed to occur in relatively confined time intervals, and the sanitization techniques used training metadata.”
Method
Research works printed but not linked: 17
- E. Breck, M. Zinkevich, N. Polyzotis, S. Whang, and S. Roy. Data validation for machine learning. In MLSys, 2019.
- 10.1038/s41598-018-24271-9 · DOI · Paper page
- L. Dong and T. Rekatsinas. Data integration and machine learning: A natural synergy. PVLDB, 11(12):2094--2097, 2018.
- 10.1145/3310205 · DOI · Paper page
- 10.1145/3035918.3054775 · DOI
- P. Li, X. Rao, J. Blase, Y. Zhang, X. Chu, and C. Zhang. CleanML: A benchmark for joint data cleaning and machine learning. CoRR, abs/1904.09483, 2019. · arXiv
- 10.1109/ICDE.2018.00062 · DOI · Paper page
- F. Nargesian, E. Zhu, R. J. Miller, K. Q. Pu, and P. C. Arocena. Data lake management: Challenges and opportunities. PVLDB, 12(12):1986--1989, 2019.
- 10.1145/3366423.3380233 · doi:10.1145/3366423.3380233
- 10.1145/3035918.3054782 · DOI · Paper page
- Y. Roh, G. Heo, and S. E. Whang. A survey on data collection for machine learning: a big data - AI integration perspective. IEEE TKDE, 2019.
- Y. Roh, K. Lee, S. E. Whang, and C. Suh. FR-Train: A mutual information-based approach to fair and robust training. In ICML, 2020.
- H. Song, M. Kim, and J. Lee. SELFIE: Refurbishing unclean samples for robust deep learning. In ICML, pages 5907--5915, 2019.
- M. Stonebraker and E. K. Rezig. Machine learning and big data: What is important? IEEE Data Eng. Bull., 2019.
- 10.14778/2831360.2831371 · DOI · Paper page
- 10.1145/3294052.3322192 · DOI
- 10.1145/3035918.3064019 · DOI · Paper page
Cited by
Data collection and quality challenges in deep learning: a data-centric AI perspective
Whang, S. E., Roh, Y., Song, H., & Lee, J.-G. (2023) · The VLDB Journal · doi:10.1007/s00778-022-00775-9 · Cited by 573
Self-citationA Review on Deep-Learning-Based Cyberbullying Detection
Hasan, M. T., Hossain, M. A. E., Mukta, M. S. H., Akter, A., Ahmed, M., & Islam, S. (2023) · Future Internet · doi:10.3390/fi15050179 · Cited by 103
“Furthermore, even the strongest DL algorithms will struggle to function without good data and present weak performance to handle biased and unclean data during model training [199].”
BackgroundTransforming Digital Marketing with Generative AI
Islam, T., Miron, A., Nandy, M., Choudrie, J., Liu, X., & Li, Y. (2024) · Computers · doi:10.3390/computers13070168 · Cited by 68
Metadata
- Authors
- Whang, S. E., & Lee, J.-G.
- Venue
- Proceedings of the VLDB Endowment
- Published
- 2020-08
- Volume
- 13
- Issue
- 12
- Pages
- 3429-3432
- ISSN
- 2150-8097 (ISSN-L)
- OpenAlex
- W3085364681
- Cited by
- 153
- References
- 2
- FWCI
- 8.60
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Data collection and quality challenges for deep learning. https://app.cochat.ai/papers/10.14778/3415478.3415562
- The paper
- Whang, S. E., & Lee, J.-G. (2020). Data collection and quality challenges for deep learning. Proceedings of the VLDB Endowment. https://doi.org/10.14778/3415478.3415562
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.