Automated data processing and feature engineering for deep learning and big data applications: A survey
Open accessThe DOI matches its Crossref record; 56 of the 56 references it cites were found in a registry or an index.
Abstract
Modern approach to artificial intelligence (AI) aims to design algorithms that learn directly from data. This approach has achieved impressive results and has contributed significantly to the progress of AI, particularly in the sphere of supervised deep learning. It has also simplified the design of machine learning systems as the learning process is highly automated. However, not all data processing tasks in conventional deep learning pipelines have been automated. In most cases data has to be manually collected, preprocessed and further extended through data augmentation before they can be effective for training. Recently, special techniques for automating these tasks have emerged. The automation of data processing tasks is driven by the need to utilize large volumes of complex, heterogeneous data for machine learning and big data applications. Today, end-to-end automated data processing systems based on automated machine learning (AutoML) techniques are capable of taking raw data and transforming them into useful features for Big Data tasks by automating all intermediate processing stages. In this work, we present a thorough review of approaches for automating data processing tasks in deep learning pipelines, including automated data preprocessing– e.g., data cleaning, labeling, missing data imputation, and categorical data encoding–as well as data augmentation (including synthetic data generation using generative AI methods) and feature engineering–specifically, automated feature extraction, feature construction and feature selection. In addition to automating specific data processing tasks, we discuss the use of AutoML methods and tools to simultaneously optimize all stages of the machine learning pipeline.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333
PyTorch: An Imperative Style, High-Performance Deep Learning Library
Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., et al. (2019) · arXiv · doi:10.48550/arxiv.1912.01703 · Cited by 31,473
High-Resolution Image Synthesis with Latent Diffusion Models
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., & Ommer, B. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01042 · Cited by 20,302
Learning Transferable Visual Models From Natural Language Supervision
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. (2021) · arXiv · doi:10.48550/arxiv.2103.00020 · Cited by 15,293
A survey on Image Data Augmentation for Deep Learning
Shorten, C., & Khoshgoftaar, T. M. (2019) · Journal of Big Data · doi:10.1186/s40537-019-0197-0 · Cited by 13,203
Representation Learning: A Review and New Perspectives
Bengio, Y., Courville, A., & Vincent, P. (2013) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2013.50 · Cited by 13,001
TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems
Abadi, M., Agarwal, A., Barham, P., Brevdo, E., Chen, Z., Citro, C., Corrado, G. S., Davis, A., Dean, J., Devin, M., et al. (2016) · arXiv · doi:10.48550/arxiv.1603.04467 · Cited by 9,408
Zagoruyko, S., & Komodakis, N. (2016) · Procedings of the British Machine Vision Conference 2016 · doi:10.5244/c.30.87 · Cited by 7,244
XGBoost: A Scalable Tree Boosting System
Chen, T., & Guestrin, C. (2016) · arXiv · doi:10.48550/arxiv.1603.02754 · Cited by 7,104
LLaMA: Open and Efficient Foundation Language Models
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. (2023) · arXiv · doi:10.48550/arxiv.2302.13971 · Cited by 4,602
A Survey on Evaluation of Large Language Models
Chang, Y., Wang, X., Wang, J., Wu, Y., Yang, L., Zhu, K., Chen, H., Yi, X., Wang, C., Wang, Y., et al. (2024) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3641289 · Cited by 4,254 · cited 1×
“…based on variational autoencoders [168], generative adversarial networks [169], diffusion models [170], autoregressive models [171], [172] and large language models (LLMs) [173] are commonly designed to directly synthesize data by themselves, thereby bypassing all intermediate processing steps.”
BackgroundRandom Erasing Data Augmentation
Zhong, Z., Zheng, L., Kang, G., Li, S., & Yang, Y. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i07.7000 · Cited by 3,617
Random Erasing Data Augmentation
Zhong, Z., Zheng, L., Kang, G., Li, S., & Yang, Y. (2017) · arXiv · doi:10.48550/arxiv.1708.04896 · Cited by 3,617
Deep Learning: Methods and Applications
Deng, L., & Yu, D. (2014) · Foundations and Trends® in Signal Processing · doi:10.1561/2000000039 · Cited by 3,480 · cited 1×
“With modern deep learning models based on neural network frameworks [27], [28], data processing tasks are usually implemented within the machine learning framework and trained end-to-end (Figure 6).”
BackgroundRandAugment: Practical automated data augmentation with a reduced search space
Cubuk, E. D., Zoph, B., Shlens, J., & Le, Q. V. (2019) · arXiv · doi:10.48550/arxiv.1909.13719 · Cited by 3,223
Improved regularization of convolutional neural networks with cutout
DeVries, T., & Taylor, G. W. (2017) · arXiv · doi:10.48550/arxiv.1708.04552 · Cited by 2,776
Deep Unsupervised Learning using Nonequilibrium Thermodynamics
Sohl-Dickstein, J., Weiss, E. A., Maheswaranathan, N., & Ganguli, S. (2015) · arXiv · doi:10.48550/arxiv.1503.03585 · Cited by 2,637 · cited 1×
“They are based on nonequilibrium thermodynamics [180].”
Deep learning applications and challenges in big data analytics
Najafabadi, M. M., Villanustre, F., Khoshgoftaar, T. M., Seliya, N., Wald, R., & Muharemagic, E. (2015) · Journal of Big Data · doi:10.1186/s40537-014-0007-7 · Cited by 2,594
Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding
Saharia, C., Chan, W., Saxena, S., Li, L., Whang, J., Denton, E., Ghasemipour, S. K. S., Ayan, B. K., Mahdavi, S. S., Lopes, R. G., et al. (2022) · arXiv · doi:10.48550/arxiv.2205.11487 · Cited by 2,528 · cited 1×
“[211].”
Hierarchical Text-Conditional Image Generation with CLIP Latents
Ramesh, A., Dhariwal, P., Nichol, A., Chu, C., & Chen, M. (2022) · arXiv · doi:10.48550/arxiv.2204.06125 · Cited by 2,511
Deep Networks with Stochastic Depth
Huang, G., Sun, Y., Liu, Z., Sedra, D., & Weinberger, K. Q. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46493-0_39 · Cited by 2,104
Neural Architecture Search: A Survey
Elsken, T., Metzen, J. H., & Hutter, F. (2018) · arXiv · doi:10.48550/arxiv.1808.05377 · Cited by 1,763
Toward expert-level medical question answering with large language models
Singhal, K., Tu, T., Gottweis, J., Sayres, R., Wulczyn, E., Amin, M., Hou, L., Clark, K., Pfohl, S. R., Cole-Lewis, H., et al. (2025) · Nature Medicine · doi:10.1038/s41591-024-03423-7 · Cited by 1,734
Auto-WEKA: combined selection and hyperparameter optimization of classification algorithms
Thornton, C., Hutter, F., Hoos, H. H., & Leyton-Brown, K. (2013) · Proceedings of the 19th ACM SIGKDD international conference on Knowledge discovery and data mining · doi:10.1145/2487575.2487629 · Cited by 1,359
GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models
Nichol, A., Dhariwal, P., Ramesh, A., Shyam, P., Mishkin, P., McGrew, B., Sutskever, I., & Chen, M. (2021) · arXiv · doi:10.48550/arxiv.2112.10741 · Cited by 1,357 · cited 2×
“Some of the most prominent data generation models in this category include Dalle-2 [209], Stable Diffusion [210], Glide [193] and Imagen [190] and StyleGAN-NADA [191].”
BackgroundMethodAutoAugment: Learning Augmentation Policies from Data
Cubuk, E. D., Zoph, B., Mane, D., Vasudevan, V., & Le, Q. V. (2018) · arXiv · doi:10.48550/arxiv.1805.09501 · Cited by 1,106
Machine learning with big data: challenges and approaches.
L'Heureux, A., Grolinger, K., Elyamany, H. F., & Capretz, M. A. M. (2017) · IEEE Access · doi:10.1109/access.2017.2696365 · Cited by 1,023 · cited 1×
“While deep learning methods have performed incredibly well, in the era of Big Data they have been shown to lack the power necessary to handle large volumes of complex heterogeneous data [1], [2].”
MethodData augmentation: A comprehensive survey of modern approaches
Mumuni, A., & Mumuni, F. (2022) · Array · doi:10.1016/j.array.2022.100258 · Cited by 891
Self-citationAutomated machine learning: Review of the state-of-the-art and opportunities for healthcare
Waring, J., Lindvall, C., & Umeton, R. (2020) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2020.101822 · Cited by 834 · cited 3×
“Indeed, healthcare [7], [8] and business applications are already benefiting immensely from these techniques [9], [10], [11].”
BackgroundSRDiff: Single image super-resolution with diffusion probabilistic models
Li, H., Yang, Y., Chang, M., Chen, S., Feng, H., Xu, Z., Li, Q., & Chen, Y. (2022) · Neurocomputing · doi:10.1016/j.neucom.2022.01.029 · Cited by 777
High-Resolution Image Synthesis with Latent Diffusion Models
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., & Ommer, B. (2021) · arXiv · doi:10.48550/arxiv.2112.10752 · Cited by 737
Extracting Training Data from Large Language Models
Carlini, N., Tramer, F., Wallace, E., Jagielski, M., Herbert-Voss, A., Lee, K., Roberts, A., Brown, T., Song, D., Erlingsson, U., et al. (2020) · arXiv · doi:10.48550/arxiv.2012.07805 · Cited by 727 · cited 1×
“Data generated this way can be used to augment existing data (e.g., in [174], [175], [176]) or completely replace natural data in situations where datasets are inadequate or are inaccessible for training machine learning models (e.g., in [177], [178]).”
MethodTime Series Data Augmentation for Deep Learning: A Survey
Wen, Q., Sun, L., Yang, F., Song, X., Gao, J., Wang, X., & Xu, H. (2021) · Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2021/631 · Cited by 727
Time Series Data Augmentation for Deep Learning: A Survey
Wen, Q., Sun, L., Yang, F., Song, X., Gao, J., Wang, X., & Xu, H. (2020) · arXiv · doi:10.48550/arxiv.2002.12478 · Cited by 727
DiffusionCLIP: Text-Guided Diffusion Models for Robust Image Manipulation
Kim, G., Kwon, T., & Ye, J. C. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.00246 · Cited by 570 · cited 4×
“DiffusionCLIP [184] introduces an approach to perform various image transformations using text-guided prompts with a diffusion model developed on the basis of CLIP [189].”
MethodInfluentialDiffWave: A Versatile Diffusion Model for Audio Synthesis
Kong, Z., Ping, W., Huang, J., Zhao, K., & Catanzaro, B. (2020) · arXiv · doi:10.48550/arxiv.2009.09761 · Cited by 569
A survey on data preprocessing for data stream mining: Current status and future directions
Ramírez-Gallego, S., Krawczyk, B., García, S., Woźniak, M., & Herrera, F. (2017) · Neurocomputing · doi:10.1016/j.neucom.2017.01.078 · Cited by 502
Autoregressive Image Generation using Residual Quantization
Lee, D., Kim, C., Kim, S., Cho, M., & Han, W.-S. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01123 · Cited by 445 · cited 1×
“…approaches based on variational autoencoders [168], generative adversarial networks [169], diffusion models [170], autoregressive models [171], [172] and large language models (LLMs) [173] are commonly designed to directly synthesize data by themselves, thereby bypassing all intermediate…”
BackgroundAvrahami, O., Fried, O., & Lischinski, D. (2023) · ACM Transactions on Graphics · doi:10.1145/3592450 · Cited by 345
Human-AI Collaboration in Data Science: Exploring Data Scientists' Perceptions of Automated AI
Wang, D., Weisz, J. D., Muller, M., Ram, P., Geyer, W., Dugan, C., Tausczik, Y., Samulowitz, H., & Gray, A. (2019) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3359313 · Cited by 342 · cited 1×
“It also enhances trust in AutoML solutions [333], [334] since developers can visually probe and validate pertinent issues related to trust and reliability in an intuitive manner.”
BackgroundApplications of multiple imputation in medical studies: from AIDS to NHANES
Barnard, J., & Meng, X.-L. (1999) · Statistical Methods in Medical Research · doi:10.1177/096228029900800103 · Cited by 340 · cited 1×
“Traditional data imputation methods (e.g., [77], [78]) rely on analytically formulated statistical techniques.”
MethodTowards Expert-Level Medical Question Answering with Large Language Models
Singhal, K., Tu, T., Gottweis, J., Sayres, R., Wulczyn, E., Hou, L., Clark, K., Pfohl, S., Cole-Lewis, H., Neal, D., et al. (2023) · arXiv · doi:10.48550/arxiv.2305.09617 · Cited by 338
The Effects of Data Quality on Machine Learning Performance on Tabular Data
Mohammed, S., Budach, L., Feuerpfeil, M., Ihde, N., Nathansen, A., Noack, N., Patzlaff, H., Naumann, F., & Harmouch, H. (2025) · Information Systems · doi:10.1016/j.is.2025.102549 · Cited by 331
RICOPILI: Rapid Imputation for COnsortias PIpeLIne
Lam, M., Awasthi, S., Watson, H. J., Goldstein, J., Panagiotaropoulou, G., Trubetskoy, V., Karlsson, R., Frei, O., Fan, C.-C., De Witte, W., et al. (2020) · Bioinformatics · doi:10.1093/bioinformatics/btz633 · Cited by 315
Compositional Visual Generation with Composable Diffusion Models
Liu, N., Li, S., Du, Y., Torralba, A., & Tenenbaum, J. B. (2022) · arXiv · doi:10.48550/arxiv.2206.01714 · Cited by 295
Peng, X., Tang, Z., Yang, F., Feris, R. S., & Metaxas, D. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00237 · Cited by 240
Large Language Models Can Self-Improve
Huang, J., Gu, S. S., Hou, L., Wu, Y., Wang, X., Yu, H., & Han, J. (2022) · arXiv · doi:10.48550/arxiv.2210.11610 · Cited by 230 · cited 1×
“This approach is already being studied [345], and preliminary results are promising.”
ResultLarge Language Models Can Self-Improve
Huang, J., Gu, S., Hou, L., Wu, Y., Wang, X., Yu, H., & Han, J. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.67 · Cited by 230
Kubota, K. J., Chen, J. A., & Little, M. A. (2016) · Movement Disorders · doi:10.1002/mds.26693 · Cited by 210
GPT3Mix: Leveraging Large-scale Language Models for Text Augmentation
Yoo, K. M., Park, D., Kang, J., Lee, S.-W., & Park, W. (2021) · Findings of the Association for Computational Linguistics: EMNLP 2021 · doi:10.18653/v1/2021.findings-emnlp.192 · Cited by 151
Mitigating Bias in Radiology Machine Learning: 1. Data Handling
Rouzrokh, P., Khosravi, B., Faghani, S., Moassefi, M., Vera Garcia, D. V., Singh, Y., Zhang, K., Conte, G. M., & Erickson, B. J. (2022) · Radiology: Artificial Intelligence · doi:10.1148/ryai.210290 · Cited by 145
Generating Datasets with Pretrained Language Models
Schick, T., & Schütze, H. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.555 · Cited by 122
Memorization vs. Generalization : Quantifying Data Leakage in NLP Performance Evaluation
Elangovan, A., He, J., & Verspoor, K. (2021) · Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume · doi:10.18653/v1/2021.eacl-main.113 · Cited by 75
Metadata
- Authors
- Mumuni, A., & Mumuni, F.
- Venue
- Journal of Information and Intelligence
- Publisher
- Elsevier BV
- Published
- 2025-03
- Volume
- 3
- Issue
- 2
- Pages
- 113-153
- ISSN
- 2949-7159 (ISSN-L)
- OpenAlex
- W4390667445
- Cited by
- 159
- References
- 56
- FWCI
- 23.66
- Type
- journal article
- Language
- en
- Open access
- diamond
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.