Data management for production quality deep learning models: Challenges and solutions
Open accessThe DOI matches its Crossref record; 25 of the 25 references it cites were found in a registry or an index.
Abstract
Deep learning (DL) based software systems are difficult to develop and maintain in industrial settings due to several challenges. Data management is one of the most prominent challenges which complicates DL in industrial deployments. DL models are data-hungry and require high-quality data. Therefore, the volume, variety, velocity, and quality of data cannot be compromised. This study aims to explore the data management challenges encountered by practitioners developing systems with DL components, identify the potential solutions from the literature and validate the solutions through a multiple case study. We identified 20 data management challenges experienced by DL practitioners through a multiple interpretive case study. Further, we identified 48 articles through a systematic literature review that discuss the solutions for the data management challenges. With the second round of multiple case study, we show that many of these solutions have limitations and are not used in practice due to a combination of four factors: high cost, lack of skill-set and infrastructure, inability to solve the problem completely, and incompatibility with certain DL use cases. Thus, data management for data-intensive DL models in production is complicated. Although the DL technology has achieved very promising results, there is still a significant need for further research in the field of data management to build high-quality datasets and streams that can be used for building production-ready DL systems. Furthermore, we have classified the data management challenges into four categories based on the availability of the solutions.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
LeCun, Y., Bengio, Y., & Hinton, G. (2015) · Nature · doi:10.1038/nature14539 · Cited by 84,126
SMOTE synthetic minority over-sampling technique
Chawla, N. V., Bowyer, K. W., Hall, L. O., & Kegelmeyer, W. P. (2002) · Journal of Artificial Intelligence Research · doi:10.1613/jair.953 · Cited by 32,865
Building Theories from Case Study Research
Eisenhardt, K. M. (1989) · The Academy of Management Review · doi:10.2307/258557 · Cited by 17,455
A survey on deep learning in medical image analysis
Litjens, G., Kooi, T., Bejnordi, B. E., Setio, A. A. A., Ciompi, F., Ghafoorian, M., van der Laak, J. A. W. M., van Ginneken, B., & Sánchez, C. I. (2017) · Medical Image Analysis · doi:10.1016/j.media.2017.07.005 · Cited by 15,332
Backpropagation Applied to Handwritten Zip Code Recognition
LeCun, Y., Boser, B., Denker, J. S., Henderson, D., Howard, R. E., Hubbard, W., & Jackel, L. D. (1989) · Neural Computation · doi:10.1162/neco.1989.1.4.541 · Cited by 11,891
TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems
Abadi, M., Agarwal, A., Barham, P., Brevdo, E., Chen, Z., Citro, C., Corrado, G. S., Davis, A., Dean, J., Devin, M., et al. (2016) · arXiv · doi:10.48550/arxiv.1603.04467 · Cited by 9,408
Qualitative Case Study Methodology: Study Design and Implementation for Novice Researchers
Baxter, P., & Jack, S. (2015) · The Qualitative Report · doi:10.46743/2160-3715/2008.1573 · Cited by 7,594
Guidelines for conducting and reporting case study research in software engineering
Runeson, P., & Höst, M. (2009) · Empirical Software Engineering · doi:10.1007/s10664-008-9102-8 · Cited by 3,862
Deep Neural Networks for YouTube Recommendations
Covington, P., Adams, J., & Sargin, E. (2016) · Proceedings of the 10th ACM Conference on Recommender Systems · doi:10.1145/2959100.2959190 · Cited by 3,539
Deep Learning Based Recommender System: A Survey and New Perspectives
Zhang, S., Yao, L., Sun, A., & Tay, Y. (2020) · ACM Computing Surveys · doi:10.1145/3285029 · Cited by 3,485
Deep Learning: Methods and Applications
Deng, L., & Yu, D. (2014) · Foundations and Trends® in Signal Processing · doi:10.1561/2000000039 · Cited by 3,480
Deep learning applications and challenges in big data analytics
Najafabadi, M. M., Villanustre, F., Khoshgoftaar, T. M., Seliya, N., Wald, R., & Muharemagic, E. (2015) · Journal of Big Data · doi:10.1186/s40537-014-0007-7 · Cited by 2,594
Doing a thematic analysis: A practical, step-by-step guide for learning and teaching scholars.
Maguire, M., & Delahunt, B. (2017) · All Ireland Journal of Higher Education · doi:10.62707/aishej.v9i3.335 · Cited by 2,099
Horovod: fast and easy distributed deep learning in TensorFlow
Sergeev, A., & Del Balso, M. (2018) · arXiv · doi:10.48550/arxiv.1802.05799 · Cited by 1,620
Self-Supervised Learning of Pretext-Invariant Representations
Misra, I., & van der Maaten, L. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00674 · Cited by 1,435 · cited 1×
“Self-supervised learning attempts to overcome these restrictions by learning image representations directly from pixels, rather than depending on pre-defined semantic labels (Misra and Maaten, 2020).”
BackgroundBig data deep learning: Challenges and perspectives
Chen, X.-W., & Lin, X. (2014) · IEEE Access · doi:10.1109/access.2014.2325029 · Cited by 1,255 · cited 1×
“Popular companies like Apple (Chen and Lin, 2014), Facebook, Microsoft is collecting a copious amount of data daily through applications like Siri, Google translator, Bing voice search (Jones, 2014) to provide a variety of other services such as reminders, weather reports, personalized…”
BackgroundChallenges and opportunities with big data
Labrinidis, A., & Jagadish, H. V. (2012) · Proceedings of the VLDB Endowment · doi:10.14778/2367502.2367572 · Cited by 1,070 · cited 1×
“Machine learning and deep learning algorithms are capable of extracting every last detail from the input data, which is then utilized to develop new rules to fulfill the function (Labrinidis and Jagadish, 2012).”
Zero-Shot Learning Through Cross-Modal Transfer
Socher, R., Ganjoo, M., Sridhar, H., Bastani, O., Manning, C. D., & Ng, A. Y. (2013) · arXiv · doi:10.48550/arxiv.1301.3666 · Cited by 997
Zero-Shot Learning — The Good, the Bad and the Ugly
Xian, Y., Schiele, B., & Akata, Z. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.328 · Cited by 891 · cited 1×
“The goal of zero-shot learning is to recognize items that were not seen during training (Xian et al., 2017).”
BackgroundRecent advances in deep learning for speech research at Microsoft
Deng, L., Li, J., Huang, J.-T., Yao, K., Yu, D., Seide, F., Seltzer, M., Zweig, G., He, X., Williams, J., et al. (2013) · 2013 IEEE International Conference on Acoustics, Speech and Signal Processing · doi:10.1109/icassp.2013.6639345 · Cited by 801 · cited 1×
“Instead, it can be any application such as a visualization tool (Matheus et al., 2018; Stadler et al., 2016), Machine Learning(ML) models (Gautam and Yadav, 2014; Tanzil et al., 2017) or Deep Learning(DL) models (Covington et al., 2016; Deng et al., 2013).”
Time Series Data Augmentation for Deep Learning: A Survey
Wen, Q., Sun, L., Yang, F., Song, X., Gao, J., Wang, X., & Xu, H. (2021) · Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2021/631 · Cited by 727
Time Series Data Augmentation for Deep Learning: A Survey
Wen, Q., Sun, L., Yang, F., Song, X., Gao, J., Wang, X., & Xu, H. (2020) · arXiv · doi:10.48550/arxiv.2002.12478 · Cited by 727
Using Self-Supervised Learning Can Improve Model Robustness and Uncertainty
Hendrycks, D., Mazeika, M., Kadavath, S., & Song, D. (2019) · arXiv · doi:10.48550/arxiv.1906.12340 · Cited by 634
Entity Embeddings of Categorical Variables
Guo, C., & Berkhahn, F. (2016) · arXiv · doi:10.48550/arxiv.1604.06737 · Cited by 308 · cited 1×
“The disadvantage is that it necessitates learning as part of the model and the creation of many more input variables (columns) (Guo and Berkhahn, 2016).”
BackgroundA Dissection of Overfitting and Generalization in Continuous Reinforcement Learning
Zhang, A., Ballas, N., & Pineau, J. (2018) · arXiv · doi:10.48550/arxiv.1806.07937 · Cited by 88
Cited by
Jarmakovica, A. (2025) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2025.1621514 · Cited by 10
Enhancing machine learning algorithm performance for PCOS diagnosis using SMOTENC on imbalanced data
Dewi, R., Sri hayati, R., Saleh, A., Hakim Tanjung, D. Y., & Jinan, A. (2025) · JITK (Jurnal Ilmu Pengetahuan dan Teknologi Komputer) · doi:10.33480/jitk.v11i1.6676 · Cited by 2
Metadata
- Authors
- Munappy, A. R., Bosch, J., Olsson, H. H., Arpteg, A., & Brinne, B.
- Venue
- Journal of Systems and Software
- Publisher
- Elsevier BV
- Published
- 2022-09
- Volume
- 191
- Pages
- 111359
- ISSN
- 0164-1212 (ISSN-L)
- OpenAlex
- W4280577524
- Cited by
- 177
- References
- 25
- FWCI
- 34.54
- Type
- journal article
- Language
- en
- Open access
- hybrid
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.