Challenges in Deploying Machine Learning: A Survey of Case Studies
Open accessThe DOI matches its Crossref record; 127 of the 128 references it cites were found in a registry or an index.
Abstract
In recent years, machine learning has transitioned from a field of academic research interest to a field capable of solving real-world business problems. However, the deployment of machine learning models in production systems can present a number of issues and concerns. This survey reviews published reports of deploying machine learning solutions in a variety of use cases, industries, and applications and extracts practical considerations corresponding to stages of the machine learning deployment workflow. By mapping found challenges to the steps of the machine learning deployment workflow, we show that practitioners face issues at each stage of the deployment process. The goal of this article is to lay out a research agenda to explore approaches addressing these challenges.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“Compared with the first step, continual monitoring is used to identify data distribution drift, control model quality, and trigger system alarms when an ML model deviates from its normal behavior in the target setting [15].”
From Toward real‐world deployment of machine learning for health care: External validation, continual monitoring, and randomized clinical trials (2024) · cites this paper as [15]
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“Our literature review was conducted using a systematic mapping protocol [54] in order to select a small set of relevant articles from the much larger collection of literature emerging at the intersection of data quality and ML.”
From A Survey of Data Quality Requirements That Matter in ML Development Pipelines (2023) · cites this paper as [54]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325 · cited 2×
“They took one of the state-of-the-art models in the field, BERT [47], and found out that depending on the chosen model size full training procedure can cost anywhere between $50k and $1.”
BackgroundQuinlan, J. R. (1986) · Machine Learning · doi:10.1007/bf00116251 · Cited by 21,439 · cited 1×
“For that reason decision trees (DT) [108], which can be considered a fairly basic ML algorithm, are widely used in practice.”
MethodArmstrong, J. S., & Georgoff, D. M. (1988) · International Journal of Forecasting · doi:10.1016/0169-2070(88)90019-2 · Cited by 17,894
Rudin, C. (2019) · Nature Machine Intelligence · doi:10.1038/s42256-019-0048-x · Cited by 10,561 · cited 1×
“Rudin [111] called the ML community to stop using black box models and explaining their behavior afterward, and instead design models that are inherently interpretable.”
BackgroundOn the Dangers of Stochastic Parrots: Can Language Models Be Too Big? 🦜
Bender, E. M., Gebru, T., McMillan-Major, A., & Shmitchell, S. (2021) · Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3442188.3445922 · Cited by 6,988
Fully homomorphic encryption using ideal lattices
Gentry, C. (2009) · Proceedings of the forty-first annual ACM symposium on Theory of computing · doi:10.1145/1536414.1536440 · Cited by 6,806 · cited 1×
“Some of the ways this trade-of is resolved in practice are diferential privacy that explicitly corrupts the data [42], homomorphic encryption that restricts the class of learning algorithm but allows for training on encrypted data [50], and federated learning that distributes training across personal devices to preser…”
BackgroundPractical Bayesian Optimization of Machine Learning Algorithms
Snoek, J., Larochelle, H., & Adams, R. P. (2012) · arXiv · doi:10.48550/arxiv.1206.2944 · Cited by 6,691 · cited 1×
“Even approaches like Hyperband [78] or Bayesian optimization [129], that are speciically designed to minimize the number of training cycles needed, are not yet able to deal with the high dimensional searches that emerge when many hyper-parameters are involved.”
Dwork, C. (2006) · Lecture Notes in Computer Science · doi:10.1007/11787006_1 · Cited by 5,291 · cited 1×
“Some of the ways this tradeoff is resolved in practice are differential privacy that explicitly corrupts the data [54], homomorphic encryption that restricts the class of learning algorithm but allows for training on encrypted data [55], and federated learning that distributes training across personal devices to prese…”
BackgroundMembership Inference Attacks Against Machine Learning Models
Shokri, R., Stronati, M., Song, C., & Shmatikov, V. (2017) · 2017 IEEE Symposium on Security and Privacy (SP) · doi:10.1109/sp.2017.41 · Cited by 4,754 · cited 1×
“They veriied it on models trained on leading ML-as-a-service providers [127], achieving anywhere from 70% up to 94% accuracy of membership inference.”
BackgroundA survey on concept drift adaptation
Gama, J., Žliobaitė, I., Bifet, A., Pechenizkiy, M., & Bouchachia, A. (2014) · ACM Computing Surveys · doi:10.1145/2523813 · Cited by 3,735 · cited 1×
“We refer interested readers to the works on using domain adaptation [48], meta-learning [46] and transfer learning [152, 157] as ways of addressing dataset shift.”
Method
Research works printed but not linked: 26
- Arif Cam, Michael Chui, and Bryce Hall. 2019. Global AI survey: AI proves its worth, but few scale impact. McKinsey Analytics (2019). https://www.mckinsey.com/featured-insights/artificial-intelligence/global-ai-survey-ai-proves-its-worth-but-few-scale-impact. Accessed from 13/06/2022.
- Royal Society (Great Britain). 2017. Machine Learning: The Power and Promise of Computers that Learn by Example: an Introduction. Royal Society.
- Ben Lorica and Nathan Paco. 2018. The State of Machine Learning Adoption in the Enterprise. O’Reilly Media.
- Colin Shearer. 2000. The CRISP-DM model: The new blueprint for data mining. J. Data Warehous. 5, 4 (2000), 13–22.
- Rajashree Y. Patil and R. V. Kulkarni. 2012. A review of data cleaning algorithms for data warehouse systems. Int. J. Comput. Sci. Inf. Technol. 3, 5 (2012), 5212–5214.
- 10.1109/TITS.2019.2962338 · DOI · Paper page
- Diego Charrez. 2019. NeurIPS 2019 Stats. Retrieved from https://medium.com/@dcharrezt/neurips-2019-stats-c91346d31c8f.
- 10.1023/A:1022643204877 · doi:10.1023/a:1022643204877 · Paper page
- 10.5555/3122009.3242042 · doi:10.5555/3122009.3242042
- Bobak Shahriari, Alexandre Bouchard-Cote, and Nando Freitas. 2016. Unbounded Bayesian optimization via regularization. In Proceedings of the 19th International Conference on Artificial Intelligence and Statistics (Proceedings of Machine Learning Research), Arthur Gretton and Christian C. Robert (Eds.), Vol. 51. PMLR, 1168–1176. Retrieved from http://proceedings.mlr.press/v51/shahriari16.html.
- Ram Ananth, Seph Mard, and Peter Simon. 2019. Opening the “Black Box”: The path to deployment of AI models in banking, white paper. DataRobot (2019). https://www.datarobot.com/resources/opening-the-black-box/. Accessed from 13/06/2022.
- Robert Soden Dennis Wagenaar Dave Luo and Annegien Tijssen. 2019. Taking Ethics Fairness and Bias Seriously in Machine Learning for Disaster Risk Management. arxiv:cs.CY/1912.05538.
- Nantheera Anantrasirichai and David Bull. 2021. Artificial intelligence in the creative industries: A review. Artif. Intell. Rev. (2021), 1–68.
- D. Malan et al. 2016. Values and the fourth industrial revolution: Connecting the dots between value, values, profit and purpose. In Proceedings of the World Economic Forum.
- Daniel Malan. 2018. The law can’t keep up with new tech. Here’s how to close the gap. In Proceedings of the World Economic Forum.
- Ramprakash Ramamoorthy P. Satya Madhuri and Malini Christina Raj. 2019. AI from labs to production—challenges and learnings. USENIX Association. https://www.usenix.org/conference/opml19/presentation/madhuri. Accessed from 13/06/2022.
- Oscar Schwartz. 2019. In 2016 Microsoft’s racist chatbot revealed the dangers of online conversation. IEEE Spectr. 11 (2019).
- Florian Tramèr, Fan Zhang, Ari Juels, Michael K. Reiter, and Thomas Ristenpart. 2016. Stealing machine learning models via prediction APIs. In Proceedings of the 25th USENIX Security Symposium (USENIX Security’16). 601–618.
- Jeremy Hermann and Mike Del Balso. 2017. Meet Michelangelo: Uber’s machine learning platform. Retrieved from https://eng.uber.com/michelangelo.
- Matei Zaharia, Andrew Chen, Aaron Davidson, Ali Ghodsi, Sue Ann Hong, Andy Konwinski, Siddharth Murching, Tomas Nykodym, Paul Ogilvie, Mani Parkhe, et al. 2018. Accelerating the machine learning lifecycle with MLflow.IEEE Data Eng. Bull. 41, 4 (2018), 39–45.
- Nick Hynes, D. Sculley, and Michael Terry. 2017. The Data Linter: Lightweight, automated sanity checking for ML data sets. In Proceedings of the NIPS MLSys Workshop.
- Matthias Feurer, Aaron Klein, Katharina Eggensperger, Jost Springenberg, Manuel Blum, and Frank Hutter. 2015. Efficient and robust automated machine learning. In Advances in Neural Information Processing Systems, C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Garnett (Eds.), Vol. 28. Curran Associates, Inc. Retrieved from https://proceedings.neurips.cc/paper/2015/file/11d0e6287202fced83f79975ec59a3a6-Paper.pdf.
- Randal S. Olson and Jason H. Moore. 2016. TPOT: A tree-based pipeline optimization tool for automating machine learning. In Proceedings of the Workshop on Automatic Machine Learning. PMLR, 66–74.
- Stephan Rabanser, Stephan Günnemann, and Zachary Chase Lipton. 2019. Failing loudly: An empirical study of methods for detecting dataset shift. In Proceedings of the Neural Information Processing Systems Foundation Conference.
- Chelsea Finn, Pieter Abbeel, and Sergey Levine. 2017. Model-agnostic meta-learning for fast adaptation of deep networks. In Proceedings of the International Conference on Machine Learning. PMLR, 1126–1135.
- A. Zujus. 2018. AI project development–how project managers should prepare. Tratto il giorno Agosto 31 (2018), 2019.
Cited by
Items 501–600 of 737
Fang, Z., & Huang, Y. (2024) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3686990 · Cited by 1
Aiding the resilience of cooperation through the use of network rankings
Miles, A. L., Cavaliere, M., & Yang, G. (2024) · PLoS ONE · doi:10.1371/journal.pone.0313198
“…the context of such a experimentation model, identifying the intentions of individuals could prove difficult and would likely require significant resources in order to aid with the classification of individuals, which could ultimately prove inaccurate over time as circumstances shift [13, 14].”
Coordination of Damping Controllers: A Novel Data-Informed Approach for Adaptability
Zelaya-Arrazabal, F., Pulgar-Painemal, H. A., Liu, J., Silva-Saravia, H., & Li, F. F. (2024) · Electric Power Systems Research · doi:10.1016/j.epsr.2024.111351
“In general, the deployment workflow considers different main steps [28].”
BackgroundMachine Learning in X-ray Scattering for Materials Discovery and CharacterizationPreprint
Davel, C. M., Bassiri‐Gharb, N., & Correa‐Baena, J. (2024) · ChemRxiv · doi:10.26434/chemrxiv-2024-x8fx0-v2
Systematic Planning and Early-Stage Development of Industrial AI Systems for Plant Optimization
Voigt, J., Gillmann, M. J., & Dröder, K. (2024) · Zukunftstechnologien für den multifunktionalen Leichtbau · doi:10.1007/978-3-658-45889-8_26
Generative Artificial Intelligence and Machine Learning for Identity and Access Management
Liveretos, A., & Lazarova, M. K. (2025) · Proceedings of the Technical University of Sofia · doi:10.47978/tus.2024.74.03.019
الجمل, ز. ش. ح. ا., الفنجري, ل., & Fahmy, H. (2025) · النشرة المعلوماتية في الحاسبات والمعلومات · doi:10.21608/fcihib.2025.261807.1105
Qi, Y. (2024) · International Conference on Power, Electrical Engineering, Electronics and Control (PEEEC) · doi:10.1109/peeec63877.2024.00028
Elnaggar, O., Paoletti, P., Hopkinson, A., & Coenen, F. (2025) · Elsevier eBooks · doi:10.1016/b978-0-443-22206-1.00004-8 · Cited by 1
Mamoru, D. L. S., & Ifthikar, A. (2024) · International Conference on Information Technology Research (ICITR) · doi:10.1109/icitr64794.2024.10857795
Using Text Mining to Identify Employee Dissatisfaction Optimized by Modified Metaheuristics
Mizdraković, V., Jovanović, L., Zivkovic, M. Z., Antonijević, M., Kaljević, J., & Bačanin, N. (2025) · Algorithms for intelligent systems · doi:10.1007/978-981-97-7946-8_1
FROM DATA TO DECISION: SIMPLIFYING FINTECH ML MODELS WITH AWS SAGEMAKER
Mada, L. (2025) · INTERNATIONAL JOURNAL OF INFORMATION TECHNOLOGY AND MANAGEMENT INFORMATION SYSTEMS · doi:10.34218/ijitmis_16_01_040
“SageMaker's built-in monitoring capabilities help financial institutions maintain model performance and comply with regulatory requirements [15].”
InfluentialTHE TECHNOLOGY STACK FUELING LARGE-SCALE PRODUCTION MACHINE LEARNING
Maheshwari, C. (2025) · INTERNATIONAL JOURNAL OF INFORMATION TECHNOLOGY AND MANAGEMENT INFORMATION SYSTEMS · doi:10.34218/ijitmis_16_01_056
Automating Data Quality Monitoring In Machine Learning Pipelines
Vijayan, N. E. (2023) · ESP International Journal of Advancements in Computational Technology · doi:10.56472/25838628/ijact-v1i3p111
Tire-Wear Estimation System Using Semantic Segmentation and Its Deployment
Kang, M. J., Kwon, K. B., & Koo, H. I. (2025) · IEEE Access · doi:10.1109/access.2025.3557381
“Fortunately, nowadays, it is widely accepted that training and deploying a model all at once (without iteration) is practically impossible, and continuous maintenance and updates based on user feedback are essential [9], [17].”
On the Challenges of Implementing MLOps for Stream Learning Algorithms
Rodrigues, M. G., Viegas, E. K., Enembreck, F., Santin, A. O., Langaro, J. S., & Filho, A. G. (2025) · Lecture notes on data engineering and communications technologies · doi:10.1007/978-3-031-87769-8_33
(2023) · International Journal of Multidisciplinary Research and Growth Evaluation · doi:10.54660/.ijmrge.2023.4.2.649-651
Bahuleyan, H., Puzikov, Y., Koriagin, E., Lasserre, J., Weffer, R., & Shirvany, R. (2024) · International Conference on Software and Information Engineering (ICSIE) · doi:10.1145/3708635.3708636
Key Challenges and Limitations of MLOps in Context of Machine Learning
Sribhashyam, V. A. S. (2025) · INTERANTIONAL JOURNAL OF SCIENTIFIC RESEARCH IN ENGINEERING AND MANAGEMENT · doi:10.55041/ijsrem46346
Costas, J., Pastor, R., Gerbolés, C. G.-C., & Díez, R. P. (2025) · Communications in computer and information science · doi:10.1007/978-3-031-86623-4_37 · Cited by 1
Kaithe, B. K. (2025) · World Journal of Advanced Engineering Technology and Sciences · doi:10.30574/wjaets.2025.15.2.0624
APPLICATION OF VGG16 ARCHITECTURE IN WOOD TYPE CLASSIFICATION USING CONVOLUTIONAL NEURAL NETWORK
Afiah, N. A., Syahrullah, S., Ardiansyah, R., Laila, R., & Pohontu, R. (2025) · Jurnal Teknik Informatika (Jutif) · doi:10.52436/1.jutif.2025.6.1.3874
CHALLENGES IN DEPLOYING LSTMS AND BLACK-BOX MODELS FOR DIVERSIFICATION – A THEORETICAL APPROACH
Mayada, C. (2025) · International Journal of Education, Business and Economics Research. · doi:10.59822/ijeber.2025.5315
Bringing Machine Learning Models Beyond the Experimental Stage with Explainable AI
Mikkelsen, N. W., Pedersen, L. S., Hussain, M., Foged, V., & Kindler, E. (2025) · IEEE/ACM International Conference on AI Engineering – Software Engineering for AI (CAIN) · doi:10.1109/cain66642.2025.00022
“Many articles focus on what makes ML different from standard software engineering [3], [6], [21].”
Enhancing industrial machine vision interpretability through saliency maps fusion
Xu, X., Gan, X., Zhou, T., Zhang, Y., & Mo, J. (2025) · Signal Image and Video Processing · doi:10.1007/s11760-025-04394-z
“Deep learning neural network models have been impeded by the opacity of their data-driven inference mechanisms and complex architectures [18 ].”
Yu, J., & Qi, H. (2025) · International Conference on Computer and Communication Systems (ICCCS) · doi:10.1109/icccs65393.2025.11069659
“By keeping data localized and sharing only model updates, FL ensures privacy preservation while reducing communication overhead [3].”
Ángeles, M. D. P., Barrios-Lopez, F., Baeza, J. Z., & Cruz, D. C. G. (2025) · Research Square · doi:10.21203/rs.3.rs-7084709/v1
Designing a Recommendation System for Optimizing E-commerce User Experiences
Goyal, T., & Brahma, B. (2025) · Lecture notes in networks and systems · doi:10.1007/978-981-96-3355-5_11
Rastakhiz, F. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5346208 · Cited by 1
Comparative Study in Fault Detection under Data Scarcity for Pharmaceutical Device Manufacturing
Kakavandi, F., Han, P., Nerenst, T. B., de Reus, R., Gomes, C. Â. G., & Larsen, P. G. (2025) · International Conference on Control, Automation and Diagnosis (ICCAD) · doi:10.1109/iccad64771.2025.11099417
“Insufficient data quality and volume significantly impact the performance of complex ML models, compromising their precision and reliability, particularly in fault detection for industrial processes [4].”
Pivoting Towards user-Centric Implementation of Business Analytics in Information SystemsPreprint
Sobrie, L., Roets, B., Verschelde, M., & Triantis, K. P. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5383812
An Approach for Integrated Development of an MLOps Architecture
Heck, P., Snoeren, J., Veracx, M., & Peeters, M. W. H. (2025) · Lecture notes in computer science · doi:10.1007/978-3-032-02138-0_1
Shetty, M. P., Praveena, S., Manivelmuralidaran, V., Shetty, S. B., Anupriya, R., & Senthilkumar, P. (2025) · De Gruyter eBooks · doi:10.1515/9783112205198-006
Edge AI-Based LSTM Voice-Controlled Mobile Robot for Industrial Pipe Inspection
Kavazović, M., & Banjanović-Mehmedović, L. (2025) · International Conference on Information, Communication and Automation Technologies (ICAT) · doi:10.1109/icat66432.2025.11189277 · Cited by 1
“[10], [11].”
MLOps for Medical Imaging: A Cloud-Edge architecture for Experiment Tracking and Model Evaluation
Saim, Akram, S. U., Harjula, E., Lovén, L., & Mehmood, H. (2025) · IEEE International Conference on Network Protocols (ICNP) · doi:10.1109/icnp65844.2025.11192460
AI-Enhanced Medical Devices for Smarter Healthcare
Bhimavarapu, U. (2025) · IGI Global eBooks · doi:10.4018/979-8-3373-1514-0.ch006
Machine Learning Based Security Situation Awareness and Prediction of New Energy Grid
Wang, X., Gu, J., Yang, W., Cao, L., Zhang, J., & Ma, Z. (2025) · IEEE World Conference on Applied Intelligence and Computing (AIC) · doi:10.1109/aic66080.2025.11212172
“By constructing multiple decision trees and integrating their predictions, it effectively improves the accuracy of the model and reduces the risk of overfitting [13-14].”
AI-Powered Cloud-Native Enterprise Applications: A Framework for Enterprise Digital Transformation
Veeravalli, P. K. (2025) · Journal of Computational Analysis and Applications · doi:10.48047/jocaaa.2025.34.10.11
Lead conversion challenges and marketing education strategies in the Ed-tech industry
Ibrahim, A. F. M. (2025) · International Journal of Research in Marketing Management and Sales · doi:10.33545/26633329.2025.v7.i2e.309
Automated machine learning for maritime accident analysis: A case study
Beljadid, M., Boukous, H. A., Beljadid, A., & Tannouche, A. (2025) · International Conference on Optimization and Applications (ICOA) · doi:10.1109/icoa66896.2025.11236882
“However, these techniques comes with a price which is having significant expertise in data modeling and analysis, that price can be an obstacle for many organizations [4].”
AdaLite: A Distilled AdaBins Model for Depth Estimation on Resource-Limited Devices
Ziara, M. C., Elbahri, M., Taleb, N. C.-E., Kpalma, K., & Ardjoun, S. A. E. M. (2025) · AI · doi:10.3390/ai6110298
“This reflects a broader pattern in deep learning, where models achieving state-of-the-art accuracy often prove impractical for real-world use due to practical pitfalls [11].”
Boukhobza, J., Burrello, A., Chang, Y.-H., Li, Y., Pagliari, D. J., Wu, C.-F., Yang, M.-C., & Yang, T.-Y. (2025) · International Conference on Compilers, Architecture, and Synthesis for Embedded Systems (CASES) · doi:10.1145/3742872.3758334
“DNN models are notoriously compute and memory intensive, which complicates their deployment at the edge [53].”
REMEDI: Relative Feature Enhanced Meta-Learning with Distillation for Imbalanced Prediction
Liu, F., & Chen, Y. (2025) · International Conference on Computer Information Science and Artificial Intelligence (CISAI) · doi:10.1145/3773365.3773403
“This creates a fundamental tension between model complexity (required for high performance) and deployment feasibility [8].”
Rocha, L., Rêgo, P. A. L., Trinta, F. A. M., Santos, E., Mattos, C., Souza, K., Almendra, C., Macedo, M. A. D. S., Carvalho, I., Nobrega, J., et al. (2025) · Simpósio Brasileiro de Engenharia de Software (SBES) · doi:10.5753/sbes.2025.11719
Порівняльна оцінка стратегій навчання моделей трансформерів у публічних хмарних платформах
Т., Kopiika, О., & Hryshchak, D. (2025) · COMPUTER-INTEGRATED TECHNOLOGIES EDUCATION SCIENCE PRODUCTION · doi:10.36910/6775-2524-0560-2025-61-17
Research on Large Model Fine-Tuning and Edge Deployment Mechanisms in Power Scenarios
Huang, Y., Yang, Y., Lin, J., Jiang, Z., & Li, R. (2025) · International Conference on Power System and Power Engineering (PSPE) · doi:10.1109/pspe66589.2025.11296397
“However, existing deep learning models generally suffer from problems such as high computational resource consumption [6] and difficulty in deployment [7], which are especially prominent in edge devices [8].”
A Formal Model for Feature Store Architecture and Governance Sivaramakrishnan Vaidyanathan
Vaidyanathan, S. (2025) · International Journal of Computational and Experimental Science and Engineering · doi:10.22399/ijcesen.4555
A Real-Time Multi-Task Deep Learning Framework For Smart Water Quality Prediction
Helaly, M. A., Rady, S., Morsey, M. M., & Aref, M. M. (2025) · International journal of intelligent computing and information sciences/International Journal of Intelligent Computing and Information Sciences · doi:10.21608/ijicis.2025.442640.1438
Coordinated physics-integrated machine learning and its application in tunnel engineering
Liu, S., Ma, M., Su, C., Zhang, Q., & Kang, Y. (2026) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2025.113698 · Cited by 1
Autonomic Service Mesh Using Machine Learning Operations
Gomez, M. A., Bhatia, A., Vadhyar, A., Mohammed, S. A., & Prajwal, S. (2026) · Lecture notes in networks and systems · doi:10.1007/978-981-96-9048-0_10
Integration of MLOps into the Software Development Lifecycle: A Systematic Review
Cancañón-Rodríguez, K., Muñoz, M., & Avila-George, H. (2025) · Mexican International Conference on Computer Science · doi:10.1109/enc68268.2025.11311934 · Cited by 3
Towards Self-learning AI-Based Quality Rules for Statistical Process Control
Dechert, M., Horozal, F., Scholze, S., Wabo, M., & Kotipalli, R. (2026) · Communications in computer and information science · doi:10.1007/978-3-032-12478-4_15
Challenges of Attack Automation in AI Models: Toward a Red Teaming Strategy
Chaitou, H., Bou‐Chaaya, K., Njaka, H. A., & Rais, H. (2025) · IEEE/ACS International Conference on Computer Systems and Applications (AICCSA) · doi:10.1109/aiccsa66935.2025.11315268
“These variations can lead to transient vulnerabilities or unforeseen misclassifica-tions, complicating security assurance efforts [17].”
From Experimentation To Enterprise Reality: Why Mlops Is The Backbone Of Production AI
Bade, S. (2026) · Journal of International Crisis and Risk Communication Research · doi:10.63278/jicrcr.vi.3563
“Model cards capture intended use cases, training data characteristics, and known limitations [8].”
InfluentialOptimizing Cloud Data Costs: FinOps and Usage-Based Workload Segmentation Strategies
Boggavarapu, V. (2026) · Journal of Information Systems Engineering & Management · doi:10.52783/jisem.v11i1s.14037
“Organizations frequently underestimate expenses associated with feature engineering, model validation, and continuous retraining cycles [11].”
Machine Learning–Based Communication and Network Automation
Revathy, J. S., & Mangaiyarkkarasi, J. (2026) · Wiley eBooks · doi:10.1002/9781394238118.ch9
Interpretable multimodal reasoning for robo-advisory: the FinErva framework
Chi, J. (2026) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2025.1752580 · Cited by 1
Generative AI Models to Revolutionize Data Engineering: Technical Perspective in Data Transformation
Kunchakuri, N., Pachigolla, Y. R., Samudrala, R., & Sinha, A. R. (2025) · International Conference on Information Systems and Computer Networks (ISCON) · doi:10.1109/iscon65210.2025.11341146
“Data engineering also plays a significant role in the capabilities and performance of Gen AI models, and consists of planning, building, and maintaining data infrastructure [7].”
Role of Generative AI in Data Engineering to Build Scalable Models in Cloud Technology
Kunchakuri, N., Samudrala, R., Patchigolla, V. L., & Sinha, A. R. (2025) · International Conference on Information Systems and Computer Networks (ISCON) · doi:10.1109/iscon65210.2025.11341238
“The capabilities and performance of Gen AI models are significantly shaped by data engineering, which includes the planning, building, and upkeep of data infrastructure [6].”
Condori-Luna, G. F., Oliveros, D. A. V., & Reis, M. S. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-15984-7_27
Opt-AEDDM: toward optimizing autoencoders for effective concept drift detection
Ali, U., & Mahmood, T. (2026) · Knowledge and Information Systems · doi:10.1007/s10115-025-02664-3
Probabilistic Runtime Verification, Evaluation, and Risk Assessment of Visual Deep Learning Systems
Torpmann-Hagen, B., Halvorsen, P., Riegler, M. A., & Johansen, D. (2026) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3795523
“In particular, we further discuss the prevalence of generalization failure and its effects, how current approaches to the assessment and verification of DNN fail to capture these shortcomings, and outline related work on OOD detection and distributional shift.”
Transforming Knowledge Management Through Synergistic Artificial Intelligence–Human Collaboration
Dawarka, V., & Huzooree, G. (2026) · Emerald Group Publishing Limited eBooks · doi:10.1108/978-1-83662-778-420261006
Muhisn, Z. A. A., Boujelben, Y., & Zarai, F. (2026) · Annals of Telecommunications · doi:10.1007/s12243-026-01159-8
When Systems Are Uncertain: Human Reliability Engineering in AI-Driven Production
Kaimal, S. V. (2026) · Journal of Information Systems Engineering & Management · doi:10.52783/jisem.v11i2s.14385
Federl, A., Böhmisch, M., Sagstetter, V., Fischerauer, G., & Bösnecker, R. (2026) · Electronics · doi:10.3390/electronics15040852
Scalable Data Mining Algorithm for Predictive Healthcare Analytics
Shinde, S. G., Kharade, K. G., & Kamat, R. K. (2025) · International Conference on Science Technology Engineering and Mathematics (ICONSTEM) · doi:10.1109/iconstem65670.2025.11374473
“Local and global explanations based on Large Impact Models (LIME) and SHAP offer an explanation of a prediction for a complex model for a clinician [8].”
Kumar, G., & Varshney, N. (2025) · International Conference on Electronics and Computing, Communication Networking Automation Technologies (ICEC2NT) · doi:10.1109/icec2nt65402.2025.11379942
“In [8] highlighted the difficulty of deploying DL models in clinical environments due to latency, interpretability, and hardware limitations.”
Pamula, G. K. (2026) · IEEE International Conference on AI in Cybersecurity (ICAIC) · doi:10.1109/icaic67076.2026.11395806
“The last step in achieving the MMF was to integrate governance into the CI/CD process using an automated data quality and bias validation process at various points along the way [7].”
Scicluna, A., Paul, G., Kirchner, N., Wendel, A., & Virgona, A. (2025) · IEEE International Conference on Robotics and Biomimetics (ROBIO) · doi:10.1109/robio66223.2025.11376337
“In-situ field testing, where systems are evaluated on operational machinery in real-world environments [21], achieves end-to-end and black-box assessment but is fundamentally not reproducible.”
Adaptive Data-Centric Artificial Intelligence for Robust Decision-Making in Dynamic Environments
., M., Azhar, M., & Chan, K. L. (2026) · Journal of Data Sciences · doi:10.61453/jods.v20260101
Hybrid Inference Pipelines for Spam Detection: A Reproducible Case Study in MLOps
Bharti, V. (2026) · IEEE Annual Computing and Communication Workshop and Conference (CCWC) · doi:10.1109/ccwc67433.2026.11393849
“Subsequent research emphasized observability and drift monitoring [1], [5], [6], addressing metrics, logs, and data validation in production.”
Peng, Z., Xu, Y., Zhang, S., & Yang, Y. (2026) · Neural Networks · doi:10.1016/j.neunet.2026.108709
Kaimal, S. V. (2026) · Computer Fraud & Security · doi:10.52710/cfs.947
“Distributed tracing can be applied to a request throughout feature engineering, model scoring, and post-processing to diagnose issues more specific to ML [14].”
A Study on Machine Learning Models Performance Analysis in Smart Home Appliances
Karrolla, B., Kar, U. N., & Das, R. K. (2025) · OITS International Conference on Information Technology (OCIT) · doi:10.1109/ocit66168.2025.11399953
Automated Machine Learning (AutoML) for Scalable AI Solutions
Pawar, P. P., Raghavendra, G., Joshi, S., Reddy, L. C. S., Kumar, B. V., & Priyanka, T. P. (2025) · International Conference on Computational Intelligence and Network Systems (CINS) · doi:10.1109/cins67018.2025.11411969
“AutoML strives to take a complex process and make it easier for non-experts to be able to develop an acceptable ML model while also enabling experts to more rapidly iterate through experimentation and deployment [4].”
Chikhalkar, S. (2026) · Computer Fraud & Security · doi:10.52710/cfs.957
Chidum, P., & AMADI, A. O. (2026) · International Journal of Research Publication and Reviews · doi:10.55248/gengpi.07.0226.0352
An Integrated Deep Learning Framework for Trading Card Grading: Design, Deployment, and Evaluation
Nahar, L., Islam, M. S., Awrangjeb, M., & Verhoeve, R. (2025) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata66926.2025.11401677
“In another literature, Paleyes et al. [20] advocates for integrating user feedback into deployment loops, which can be implemented by capturing and injecting expert decisions directly into our pipeline.”
Esteban, A., & Calvache, C. D. J. P. (2026) · Inge CUC · doi:10.17981/ingecuc.22.1.2026.03
Proposing a Framework for Machine Learning Adoption on Legacy Systems
Rahman, A., & Alhoori, H. (2025) · IEEE ... International Conference on Data Mining workshops · doi:10.1109/icdmw69685.2025.00194
“These major concerns surrounding data management, re-training, and security necessitate further research to streamline the deployment of ML models in production [24].”
Dmello, A. A. (2026) · Journal of Information Systems Engineering & Management · doi:10.52783/jisem.v11i2s.14479
“…systems involves navigating numerous challenges that extend far beyond algorithm development, including issues related to configuration management, data dependencies, model serving infrastructure, and the continuous monitoring required to detect when models begin to fail in production settings [2].”
MTSC-Transformer: early-age mass concrete temperature field reconstruction using sparse sensor dataPreprint
Meng, Y., Yao, C., Yang, M., Zhou, X., Shen, T., & Wang, C. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.6401882
A Practical Illustration of the Machine Learning Workflow Using a Multiple-Choice Question Dataset
Adeel, M., & Al-Sultan, S. (2026) · Springer eBooks · doi:10.1007/978-3-319-31816-5_4521-1
Bias by Design? A Lifecycle Analysis of Bias in Air Quality Data
Hussain, E., Li, Y., & Ahad, M. A. R. (2025) · IEEE International Conference on Advances in Data-Driven Analytics and Intelligent Systems (ADACIS) · doi:10.1109/adacis65663.2025.11436650
SpaceFusion++: An operator fusion scheduler for neural language model inference
Zhu, L., Yao, J., & Guan, H. (2026) · Journal of Systems Architecture · doi:10.1016/j.sysarc.2026.103708
An Efficient Deployment of Deep Learning LSTM Models using Docker and Kubernetes (Minikube)
Rathor, K., Singh, H., & Thakral, P. (2026) · International Conference on Signal Processing and Electronics Design (ICSPED) · doi:10.1109/icsped65817.2026.11448373
“These frameworks all impose high computational requirements and come with sometimes complex runtime environments that are often hard to work with, leading to issues with replicability, scalability, and just running consistent experiments on single computational units [4], [5].”
Adeel, M., & Al-Sultan, S. (2025) · Springer eBooks · doi:10.1007/978-3-319-31816-5_4520-1
Machine learning tools in the 21st century: a historical review
Primero-Huerta, C., & Sánchez-DelaCruz, E. (2026) · Evolutionary Intelligence · doi:10.1007/s12065-026-01176-5
Daniel, S. (2026) · IgMin Research · doi:10.61927/igmin336
Financial Data Acquisition Method Based on Bayesian Model and Machine Learning
Si, B. X. (2025) · International Conference on Information Technology, Communication Ecosystem and Management (ITCEM) · doi:10.1109/itcem68692.2025.00035
“Algorithms Due to the high accuracy and adaptability of RF in ensemble learning to high-dimensional data, this study chose the RF algorithm for research [13,14].”
Mohammadi, M., & Esfahanian, V. (2026) · Energy · doi:10.1016/j.energy.2026.140838
Garcia, M. A. J., & Herrera, R. D. J. G. (2026) · International Journal of Business Analytics · doi:10.4018/ijban.406288
Gemidi, A. S. K. (2026) · Journal of Computational Analysis and Applications · doi:10.48047/jocaaa.2026.35.04.03
Optimisation of Energy Consumption of Data Center using Artificial Intelligence
Grill, L. (2023) · theses.fr (ABES) · doi:10.70675/375e59d6z5ae2z422cz9673z43d7d3b9b5b1
Decentralized Disease Prediction: A Federated Learning Perspective
Demir, A., Kulaksız, A. Y., Büyüktanır, B., Baydoğmus, G. K., & Yıldız, K. (2026) · Afyon Kocatepe University Journal of Sciences and Engineering · doi:10.35414/akufemubid.1679417
Raman, K., Joshi, S., Soman, V., & Aror, D. (2024) · International Journal of AI BigData Computational and Management Studies · doi:10.63282/3050-9416.ijaibdcms-v5i4p123
Chawla, M., Mukherjee, A., Prasad, L. J., Shetty, N., & Sharma, R. (2024) · International Journal of Emerging Trends in Computer Science and Information Technology · doi:10.63282/3050-9246.ijetcsit-v5i3p117
Selecting test cases to reveal defects in recurrent neural networks
Xu, H., Song, X., Sun, H., Yu, H., & Zhu, Z. (2026) · Neurocomputing · doi:10.1016/j.neucom.2026.133673
Metadata
- Authors
- Paleyes, A., Urma, R.-G., & Lawrence, N. D.
- Affiliations
- University of Cambridge (GB)
- Cambridge Spark, Cambridge, United Kingdom
- Venue
- ACM Computing Surveys
- Published
- 2022-12-07
- Volume
- 55
- Issue
- 6
- Pages
- 1-29
- ISSN
- 0360-0300 (ISSN-L)
- 1557-7341
- OpenAlex
- W3104128335
- Semantic Scholar
- b1f574c47d0b6e3032246770b9cbebb9c7bd0c7f
- Cited by
- 638
- References
- 128
- Citations by year
- 2026: 149 · 2025: 156 · 2024: 166 · 2023: 93 · 2022: 46
- FWCI
- 114.17
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- ACM Computing Surveys (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- White Rose Research Online (University of Leeds, The University of Sheffield, University of York) (submittedVersion)
- Apollo (University of Cambridge) (submittedVersion)
- arXiv (Cornell University)
- Apollo
- Funders
- Alan Turing Institute
- UK Research & Innovation
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
- Physical Sciences › Computer Science › Artificial Intelligence › Artificial Intelligence Applications
- Physical Sciences › Computer Science › Information Systems › Cloud Computing and Resource Management
- Keywords
- machine learning deployment, production systems, deployment challenges, industry case studies
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.