Challenges in Deploying Machine Learning: A Survey of Case Studies
Open accessThe DOI matches its Crossref record; 127 of the 128 references it cites were found in a registry or an index.
Abstract
In recent years, machine learning has transitioned from a field of academic research interest to a field capable of solving real-world business problems. However, the deployment of machine learning models in production systems can present a number of issues and concerns. This survey reviews published reports of deploying machine learning solutions in a variety of use cases, industries, and applications and extracts practical considerations corresponding to stages of the machine learning deployment workflow. By mapping found challenges to the steps of the machine learning deployment workflow, we show that practitioners face issues at each stage of the deployment process. The goal of this article is to lay out a research agenda to explore approaches addressing these challenges.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“Compared with the first step, continual monitoring is used to identify data distribution drift, control model quality, and trigger system alarms when an ML model deviates from its normal behavior in the target setting [15].”
From Toward real‐world deployment of machine learning for health care: External validation, continual monitoring, and randomized clinical trials (2024) · cites this paper as [15]
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“Our literature review was conducted using a systematic mapping protocol [54] in order to select a small set of relevant articles from the much larger collection of literature emerging at the intersection of data quality and ML.”
From A Survey of Data Quality Requirements That Matter in ML Development Pipelines (2023) · cites this paper as [54]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325 · cited 2×
“They took one of the state-of-the-art models in the field, BERT [47], and found out that depending on the chosen model size full training procedure can cost anywhere between $50k and $1.”
BackgroundQuinlan, J. R. (1986) · Machine Learning · doi:10.1007/bf00116251 · Cited by 21,439 · cited 1×
“For that reason decision trees (DT) [108], which can be considered a fairly basic ML algorithm, are widely used in practice.”
MethodArmstrong, J. S., & Georgoff, D. M. (1988) · International Journal of Forecasting · doi:10.1016/0169-2070(88)90019-2 · Cited by 17,894
Rudin, C. (2019) · Nature Machine Intelligence · doi:10.1038/s42256-019-0048-x · Cited by 10,561 · cited 1×
“Rudin [111] called the ML community to stop using black box models and explaining their behavior afterward, and instead design models that are inherently interpretable.”
BackgroundOn the Dangers of Stochastic Parrots: Can Language Models Be Too Big? 🦜
Bender, E. M., Gebru, T., McMillan-Major, A., & Shmitchell, S. (2021) · Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3442188.3445922 · Cited by 6,988
Fully homomorphic encryption using ideal lattices
Gentry, C. (2009) · Proceedings of the forty-first annual ACM symposium on Theory of computing · doi:10.1145/1536414.1536440 · Cited by 6,806 · cited 1×
“Some of the ways this trade-of is resolved in practice are diferential privacy that explicitly corrupts the data [42], homomorphic encryption that restricts the class of learning algorithm but allows for training on encrypted data [50], and federated learning that distributes training across personal devices to preser…”
BackgroundPractical Bayesian Optimization of Machine Learning Algorithms
Snoek, J., Larochelle, H., & Adams, R. P. (2012) · arXiv · doi:10.48550/arxiv.1206.2944 · Cited by 6,691 · cited 1×
“Even approaches like Hyperband [78] or Bayesian optimization [129], that are speciically designed to minimize the number of training cycles needed, are not yet able to deal with the high dimensional searches that emerge when many hyper-parameters are involved.”
Dwork, C. (2006) · Lecture Notes in Computer Science · doi:10.1007/11787006_1 · Cited by 5,291 · cited 1×
“Some of the ways this tradeoff is resolved in practice are differential privacy that explicitly corrupts the data [54], homomorphic encryption that restricts the class of learning algorithm but allows for training on encrypted data [55], and federated learning that distributes training across personal devices to prese…”
BackgroundMembership Inference Attacks Against Machine Learning Models
Shokri, R., Stronati, M., Song, C., & Shmatikov, V. (2017) · 2017 IEEE Symposium on Security and Privacy (SP) · doi:10.1109/sp.2017.41 · Cited by 4,754 · cited 1×
“They veriied it on models trained on leading ML-as-a-service providers [127], achieving anywhere from 70% up to 94% accuracy of membership inference.”
BackgroundA survey on concept drift adaptation
Gama, J., Žliobaitė, I., Bifet, A., Pechenizkiy, M., & Bouchachia, A. (2014) · ACM Computing Surveys · doi:10.1145/2523813 · Cited by 3,735 · cited 1×
“We refer interested readers to the works on using domain adaptation [48], meta-learning [46] and transfer learning [152, 157] as ways of addressing dataset shift.”
Method
Research works printed but not linked: 26
- Arif Cam, Michael Chui, and Bryce Hall. 2019. Global AI survey: AI proves its worth, but few scale impact. McKinsey Analytics (2019). https://www.mckinsey.com/featured-insights/artificial-intelligence/global-ai-survey-ai-proves-its-worth-but-few-scale-impact. Accessed from 13/06/2022.
- Royal Society (Great Britain). 2017. Machine Learning: The Power and Promise of Computers that Learn by Example: an Introduction. Royal Society.
- Ben Lorica and Nathan Paco. 2018. The State of Machine Learning Adoption in the Enterprise. O’Reilly Media.
- Colin Shearer. 2000. The CRISP-DM model: The new blueprint for data mining. J. Data Warehous. 5, 4 (2000), 13–22.
- Rajashree Y. Patil and R. V. Kulkarni. 2012. A review of data cleaning algorithms for data warehouse systems. Int. J. Comput. Sci. Inf. Technol. 3, 5 (2012), 5212–5214.
- 10.1109/TITS.2019.2962338 · DOI · Paper page
- Diego Charrez. 2019. NeurIPS 2019 Stats. Retrieved from https://medium.com/@dcharrezt/neurips-2019-stats-c91346d31c8f.
- 10.1023/A:1022643204877 · doi:10.1023/a:1022643204877 · Paper page
- 10.5555/3122009.3242042 · doi:10.5555/3122009.3242042
- Bobak Shahriari, Alexandre Bouchard-Cote, and Nando Freitas. 2016. Unbounded Bayesian optimization via regularization. In Proceedings of the 19th International Conference on Artificial Intelligence and Statistics (Proceedings of Machine Learning Research), Arthur Gretton and Christian C. Robert (Eds.), Vol. 51. PMLR, 1168–1176. Retrieved from http://proceedings.mlr.press/v51/shahriari16.html.
- Ram Ananth, Seph Mard, and Peter Simon. 2019. Opening the “Black Box”: The path to deployment of AI models in banking, white paper. DataRobot (2019). https://www.datarobot.com/resources/opening-the-black-box/. Accessed from 13/06/2022.
- Robert Soden Dennis Wagenaar Dave Luo and Annegien Tijssen. 2019. Taking Ethics Fairness and Bias Seriously in Machine Learning for Disaster Risk Management. arxiv:cs.CY/1912.05538.
- Nantheera Anantrasirichai and David Bull. 2021. Artificial intelligence in the creative industries: A review. Artif. Intell. Rev. (2021), 1–68.
- D. Malan et al. 2016. Values and the fourth industrial revolution: Connecting the dots between value, values, profit and purpose. In Proceedings of the World Economic Forum.
- Daniel Malan. 2018. The law can’t keep up with new tech. Here’s how to close the gap. In Proceedings of the World Economic Forum.
- Ramprakash Ramamoorthy P. Satya Madhuri and Malini Christina Raj. 2019. AI from labs to production—challenges and learnings. USENIX Association. https://www.usenix.org/conference/opml19/presentation/madhuri. Accessed from 13/06/2022.
- Oscar Schwartz. 2019. In 2016 Microsoft’s racist chatbot revealed the dangers of online conversation. IEEE Spectr. 11 (2019).
- Florian Tramèr, Fan Zhang, Ari Juels, Michael K. Reiter, and Thomas Ristenpart. 2016. Stealing machine learning models via prediction APIs. In Proceedings of the 25th USENIX Security Symposium (USENIX Security’16). 601–618.
- Jeremy Hermann and Mike Del Balso. 2017. Meet Michelangelo: Uber’s machine learning platform. Retrieved from https://eng.uber.com/michelangelo.
- Matei Zaharia, Andrew Chen, Aaron Davidson, Ali Ghodsi, Sue Ann Hong, Andy Konwinski, Siddharth Murching, Tomas Nykodym, Paul Ogilvie, Mani Parkhe, et al. 2018. Accelerating the machine learning lifecycle with MLflow.IEEE Data Eng. Bull. 41, 4 (2018), 39–45.
- Nick Hynes, D. Sculley, and Michael Terry. 2017. The Data Linter: Lightweight, automated sanity checking for ML data sets. In Proceedings of the NIPS MLSys Workshop.
- Matthias Feurer, Aaron Klein, Katharina Eggensperger, Jost Springenberg, Manuel Blum, and Frank Hutter. 2015. Efficient and robust automated machine learning. In Advances in Neural Information Processing Systems, C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Garnett (Eds.), Vol. 28. Curran Associates, Inc. Retrieved from https://proceedings.neurips.cc/paper/2015/file/11d0e6287202fced83f79975ec59a3a6-Paper.pdf.
- Randal S. Olson and Jason H. Moore. 2016. TPOT: A tree-based pipeline optimization tool for automating machine learning. In Proceedings of the Workshop on Automatic Machine Learning. PMLR, 66–74.
- Stephan Rabanser, Stephan Günnemann, and Zachary Chase Lipton. 2019. Failing loudly: An empirical study of methods for detecting dataset shift. In Proceedings of the Neural Information Processing Systems Foundation Conference.
- Chelsea Finn, Pieter Abbeel, and Sergey Levine. 2017. Model-agnostic meta-learning for fast adaptation of deep networks. In Proceedings of the International Conference on Machine Learning. PMLR, 1126–1135.
- A. Zujus. 2018. AI project development–how project managers should prepare. Tratto il giorno Agosto 31 (2018), 2019.
Cited by
Items 301–400 of 737
Machine Learning for Internet of Things: Applications and Discussions
Park, H., Song, S., Nguyen, T., & Park, L. (2024) · International Conference on Artificial Intelligence in Information and Communication (ICAIIC) · doi:10.1109/icaiic60209.2024.10463332 · Cited by 6
“To handle heterogeneous data, machine learning (ML) has attracted increasing interest in handling large-scale and high-dimensional data [5].”
BackgroundCao, H., Finer, J., Megjhani, M., Nametz, D. C., Lorenzi, V., Mamykina, L., Meyers, R., Rossetti, S. C., & Park, S. (2022) · IEEE Healthcare Innovations and Point of Care Technologies Conference (HI-POCT) · doi:10.1109/hi-poct54491.2022.9744076 · Cited by 4
Detection of outliers in classification by using quantified uncertainty in neural networks
Malmström, M., Skog, I., Axehill, D., & Gustafsson, F. (2022) · International Conference on Information Fusion (FUSION) · doi:10.23919/fusion49751.2022.9841376 · Cited by 4
Visual explanation and robustness assessment optimization of saliency maps for image classification
Xu, X., & Mo, J. (2022) · The Visual Computer · doi:10.1007/s00371-022-02715-8 · Cited by 4
“Usually, end-users tend to be more skeptical of CNN-based applications than pattern-based machine vision solutions [3].”
BackgroundSelf-sustaining Software Systems (S4): Towards Improved Interpretability and Adaptation
Cabrera, C., Paleyes, A., & Lawrence, N. D. (2024) · doi:10.1145/3643657.3643910 · Cited by 9
“Deployed systems must respond to changing goals, variable data, unexpected failures, and security threats, among other variables that emerge from demanding requirements and real-world environments [24].”
Self-citationLe, T.-D., Noumeir, R., Rambaud, J., Sans, G., & Jouvet, P. A. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2104.03934 · Cited by 5
“Unfortunately, not all cases can apply deep learning, especially with limited data [16].”
BackgroundExplanatory Model Monitoring to Understand the Effects of Feature Shifts on Performance
Decker, T., Koebler, A., Lebacher, M., Thon, I., Tresp, V., & Buettner, F. (2024) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3637528.3671959 · Cited by 13
Lifecycle Models in Machine Learning Development
Crespí, A., Mesquida, A., Monserrat, M., & Mas, A. (2025) · Expert Systems · doi:10.1111/exsy.70029 · Cited by 6
Luo, T., Qi, L., Yang, J. J., Li, J., Liu, Q., Du, Y.-Y., Wang, T., & Fang, G. (2025) · Optics Express · doi:10.1364/oe.566641 · Cited by 3
Liu, S., Ma, M., Su, C., Zhang, Q., & Kang, Y. (2025) · Measurement · doi:10.1016/j.measurement.2025.118718 · Cited by 3
The Feature Stores in Streamlining MLOps Workflows
Nunes, C. E. R., & Ashofteh, A. (2025) · IT Professional · doi:10.1109/mitp.2025.3532129 · Cited by 5
NExNet Seg: Neuron Expansion Network for Medical Image Segmentation
Reyes-Angulo, A. A., & Paheding, S. (2025) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw67362.2025.00005 · Cited by 3
“High computational costs and the need for extensive parameter tun-2 ing continue to hinder real-world applicability, particularly in low-resource environments [25].”
Developing empirical indices for structural engineering problems via machine learning
Naser, M. Z., & Hawileh, R. A. (2025) · Engineering Structures · doi:10.1016/j.engstruct.2025.121609 · Cited by 3
Şahin, E. K., Demir, S., Ozturk, M., & Duzce, M. S. (2025) · Advances in Space Research · doi:10.1016/j.asr.2025.11.048 · Cited by 3
Sayour, M. B. A., Naranje, V. G., & Phanden, R. K. (2025) · Management & Sustainability An Arab Review · doi:10.1108/msar-03-2025-0086 · Cited by 3
Machine Learning for Improving Educational Outcomes in Primary and Secondary Schools
Vardhan, A., Saini, S., Sharma, S., & Jhingran, A. (2026) · Springer eBooks · doi:10.1007/978-981-95-4925-2_8 · Cited by 3
Padial-Fuillerat, E., Martínez-Manjón, J. E., Zwir, I. J. S., Arroyo, M. J., Blázquez‐Sánchez, M., Rodríguez‐Temporal, D., Rodríguez, B. M., Mancera, L., & del Val, C. (2026) · Journal of Proteome Research · doi:10.1021/acs.jproteome.5c00534 · Cited by 5
WHY ML MODELS FAIL TO REACH PRODUCTION
Oliveira, O. O. L. P. (2026) · Journal International Review of Research Studies · doi:10.66104/wbexse86 · Cited by 5
Industrial MLOps: a systematic review of architectures and implementation challenges
Rajashekarappa, M., Bekar, E. T., Karlsson, A., Bokrantz, J., Subramaniyan, M., & Skoogh, A. (2026) · Production & Manufacturing Research · doi:10.1080/21693277.2026.2658878 · Cited by 4
“Scholarly work increasingly highlights the significance of AI and ML in modern business practices, as more industries begin to leverage these technologies to address practical challenges (Lawrence et al., 2022).”
Ouedraogo, R. S., McCloskey, P., Mwaipopo, B. V., Timilsena, B. P., Kiruwa, F. H., Kemboi, M. J., Shimwela, M. M., & Hughes, D. P. (2026) · Food and Energy Security · doi:10.1002/fes3.70273 · Cited by 4
Kim, D., Ahmed, S. S., Amjad, A., Won, K., & Xian, X. (2026) · Biosensors · doi:10.3390/bios16060344 · Cited by 6
“When datasets are small or privacy prevents broad data sharing, simpler models such as linear or logistic regression and tree-based ensembles tend to be more stable and easier to calibrate and explain [38,39].”
Mathematical and Statistical Frameworks Fostering Advances in AI Systems and Computing
Oliveira, T., & Teodoro, M. F. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-65223-3_16 · Cited by 3
Case Studies and Applications of Generative AI in Real-World Cybersecurity Scenarios
Khan, A., Jhanjhi, N. Z., Alı, G. A., Ray, S. K., & Wassan, S. (2024) · Advances in human and social aspects of technology book series · doi:10.4018/979-8-3693-8939-3.ch003 · Cited by 3
A Survey of Intrusion Detection Systems Based On Deep Learning for IoT Data
Moudi, M., Soleimani, A., & nia, A. H. (2024) · Journal of Information Systems and Telecommunication (JIST) · doi:10.61186/jist.44521.12.47.197 · Cited by 5
“Recently, machine learning, like artificial intelligence, has been interested in university research and has been able to solve problems in real-world businesses largely [15].”
Boosting total electron content forecasting based on deep learning toward an operational service
Molina, M. G., Namour, J. H., Cesaroni, C., Spogli, L., Argüelles, N. B., & Asamoah, E. N. (2025) · Journal of Atmospheric and Solar-Terrestrial Physics · doi:10.1016/j.jastp.2025.106427 · Cited by 4
Ren, Y., He, Y., Zhang, X., Yen, A. H., & Li, G. P. (2025) · Machines · doi:10.3390/machines13040304 · Cited by 3
Two-stage ensemble learning framework for automated classification of keratoconus severity
Muhsin, Z. J., Qahwaji, R., Ghafir, I., AlShawabkeh, M., Bdour, M. D. A., AlRyalat, S. A., & Al-Taee, M. A. (2025) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2025.110568 · Cited by 4
Effective ML Model Versioning in Edge Networks
Gentzen, F., Bensalem, M., & Jukan, A. (2025) · International Wireless Communications and Mobile Computing Conference (IWCMC) · doi:10.1109/iwcmc65282.2025.11059571 · Cited by 3
“The field of AI and ML has gone through unprecedented advancements over the past decade, whereby AI/ML models are continuously improving in size, performance, robustness and accuracy [1].”
MFFN: Multiscale Feature Fusion Network with Pruning for photovoltaic cell anomaly detection
He, Z., Chen, G., Ruan, Z., & Wang, L. (2025) · Nondestructive Testing And Evaluation · doi:10.1080/10589759.2025.2533387 · Cited by 3
“Additionally, the large model parameters of existing solutions hinder their deployment on mobile devices, creating a barrier to widespread practical application [35].”
Rubo, R. A., Michelon, M. F., & Carneiro, C. D. C. (2022) · Earth Science Informatics · doi:10.1007/s12145-022-00901-9 · Cited by 4
“Poor quality of data and small datasets are usually observed (Paleyes et al. 2022).”
BackgroundSaliency Guided Debiasing: Detecting and mitigating biases in LMs using feature attribution
Joshi, R. K., Chatterjee, A., & Ekbal, A. (2023) · Neurocomputing · doi:10.1016/j.neucom.2023.126851 · Cited by 4
Resource Allocation of Large-Scale Communication Network based on Machine Learning
Zhang, Z., Mo, R., Hou, X., Wei, X., & Guo, S. (2024) · International Conference on Integrated Circuits and Communication Systems (ICICACS) · doi:10.1109/icicacs60521.2024.10498186 · Cited by 3
“In wireless communication networks, cooperation between nodes is an important issue [14-15].”
BackgroundMIDAS: Multi-layered attack detection architecture with decision optimisation
Rendall, K., Mylonas, A., Vidalis, S., & Gritzalis, D. A. (2024) · Computers & Security · doi:10.1016/j.cose.2024.104154 · Cited by 4
“The work in (Paleyes et al., 2023) presents challenges with resource-constrained environments, suggesting that one should consider the practicalities of deploying deep learning to networked systems that have very limited energy, memory, and data transmission.”
Zhao, Y. (2024) · International Conference on Intelligent Algorithms for Computational Intelligence Systems (IACIS) · doi:10.1109/iacis61494.2024.10721756 · Cited by 3
“Through a series of case studies, Paleyes A investigated the challenges in machine learning deployment, including data quality, model interpretation, computing resources and ethical issues, and analyzed the impact of these challenges on machine learning applications [15].”
BackgroundChinnaraju, A. (2024) · Journal of Emerging Technologies and Innovative Research · doi:10.56975/jetir.v11i11.559859 · Cited by 4
HP_FLAP: homomorphic and polymorphic federated learning aggregation of parameters framework
Moshawrab, M., Adda, M., Bouzouane, A., Ibrahim, H., & Raad, A. (2025) · Cybersecurity · doi:10.1186/s42400-024-00341-6 · Cited by 4
Fundamentals of Evolutionary Machine Learning
Banzhaf, W., & Machado, P. (2023) · Genetic and evolutionary computation · doi:10.1007/978-981-99-3814-8_1 · Cited by 7
Malmström, M., Skog, I., Axehill, D., & Gustafsson, F. (2023) · IFAC-PapersOnLine · doi:10.1016/j.ifacol.2023.10.077 · Cited by 3
A Review of the Internet of Things and Deep Learning in Agriculture: A Smart Agriculture Prespective
Mirani, A. A., & Azeem Ayaz Mirani, S. Z. (2023) · Journal of Hunan University Natural Sciences · doi:10.55463/issn.1674-2974.50.10.18 · Cited by 4
Software Defect Prediction For Quality Evaluation Using Learning Techniques Ensemble Stacking
Kusuma, M. R., Gata, W., Kurniawan, S., Saputra, D. D., & Panggabean, S. (2023) · Inspiration Jurnal Teknologi Informasi dan Komunikasi · doi:10.35585/inspir.v13i2.58 · Cited by 3
Yang, L., Cheng, J., Luo, Y., Zhou, T., Zhang, X., Shi, L., & Xu, Y. (2024) · The Canadian Journal of Chemical Engineering · doi:10.1002/cjce.25373 · Cited by 4
Study of the Theory and Practice of Educational Supervision in Islamic Education in Indonesia
Musnandar, A., Sutrisno, S., Jima’ain, M. T. A., Prabowo, G. C. W., & Subhi, N. D. N. (2024) · Lingeduca Journal of Language and Education Studies · doi:10.70177/lingeduca.v3i3.1360 · Cited by 4
With False Friends Like These, Who Can Notice Mistakes?
Tao, L., Feng, L., Yi, J., & Chen, S. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i8.20822 · Cited by 7
“Figure 1(a) illustrates the machine learning (ML) workflow (Paleyes, Urma, and Lawrence 2020; Paterson, Calinescu, and Ashmore 2021), the process of developing an ML-based solution in an industrial setting.”
BackgroundGit Workflow for Active Learning: A Development Methodology Proposal for Data-Centric AI Projects
Stieler, F., & Bauer, B. (2023) · International Conference on Evaluation of Novel Approaches to Software Engineering · doi:10.5220/0011988400003464 · Cited by 3
Neural forecasting at scalePreprint
Chatigny, P., Wang, S., Patenaude, J.-M., & Oreshkin, B. N. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2109.09705 · Cited by 5
“Indeed, the deployment of a reliable DNN with an automatic training procedure is far more challenging because of this cost and other factors such as optimal architecture and hyperparameter tuning which various authors discused in previous studies [10, 11].”
Towards better data discovery and collection with flow-based programmingPreprint
Paleyes, A., Cabrera, C., & Lawrence, N. D. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2108.04105 · Cited by 5
“However, machine learning poses a new set of challenges that the majority of existing software architectures are not designed for [Paleyes et al., 2020].”
BackgroundSelf-citationML-EXray: Visibility into ML Deployment on the EdgePreprint
Qiu, H., Vavelidou, I., Li, J., Pergament, E., Warden, P., Chinchali, S., Asgar, Z., & Katti, S. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2111.04779 · Cited by 5
“Our work looks beyond the output and provides visibility inside the model, and focuses on the system issues of edge deployment (Paleyes et al., 2021).”
BackgroundAdinkra Symbol Recognition Using Classical Machine Learning and Deep LearningPreprint
Asamoah, K. O., Adjeisah, M., Yeboah, M. A., Raji, R. K., Achaab, G. F., & Adjei, K. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.4836682 · Cited by 4
Challenges in Building Predictive Models
Nayak, R., Rajaramesh, C., & Ghugar, U. (2024) · Wiley eBooks · doi:10.1002/9781394227990.ch2 · Cited by 3
The Penalized Inverse Probability Measure for Conformal Classification
Melki, P., Bombrun, L., Diallo, B., Dias, J., & da Costa, J. (2024) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw63382.2024.00355 · Cited by 6
Artificial intelligence and machine learning for foot and ankle disorders
Gupta, S. B., Kour, H., & Antony, P. (2024) · Elsevier eBooks · doi:10.1016/b978-0-443-32892-3.00013-0 · Cited by 3
Industrial Maturity of Machine Learning Solutions Within the Food Industry
Gradl, L., Reis, L., & Buettner, R. (2025) · IEEE Access · doi:10.1109/access.2025.3558091 · Cited by 2
“Therefore, this step is demanded and seen as necessary by the current state of research [14], however the results in this study, confirmed by other studies [175], show that the implementation of this requirement has so far only been realised to a limited extent.”
InfluentialAn Exploratory Study of Security Vulnerabilities in Machine Learning Deployment Projects
Rahman, A., Skjellum, A., & Zhang, Y. (2025) · doi:10.1109/svm66695.2025.00009 · Cited by 3
“Deployment of ML models can occur using on-prem servers, cloud-based infrastructure, or edge devices [11].”
Vella, D. D., Senarath, Y., Hashmi, E., Abomhara, M., Shaikh, S., Igland, T., Dralega, C. A., Yayilgan, S. Y., Purohit, H., & Akerkar, R. A. (2025) · Digital Society · doi:10.1007/s44206-025-00212-8 · Cited by 2
“Significant efforts have been made to address the challenges encountered when deploying ML systems in real-world scenarios (Paleyes et al., 2022; Papagiannidis et al., 2021 ).”
Adaptive Sacrifice for QoS-Aware Routing: A Graph Reinforcement Learning Approach
Song, Y., Zhou, J.-L., Yu, S.-D., & Liu, J. (2026) · IEEE Transactions on Mobile Computing · doi:10.1109/tmc.2025.3607388 · Cited by 1
“More critically, training agents directly in production networks is nearly infeasible due to safety concerns and pro-hibitive costs [21].”
Ng, Y. J. X., Cheong, S. H. R., Ang, W. W., Lau, Y., & Lau, S. T. (2025) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2025.103274 · Cited by 2
Toward a Secure Framework for Regulating Artificial Intelligence Systems
Elahi, H., Liu, N., Chen, J., & Zhang, F. (2025) · IEEE Transactions on Dependable and Secure Computing · doi:10.1109/tdsc.2025.3616288 · Cited by 4
“3) Regulatory Evolution: It is anticipated that the regulatory framework surrounding AI will continue to evolve in response to emerging AI capabilities and feedback [22].”
Zheng, X., Huang, Y., Xie, Z., & Zheng, A. (2025) · Scientific Reports · doi:10.1038/s41598-025-24399-5 · Cited by 2
“With the increasing diversity of ecological perception data, machine learning has become an important tool for feature identification 30 .”
Lahilote, J. J., Putrada, A. G., & Wicaksono, R. L. (2025) · International Conference on Information Technology Research and Innovation (ICITRI) · doi:10.1109/icitri67507.2025.11232988 · Cited by 2
“Paleyes et al. [16] from Cambridge University analyze the practical challenges in implementing a machine learning-based forecasting system.”
Machine learning for software aging detection: A systematic mapping study
Moura, R. J., Nascimento, M. G., Machida, F., Cotroneo, D., & Andrade, E. (2025) · Journal of Systems and Software · doi:10.1016/j.jss.2025.112715 · Cited by 2
Zhao, Y., Srivastava, G., & Ullah, F. (2025) · IEEE Internet of Things Journal · doi:10.1109/jiot.2025.3642146 · Cited by 2
“Edge devices present additional challenges due to their limited processing power, memory, and battery capacity [5].”
Quantum Machine Learning (QML): Variational Classifiers, Quantum Kernels and Hybrid Architectures
Vengathattil, S. (2025) · World Journal of Advanced Research and Reviews · doi:10.30574/wjarr.2025.28.1.3416 · Cited by 2
“Training deep models (as high dimensional parameters) and processing the massive dataset within reasonable time limits are bottlenecks (Paleyes, Urma, & Lawrence, 2022).”
Reference model for preventive software maintenance
Chupryna, A., & Repikhov, V. (2025) · Management Information System and Devises · doi:10.30837/0135-1710.2025.187.254 · Cited by 2
“The first systematic review of the challenges of maintaining ML systems was published only in 2022 [16], which indicates the novelty and insufficient study of the topic.”
Machine Learning-Driven Cyber Defense: Enhancing U.S. Critical Infrastructure Resilience
Jabed, M. M. I., Sarwar, J., Afrin, S., & Gupta, A. B. (2026) · International Journal of Innovative Science and Research Technology (IJISRT) · doi:10.38124/ijisrt/26jan1061 · Cited by 3
Wang, N., & Leem, S.-G. (2026) · IntechOpen eBooks · doi:10.5772/intechopen.1013232 · Cited by 4
Buhai, D., Zhuchenko, A., Жученко, О. А., Ковалюк, Д. О., & Skladannyy, D. M. (2026) · Technology audit and production reserves · doi:10.15587/2706-5448.2026.352435 · Cited by 2
Qin, X., Luan, S., See, J., Boukhers, Z., Yang, C., & Li, Z. (2026) · Information and Software Technology · doi:10.1016/j.infsof.2026.108337 · Cited by 10
“…progressive delivery stage exposure across cohorts (Governor, 2018); MLOps pipelines add offline validation and staged rollout for learned models (Paleyes, Urma and Lawrence, 2022; Ashmore, Calinescu and Paterson, 2021; Sculley, Holt, Golovin, Davydov, Phillips, Ebner, Chaudhary, Young, Crespo…”
Wu, Y., Zhu, J., Yuan, J., & Huang, K. (2026) · Computational Biology and Chemistry · doi:10.1016/j.compbiolchem.2026.108991 · Cited by 2
Kumar, P., Verma, S., & Kaur, R. (2026) · Catalysis Today · doi:10.1016/j.cattod.2026.115861 · Cited by 2
Sharma, K., Sharma, P., & Sharma, P. (2026) · Indian Heart Journal · doi:10.1016/j.ihj.2026.06.002 · Cited by 4
Gonçalves, H. O., Amaral, R., & Pereira, J. C. (2026) · Risk Analysis · doi:10.1111/risa.70297 · Cited by 2
Operational evidence standards for machine learning in wastewater treatment
Jiang, S., Yang, Y., Zhang, K., Mao, Y., & Cheng, X. (2026) · npj Clean Water · doi:10.1038/s41545-026-00610-6 · Cited by 2
OpenSLU: A Unified, Modularized, and Extensible Toolkit for Spoken Language Understanding
Qin, L., Chen, Q., Xiao, X., Feng, Y., & Che, W. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-demo.9 · Cited by 6
Data Engineering Challenges in AI automation
Reddy, D. (2023) · International Conference on Computing, Electronics & Communications Engineering (iCCECE) · doi:10.1109/iccece59400.2023.10238496 · Cited by 3
“The issues related to data management for AI applications, particularly the provision of high-quality data to the AI models at scale, have been recognized as one of the key challenges [74].”
BackgroundRumour Detection in the Wild: A Browser Extension for Twitter
Jovanović, A., & Roß, B. (2023) · Workshop for NLP Open Source Software (NLP-OSS) · doi:10.18653/v1/2023.nlposs-1.15 · Cited by 2
“These results confirm findings of Horne et al. (2019); Paleyes et al. (2022); Lobo et al. (2020), and show the importance of retraining schedules and engines when deploying machine learning models in browser extension tools (Thi-lakarathna et al., 2020).”
ResultElsayad, A. M., Mubarak, F., Abdullah, H., Fahhad, M., & Saad, N. (2024) · National Radio Science Conference · doi:10.1109/nrsc61581.2024.10510535 · Cited by 2
“However, challenges persist in passive landmine detection and classification with machine learning, including data imbalance, limited data availability, and model interpretability [6].”
BackgroundA Prototype for Machine Learning Model Deployment in Cloud Environment
Ramalakshmi, K., Agarwal, A., Gunasekaran, H., Khan, S. A., & Mandal, N. K. (2023) · International Conference on New Frontiers in Communication, Automation, Management and Security (ICCAMS) · doi:10.1109/iccams60113.2023.10526026 · Cited by 2
“The details will then be sent to the database to check the user details [10].”
BackgroundFathi, K., & van de Venn, H. W. (2024) · Industrial engineering and management. · doi:10.5772/intechopen.1005511 · Cited by 2
Post deployment recycling of machine learning models: Don’t Throw Away Your Old Models!
Patel, H. J., Adams, B., & Hassan, A. E. (2024) · Empirical Software Engineering · doi:10.1007/s10664-024-10492-2 · Cited by 2
Assessing Air-Interface Dataset Similarity and Diversity for AI-Enabled Wireless Communications
Tang, H., Yang, L., Zhou, R., Liang, J., Hong, W., Wang, X., Shi, Q., & Luo, Z.-Q. T. (2024) · IEEE International Conference on Communications workshops · doi:10.1109/iccworkshops59551.2024.10615485 · Cited by 2
“Training AI models for wireless communication systems is a complex task due to the challenges in obtaining suitable training datasets [5], [6].”
BackgroundEvolution of Maintenance Processes in Industry 4.0- A Review
Afolalu, S. A., Nwankwo, S. O., & Azeez, T. M. (2024) · International Conference on Science, Engineering and Business for Driving Sustainable Development Goals (SEB4SDG) · doi:10.1109/seb4sdg60871.2024.10629886 · Cited by 3
“It was thought that since environmental deterioration was getting worse, it needed to be fixed to be safer and last longer [45] 2.4 The role of predictive maintenance in industry 4.0 and its advantages over traditional approach In Interdisciplinary investigations groups, predictive maintenance…”
BackgroundInfluentialApplying Random Forests in Federated Learning: A Synthesis of Aggregation Techniques
Bodynek, M., Leiser, F., Thiebes, S., & Sunyaev, A. (2025) · Lecture notes in information systems and organisation · doi:10.1007/978-3-031-80119-8_25 · Cited by 2
Teaching Machine Learning as Part of Agile Software Engineering
Chenoweth, S., & Linos, P. K. (2025) · IEEE Transactions on Education · doi:10.1109/te.2025.3572355 · Cited by 2
“In2020Paleyes et al. [11] summarized surveys of the deployment of ML systems.”
Edge-based Machine Learning Models in Iot Devices for Improved Anomaly and Intrusion Detection
Kindong, T., & Iqbal, S. (2025) · International Conference on Cryptography, Security and Privacy (CSP) · doi:10.1109/csp66295.2025.00029 · Cited by 2
“Data collection and preparation for subsequent machine-learning procedures are the initial and most labor-intensive steps in a machine-learning workflow [18,19].”
Global Explanations for Multivariate time series models
Arya, V., Saha, D., Hans, S., Rajasekharan, A., & Tang, T. (2023) · Joint International Conference on Data Science & Management of Data (CODS-COMAD) · doi:10.1145/3570991.3570998 · Cited by 4
“Additionally, different stakeholders including domain practitioners, model developers, regulators, and impacted users require different types of explanations [2, 7, 23, 31].”
BackgroundAmorim, B. F., Costa, L., Souza, F. C. M., & Souza, A. C. C. (2023) · Anais do Computer on the Beach · doi:10.14210/cotb.v14.p379-386 · Cited by 3
AI Management Model for Production
Heymann, H., Hellmich, J. H., Frye, M., Grunert, D., & Schmitt, R. H. (2023) · IMEKO TC10 Conference on Measurement for Diagnostics, Optimisation and Control · doi:10.21014/tc10-2023.002 · Cited by 2
Facilitating Model Training With Automated Techniques
Mursa, B.-E.-M., Kuti-Kreszács, M., Moroz-Dubenco, C., & Bota, F. (2023) · Studia Universitatis Babeș-Bolyai Informatica · doi:10.24193/subbi.2023.2.04 · Cited by 2
Artificial Intelligence in the Public Sector in Shanghai: Conceptualization, Context, and Capacity
Todaro, D. (2024) · Springer eBooks · doi:10.1007/978-981-97-0597-9_2 · Cited by 5
DIGITAL TRANSFORMATION IN INDUSTRY 4.0: A BIBLIOMETRIC ANALYSIS
Sopi, J. M., & Hanafi, M. H. (2024) · International Journal of Innovation and Industrial Revolution · doi:10.35631/ijirev.616002 · Cited by 2
Omar, R., Bogner, J., Leest, J., Stoico, V., Lago, P., & Muccini, H. (2024) · International Conference on ICT for Sustainability (ICT4S) · doi:10.1109/ict4s64576.2024.00026 · Cited by 7
“However, the successful development and especially the long-term operation of ML-enabled systems requires considerable expertise due to many challenges [3, 4].”
Nevin, J., Lees, M., & Groth, P. (2025) · PeerJ Computer Science · doi:10.7717/peerj-cs.2793 · Cited by 4
“…as there are many costs associated with the retraining, retesting, and redeployment of the models to production ( Wiggers, 2019 ; Hecht, 2019 ; Paleyes, Urma & Lawrence, 2022 ), with costs for ML projects performed by ITRex, for example, varying between $26,000 and $300,000 ( Shashkina, 2024…”
Towards Hybrid Architectures for Big Data Analytics: Insights From Spark-MPI Integration
Zhou, M., Li, Q., Cai, M., Xu, C., & Wang, Y. (2025) · IEEE Transactions on Services Computing · doi:10.1109/tsc.2025.3562342 · Cited by 2
Comparative Analysis of Open Standards for Machine Learning Model Deployments
Singh, S., Singh, N. K., & Singh, V. K. (2022) · Lecture notes in networks and systems · doi:10.1007/978-981-16-5987-4_51 · Cited by 2
Towards A Platform and Benchmark Suite for Model Training on Dynamic Datasets
Böther, M., Strati, F., Gsteiger, V., & Klimovic, A. (2023) · Workshop on Machine Learning and Systems (EuroMLSys) · doi:10.1145/3578356.3592585 · Cited by 7
“[52] Andrei Paleyes, Raoul-Gabriel Urma, and Neil D.”
BackgroundTowards Feature-Based Analysis of the Machine Learning Development Lifecycle
Hu, B. C., & Chećhik, M. (2023) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3611643.3613082 · Cited by 3
Development and Deployment Challenges of Machine Learning Systems
Honkanen, T., O’Dwyer, J., Salminen, V., & Jussila, J. (2024) · Communications in computer and information science · doi:10.1007/978-3-031-64359-0_15 · Cited by 2
Human Guidance Approaches for the Genetic Improvement of Software
Craine, B., Rainford, P. F., & Porter, B. (2024) · doi:10.1145/3643692.3648263 · Cited by 2
“HG is a technique to bias algorithms to produce more desirable outputs, and is being explored in multiple generative and synthesis fields [4, 11–13] to bridge shortcomings in various machine learning (ML) applications [5, 8].”
MethodRamadhan, A., Mendonça, I., Aritsugi, M., & Chandra, I. (2024) · International Conference on Wireless and Telematics (ICWT) · doi:10.1109/icwt62080.2024.10674689 · Cited by 3
“Machine learning, a subset of artificial intelligence, offers a promising solution [15]-[16].”
BackgroundDew Computing in Industrial Automation: Applying Machine Learning for Process Control
Šverko, M., Tanković, N., & Etinger, D. (2021) · IEEE Annual International Computer Software and Applications Conference (COMPSAC) · doi:10.1109/compsac51774.2021.00268 · Cited by 2
“Cloud ML platform performs data management, model learning, model verification, and model deployment [14].”
Metadata
- Authors
- Paleyes, A., Urma, R.-G., & Lawrence, N. D.
- Affiliations
- University of Cambridge (GB)
- Cambridge Spark, Cambridge, United Kingdom
- Venue
- ACM Computing Surveys
- Published
- 2022-12-07
- Volume
- 55
- Issue
- 6
- Pages
- 1-29
- ISSN
- 0360-0300 (ISSN-L)
- 1557-7341
- OpenAlex
- W3104128335
- Semantic Scholar
- b1f574c47d0b6e3032246770b9cbebb9c7bd0c7f
- Cited by
- 638
- References
- 128
- Citations by year
- 2026: 149 · 2025: 156 · 2024: 166 · 2023: 93 · 2022: 46
- FWCI
- 114.17
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- ACM Computing Surveys (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- White Rose Research Online (University of Leeds, The University of Sheffield, University of York) (submittedVersion)
- Apollo (University of Cambridge) (submittedVersion)
- arXiv (Cornell University)
- Apollo
- Funders
- Alan Turing Institute
- UK Research & Innovation
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
- Physical Sciences › Computer Science › Artificial Intelligence › Artificial Intelligence Applications
- Physical Sciences › Computer Science › Information Systems › Cloud Computing and Resource Management
- Keywords
- machine learning deployment, production systems, deployment challenges, industry case studies
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.