Challenges in Deploying Machine Learning: A Survey of Case Studies
Open accessThe DOI matches its Crossref record; 127 of the 128 references it cites were found in a registry or an index.
Abstract
In recent years, machine learning has transitioned from a field of academic research interest to a field capable of solving real-world business problems. However, the deployment of machine learning models in production systems can present a number of issues and concerns. This survey reviews published reports of deploying machine learning solutions in a variety of use cases, industries, and applications and extracts practical considerations corresponding to stages of the machine learning deployment workflow. By mapping found challenges to the steps of the machine learning deployment workflow, we show that practitioners face issues at each stage of the deployment process. The goal of this article is to lay out a research agenda to explore approaches addressing these challenges.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“Compared with the first step, continual monitoring is used to identify data distribution drift, control model quality, and trigger system alarms when an ML model deviates from its normal behavior in the target setting [15].”
From Toward real‐world deployment of machine learning for health care: External validation, continual monitoring, and randomized clinical trials (2024) · cites this paper as [15]
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“Our literature review was conducted using a systematic mapping protocol [54] in order to select a small set of relevant articles from the much larger collection of literature emerging at the intersection of data quality and ML.”
From A Survey of Data Quality Requirements That Matter in ML Development Pipelines (2023) · cites this paper as [54]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325 · cited 2×
“They took one of the state-of-the-art models in the field, BERT [47], and found out that depending on the chosen model size full training procedure can cost anywhere between $50k and $1.”
BackgroundQuinlan, J. R. (1986) · Machine Learning · doi:10.1007/bf00116251 · Cited by 21,439 · cited 1×
“For that reason decision trees (DT) [108], which can be considered a fairly basic ML algorithm, are widely used in practice.”
MethodArmstrong, J. S., & Georgoff, D. M. (1988) · International Journal of Forecasting · doi:10.1016/0169-2070(88)90019-2 · Cited by 17,894
Rudin, C. (2019) · Nature Machine Intelligence · doi:10.1038/s42256-019-0048-x · Cited by 10,561 · cited 1×
“Rudin [111] called the ML community to stop using black box models and explaining their behavior afterward, and instead design models that are inherently interpretable.”
BackgroundOn the Dangers of Stochastic Parrots: Can Language Models Be Too Big? 🦜
Bender, E. M., Gebru, T., McMillan-Major, A., & Shmitchell, S. (2021) · Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3442188.3445922 · Cited by 6,988
Fully homomorphic encryption using ideal lattices
Gentry, C. (2009) · Proceedings of the forty-first annual ACM symposium on Theory of computing · doi:10.1145/1536414.1536440 · Cited by 6,806 · cited 1×
“Some of the ways this trade-of is resolved in practice are diferential privacy that explicitly corrupts the data [42], homomorphic encryption that restricts the class of learning algorithm but allows for training on encrypted data [50], and federated learning that distributes training across personal devices to preser…”
BackgroundPractical Bayesian Optimization of Machine Learning Algorithms
Snoek, J., Larochelle, H., & Adams, R. P. (2012) · arXiv · doi:10.48550/arxiv.1206.2944 · Cited by 6,691 · cited 1×
“Even approaches like Hyperband [78] or Bayesian optimization [129], that are speciically designed to minimize the number of training cycles needed, are not yet able to deal with the high dimensional searches that emerge when many hyper-parameters are involved.”
Dwork, C. (2006) · Lecture Notes in Computer Science · doi:10.1007/11787006_1 · Cited by 5,291 · cited 1×
“Some of the ways this tradeoff is resolved in practice are differential privacy that explicitly corrupts the data [54], homomorphic encryption that restricts the class of learning algorithm but allows for training on encrypted data [55], and federated learning that distributes training across personal devices to prese…”
BackgroundMembership Inference Attacks Against Machine Learning Models
Shokri, R., Stronati, M., Song, C., & Shmatikov, V. (2017) · 2017 IEEE Symposium on Security and Privacy (SP) · doi:10.1109/sp.2017.41 · Cited by 4,754 · cited 1×
“They veriied it on models trained on leading ML-as-a-service providers [127], achieving anywhere from 70% up to 94% accuracy of membership inference.”
BackgroundA survey on concept drift adaptation
Gama, J., Žliobaitė, I., Bifet, A., Pechenizkiy, M., & Bouchachia, A. (2014) · ACM Computing Surveys · doi:10.1145/2523813 · Cited by 3,735 · cited 1×
“We refer interested readers to the works on using domain adaptation [48], meta-learning [46] and transfer learning [152, 157] as ways of addressing dataset shift.”
Method
Research works printed but not linked: 26
- Arif Cam, Michael Chui, and Bryce Hall. 2019. Global AI survey: AI proves its worth, but few scale impact. McKinsey Analytics (2019). https://www.mckinsey.com/featured-insights/artificial-intelligence/global-ai-survey-ai-proves-its-worth-but-few-scale-impact. Accessed from 13/06/2022.
- Royal Society (Great Britain). 2017. Machine Learning: The Power and Promise of Computers that Learn by Example: an Introduction. Royal Society.
- Ben Lorica and Nathan Paco. 2018. The State of Machine Learning Adoption in the Enterprise. O’Reilly Media.
- Colin Shearer. 2000. The CRISP-DM model: The new blueprint for data mining. J. Data Warehous. 5, 4 (2000), 13–22.
- Rajashree Y. Patil and R. V. Kulkarni. 2012. A review of data cleaning algorithms for data warehouse systems. Int. J. Comput. Sci. Inf. Technol. 3, 5 (2012), 5212–5214.
- 10.1109/TITS.2019.2962338 · DOI · Paper page
- Diego Charrez. 2019. NeurIPS 2019 Stats. Retrieved from https://medium.com/@dcharrezt/neurips-2019-stats-c91346d31c8f.
- 10.1023/A:1022643204877 · doi:10.1023/a:1022643204877 · Paper page
- 10.5555/3122009.3242042 · doi:10.5555/3122009.3242042
- Bobak Shahriari, Alexandre Bouchard-Cote, and Nando Freitas. 2016. Unbounded Bayesian optimization via regularization. In Proceedings of the 19th International Conference on Artificial Intelligence and Statistics (Proceedings of Machine Learning Research), Arthur Gretton and Christian C. Robert (Eds.), Vol. 51. PMLR, 1168–1176. Retrieved from http://proceedings.mlr.press/v51/shahriari16.html.
- Ram Ananth, Seph Mard, and Peter Simon. 2019. Opening the “Black Box”: The path to deployment of AI models in banking, white paper. DataRobot (2019). https://www.datarobot.com/resources/opening-the-black-box/. Accessed from 13/06/2022.
- Robert Soden Dennis Wagenaar Dave Luo and Annegien Tijssen. 2019. Taking Ethics Fairness and Bias Seriously in Machine Learning for Disaster Risk Management. arxiv:cs.CY/1912.05538.
- Nantheera Anantrasirichai and David Bull. 2021. Artificial intelligence in the creative industries: A review. Artif. Intell. Rev. (2021), 1–68.
- D. Malan et al. 2016. Values and the fourth industrial revolution: Connecting the dots between value, values, profit and purpose. In Proceedings of the World Economic Forum.
- Daniel Malan. 2018. The law can’t keep up with new tech. Here’s how to close the gap. In Proceedings of the World Economic Forum.
- Ramprakash Ramamoorthy P. Satya Madhuri and Malini Christina Raj. 2019. AI from labs to production—challenges and learnings. USENIX Association. https://www.usenix.org/conference/opml19/presentation/madhuri. Accessed from 13/06/2022.
- Oscar Schwartz. 2019. In 2016 Microsoft’s racist chatbot revealed the dangers of online conversation. IEEE Spectr. 11 (2019).
- Florian Tramèr, Fan Zhang, Ari Juels, Michael K. Reiter, and Thomas Ristenpart. 2016. Stealing machine learning models via prediction APIs. In Proceedings of the 25th USENIX Security Symposium (USENIX Security’16). 601–618.
- Jeremy Hermann and Mike Del Balso. 2017. Meet Michelangelo: Uber’s machine learning platform. Retrieved from https://eng.uber.com/michelangelo.
- Matei Zaharia, Andrew Chen, Aaron Davidson, Ali Ghodsi, Sue Ann Hong, Andy Konwinski, Siddharth Murching, Tomas Nykodym, Paul Ogilvie, Mani Parkhe, et al. 2018. Accelerating the machine learning lifecycle with MLflow.IEEE Data Eng. Bull. 41, 4 (2018), 39–45.
- Nick Hynes, D. Sculley, and Michael Terry. 2017. The Data Linter: Lightweight, automated sanity checking for ML data sets. In Proceedings of the NIPS MLSys Workshop.
- Matthias Feurer, Aaron Klein, Katharina Eggensperger, Jost Springenberg, Manuel Blum, and Frank Hutter. 2015. Efficient and robust automated machine learning. In Advances in Neural Information Processing Systems, C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Garnett (Eds.), Vol. 28. Curran Associates, Inc. Retrieved from https://proceedings.neurips.cc/paper/2015/file/11d0e6287202fced83f79975ec59a3a6-Paper.pdf.
- Randal S. Olson and Jason H. Moore. 2016. TPOT: A tree-based pipeline optimization tool for automating machine learning. In Proceedings of the Workshop on Automatic Machine Learning. PMLR, 66–74.
- Stephan Rabanser, Stephan Günnemann, and Zachary Chase Lipton. 2019. Failing loudly: An empirical study of methods for detecting dataset shift. In Proceedings of the Neural Information Processing Systems Foundation Conference.
- Chelsea Finn, Pieter Abbeel, and Sergey Levine. 2017. Model-agnostic meta-learning for fast adaptation of deep networks. In Proceedings of the International Conference on Machine Learning. PMLR, 1126–1135.
- A. Zujus. 2018. AI project development–how project managers should prepare. Tratto il giorno Agosto 31 (2018), 2019.
Cited by
On Predictive Maintenance in Industry 4.0: Overview, Models, and Challenges
Achouch, M., Dimitrova, M., Ziane, K., Sattarpanah Karganroudi, S., Dhouib, R., Ibrahim, H., & Adda, M. (2022) · Applied Sciences · doi:10.3390/app12168081 · Cited by 528
“Compared to traditional enterprise software updates, this cycle of improving predictive maintenance models is much more complex [54].”
MethodModel aggregation techniques in federated learning: A comprehensive survey
Qi, P., Chiaro, D., Guzzo, A., Ianni, M., Fortino, G., & Piccialli, F. (2024) · Future Generation Computer Systems · doi:10.1016/j.future.2023.09.008 · Cited by 452
Technology readiness levels for machine learning systems
Lavin, A., Gilligan-Lee, C. M., Visnjic, A., Ganju, S., Newman, D., Ganguly, S., Lange, D., Baydin, A. G., Sharma, A., Gibson, A., et al. (2022) · Nature Communications · doi:10.1038/s41467-022-33128-9 · Cited by 209
Exploiting BERT for Multimodal Target Sentiment Classification through Input Space Translation
Khan, Z., & Fu, Y. (2021) · Proceedings of the 29th ACM International Conference on Multimedia · doi:10.1145/3474085.3475692 · Cited by 193
“However, the remarkable ability of transformers to adapt without architectural modifications [18], as well as the practical desirability of deploying [26] popular baseline architectures without modifications, has motivated research into exploiting large language models for target/aspect sentiment classification withou…”
Cross-Validation Visualized: A Narrative Guide to Advanced Methods
Allgaier, J., & Pryss, R. (2024) · Machine Learning and Knowledge Extraction · doi:10.3390/make6020065 · Cited by 176
“It is critical to acknowledge, as [20] emphasizes, that reliance on simulation-based testing is constrained by the underlying assumptions of the simulations, and thus cannot fully substitute for empirical testing in real-world scenarios.”
BackgroundSteidl, M., Felderer, M., & Ramler, R. (2023) · Journal of Systems and Software · doi:10.1016/j.jss.2023.111615 · Cited by 126
“[37] identify data collection as a challenge, covering issues regarding related storage location and understanding the data set’s structure.”
BackgroundAn Efficient Optimization Technique for Training Deep Neural Networks
Mehmood, F., Ahmad, S., & Whangbo, T. K. (2023) · Mathematics · doi:10.3390/math11061360 · Cited by 153
Overview of leakage scenarios in supervised machine learning
Sasse, L., Nicolaisen‐Sobesky, E., Dukart, J., Eickhoff, S. B., GOETZ, M. A., Hamdan, S., Komeyer, V., Kulkarni, A. J., Lahnakoski, J. M., Love, B. C., et al. (2025) · Journal Of Big Data · doi:10.1186/s40537-025-01193-8 · Cited by 145
Machine learning applications for electrospun nanofibers: a review
Subeshan, B., Atayo, A., & Asmatulu, E. (2024) · Journal of Materials Science · doi:10.1007/s10853-024-09994-7 · Cited by 140
“Additionally, it is imperative to acknowledge that ML models are inclined to degradation and need regular inspection and maintenance to uphold their performance standards [149].”
Reviewing Federated Machine Learning and Its Use in Diseases Prediction
Moshawrab, M., Adda, M., Bouzouane, A., Ibrahim, H., & Raad, A. (2023) · Sensors · doi:10.3390/s23042112 · Cited by 114
Deepfake Media Forensics: Status and Future Challenges
Amerini, I., Barni, M., Battiato, S., Bestagini, P., Boato, G., Bruni, V., Caldelli, R., De Natale, F., De Nicola, R., Guarnera, L., et al. (2025) · Journal of Imaging · doi:10.3390/jimaging11030073 · Cited by 113
“Leveraging Machine Learning Operation (MLOp) pipelines, future detection models can achieve continuous updates and seamless integration into real-world workflows, enhancing their robustness against evolving threats [156,157].”
Interpretable models for extrapolation in scientific machine learning
Muckley, E. S., Saal, J. E., Meredig, B., Roper, C. S., & Martin, J. H. (2023) · Digital Discovery · doi:10.1039/d3dd00082f · Cited by 134
Shankar, S., Garcia, R., Hellerstein, J. M., & Parameswaran, A. (2024) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3653697 · Cited by 152
Fedstellar: A Platform for Decentralized Federated Learning
Beltrán, E. T. M., Gómez, Á. L. P., Feng, C., Sánchez, P. M. S., Bernal, S. L., Bovet, G., Pérez, M. G., Pérez, G. M., & Celdrán, A. H. (2023) · Expert Systems with Applications · doi:10.1016/j.eswa.2023.122861 · Cited by 103
“With their heavy reliance on centralized data collection and processing, traditional Machine Learning (ML) and Deep Learning (DL) techniques increasingly face privacy, scalability, computational efficiency, and latency issues (Paleyes et al., 2022).”
BackgroundA Survey of Data Quality Requirements That Matter in ML Development Pipelines
Priestley, M., O’donnell, F., & Simperl, E. (2023) · Journal of Data and Information Quality · doi:10.1145/3592616 · Cited by 86
“C l a r i t y o f t h e d a t a p r e - p r o c e ss i n g s e q u e n c e c a n b e i m p r o v e d u s i n g d o c u m e n t a t i o n a n d p u b l i c a t i o n o f c o d e [ 52 , 72 ] .”
BackgroundInfluentialRecent advancements in machine learning enabled portable and wearable biosensors
Kadian, S., Kumari, P., Shukla, S., & Narayan, R. J. (2023) · Talanta Open · doi:10.1016/j.talo.2023.100267 · Cited by 91
Encrypted Network Traffic Analysis and Classification Utilizing Machine Learning
Alwhbi, I. A., Zou, C. C., & Alharbi, R. N. (2024) · Sensors · doi:10.3390/s24113509 · Cited by 81
“…network traffic analysis and classification, it can be deployed in a cloud platform on received real-time monitored data, on the premises if an organization deploys it on its own server due to various concerns, or on edge devices with limited connectivity or for strict real-time requirements [76].”
BackgroundSilvestri, S., Islam, S., Papastergiou, S., Tzagkarakis, C., & Ciampi, M. (2023) · Sensors · doi:10.3390/s23020651 · Cited by 76
“Moreover, the deployment of ML-based systems in critical environments, such as the HCII, further complicates this scenario, due to the difficulties of acquiring and managing the data from the target environment and integrating the ML technologies within the existing HCII systems [24].”
Reviewing Multimodal Machine Learning and Its Use in Cardiovascular Diseases Detection
Moshawrab, M., Adda, M., Bouzouane, A., Ibrahim, H., & Raad, A. (2023) · Electronics · doi:10.3390/electronics12071558 · Cited by 84
“To ensure that the models can cope with the increase in data volume and complexity in the future, it is necessary to develop models that are scalable to enable effective training and deployment, reduce computational costs, and scale the models [25–27”
MethodAI system engineering key challenges and lessons learned
Fischer, L., Ehrlinger, L., Geist, V., Ramler, R., Sobiezky, F., Zellinger, W., Brunner, D., Kumar, M., & Moser, B. (2020) · Machine Learning and Knowledge Extraction · doi:10.3390/make3010004 · Cited by 71
Monitoring machine learning models: a categorization of challenges and methods
Schröder, T., & Schulz, M. (2022) · Data Science and Management · doi:10.1016/j.dsm.2022.07.004 · Cited by 93
A Survey on Data-Driven Scenario Generation for Automated Vehicle Testing
Cai, J., Deng, W., Guang, H., Wang, Y., Li, J., & Ding, J. (2022) · Machines · doi:10.3390/machines10111101 · Cited by 101
Nahar, N., Zhang, H., Lewis, G. A., Zhou, S., & Kästner, C. (2023) · IEEE/ACM International Conference on AI Engineering – Software Engineering for AI (CAIN) · doi:10.1109/cain58948.2023.00034 · Cited by 85
“While some practitioners give talks on challenges or write experience papers (e.g., examples in academic venues surveyed elsewhere [66], [84]), researchers have also been actively studying the challenges faced by practitioners when building software products with ML components across many projects.”
SoK: Pragmatic Assessment of Machine Learning for Network Intrusion Detection
Apruzzese, G., Laskov, P., & Schneider, J. (2023) · IEEE European Symposium on Security and Privacy · doi:10.1109/eurosp57164.2023.00042 · Cited by 116
“Then, the organization must oversee the ML solution for its entire lifecycle, which includes: data collection, preprocessing, and labeling (for both training and testing the ML model); development of the ML model (including repeated testings for parameter calibration); deployment of this ML model in the NIDS infrastru…”
BackgroundMethodArtificial Intelligence in Industry 4.0: A Review of Integration Challenges for Industrial Systems
Windmann, A., Wittenberg, P., Schieseck, M., & Niggemann, O. (2024) · IEEE International Conference on Industrial Informatics (INDIN) · doi:10.1109/indin58382.2024.10774364 · Cited by 82
The Efficiency MisnomerPreprint
Dehghani, M., Arnab, A., Beyer, L., Vaswani, A., & Tay, Y. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.12894 · Cited by 128
“One of the main considerations in designing neural network architectures is quality-cost trade-off (Paleyes et al., 2020).”
BackgroundA participatory data-centric approach to AI Ethics by Design
Gerdes, A. (2021) · Applied Artificial Intelligence · doi:10.1080/08839514.2021.2009222 · Cited by 77
“(Paleyes, Urma, and Lawrence 2021, 14) In this setting, this paper argues that bridge building is essential for establishing strong communication channels and facilitating data exploration activities between AI developers and domain experts.”
BackgroundMachine Learning (In) Security: A Stream of Problems
Ceschin, F., Botacin, M. F., Bifet, A., Pfahringer, B., Oliveira, L. S., Gomes, H. M., & Gregio, A. R. A. (2023) · Digital Threats Research and Practice · doi:10.1145/3617897 · Cited by 64
“The implementation of these approaches might also be challenging [113].”
BackgroundSecurity for Machine Learning-based Software Systems: A Survey of Threats, Practices, and Challenges
Chen, H., & Babar, M. A. (2023) · ACM Computing Surveys · doi:10.1145/3638531 · Cited by 84
“A substantial interest has been attracted to address the development challenges for machine learning-based software systems, which predominantly focus on the aspects of technical debts and quality assurance .”
BackgroundHype, Sustainability, and the Price of the Bigger-is-Better Paradigm in AI
Varoquaux, G., Luccioni, S., & Whittaker, M. (2024) · Conference on Fairness, Accountability and Transparency · doi:10.1145/3715275.3732006 · Cited by 78
“The expense and scarcity of the ingredients needed to build and operate increasingly large models benefits the actors in AI that have access to compute and distribution markets.”
Alansary, S. A., Ayyad, S. M., Talaat, F. M., & Saafan, M. M. (2025) · Knowledge and Information Systems · doi:10.1007/s10115-025-02556-6 · Cited by 65
“As shown in Table 3, there are differences between traditional centralized systems and FL. Traditional ML and DL paradigms are confronted with significant limitations, including privacy concerns, scalability issues, computational inefficiency, and latency bottlenecks [39].”
Tong, Z., Meng, Y., Zhang, J., Wu, Y., Li, Z., Wang, D., Li, X., & Ou, G. (2023) · International Journal of Coal Geology · doi:10.1016/j.coal.2023.104435 · Cited by 47
Large Language Model Supply Chain: A Research Agenda
Wang, S., Zhao, Y., Hou, X., & Wang, H. (2024) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3708531 · Cited by 124
Machine learning operations landscape: platforms and tools
Berberi, L., Kozlov, V. Y., Nguyen, G., Díaz, J. S.-P., Calatrava, A., Moltó, G., Tran, V., & García, Á. L. (2025) · Artificial Intelligence Review · doi:10.1007/s10462-025-11164-3 · Cited by 61
“…and resource consump - tion in the ML pipeline, Moreschini et al. ( 2022 ) proposing a graphical representation for DevOps in ML-based applications, Paleyes et al. ( 2022 ) exploring challenges in the deploy - ment of ML, Barrak et al. ( 2022 ) investigating trends in the use of serverless…”
Analyzing the Evolution and Maintenance of ML Models on Hugging Face
Castaño, J., Martínez‐Fernández, S., Franch, X., & Bogner, J. (2024) · International Conference on Mining Software Repositories (MSR) · doi:10.1145/3643991.3644898 · Cited by 84
“Maintenance tasks, crucial yet challenging [9, 10], involve ensuring operational stability and efficiency through corrections, adjustments, and optimizations, aligning with ISO 25059’s software maintenance standards [11] and interpretations by Rowe et al. [12].”
MethodLeveraging AI for Network Threat Detection—A Conceptual Overview
Paracha, M. A., Jamil, S. U., Shahzad, K., Khan, M. A., & Rasheed, A. (2024) · Electronics · doi:10.3390/electronics13234611 · Cited by 51
Elnour, M., Ahmad, A. M., Abdelkarim, S. B., Fadli, F., & Naji, K. K. (2024) · Building Services Engineering Research and Technology · doi:10.1177/01436244241239290 · Cited by 41
“39 42 From another practical aspect, the implementation process of an AI-based smart manufacturing framework is presented by Shahbazi and Byun.”
Singh, J., Wazid, M., Das, A. K., Chamola, V., & Guizani, M. (2022) · Computer Communications · doi:10.1016/j.comcom.2022.06.012 · Cited by 53
Brintrup, A. M., Baryannis, G., Tiwari, A., Ratchev, S. M., Martínez-Arellano, G., & Singh, J. (2025) · Data-Centric Engineering · doi:10.1017/dce.2025.10032 · Cited by 47
“For example, a deeper investigation showed that Beijing AI Principles show disagreements between Western and non-Western AI principles, despite them using the same terminology (Paleyes et al., 2022).”
ENIDS: A Deep Learning-Based Ensemble Framework for Network Intrusion Detection Systems
Sayem, I. M., Sayed, M. I., Saha, S., & Haque, A. (2024) · IEEE Transactions on Network and Service Management · doi:10.1109/tnsm.2024.3414305 · Cited by 48
“It is now used in every sector, including healthcare, finance, transportation, marketing, cybersecurity, and natural language processing, and is revolutionizing various other fields [26].”
BackgroundAn analysis of the challenges in the adoption of MLOps
Amrit, C., & Narayanappa, A. K. (2024) · Journal of Innovation & Knowledge · doi:10.1016/j.jik.2024.100637 · Cited by 44
Miedema, E., Waschull, S., & Emmanouilidis, C. (2025) · Computers in Industry · doi:10.1016/j.compind.2025.104409 · Cited by 46
Recent Advances in Data Engineering for Networking
Zeydan, E., & Mangues‐Bafalluy, J. (2022) · IEEE Access · doi:10.1109/access.2022.3162863 · Cited by 64
“According to the authors in [183], the main challenges are mainly related to model integration (operational support, code and model reuse, software engineering anti-patterns and mixed team dynamics), model monitoring (feedback loops, outlier detection, custom design tools) and model updating…”
BackgroundBandy, J., & Vincent, N. (2021) · arXiv · doi:10.48550/arxiv.2105.05241 · Cited by 36
Mansoori, A., Zeinalnezhad, M., & Nazarimanesh, L. (2023) · Journal of Healthcare Engineering · doi:10.1155/2023/9673395 · Cited by 37
“Finding the optimal setting of hyperparameters is called hyperparameter optimization (HPO) [22].”
BackgroundWhat Do Users Ask in Open-Source AI Repositories? An Empirical Study of GitHub Issues
Yang, Z., Wang, C., Shi, J., Hoang, T., Kochhar, P. S., Lu, Q., Xing, Z., & Lo, D. (2023) · International Conference on Mining Software Repositories (MSR) · doi:10.1109/msr59073.2023.00024 · Cited by 41
“Paleyes et al. [61] presented a survey of case studies of the challenges and practices in deploying AI systems.”
BackgroundToward Improved Machine Learning-Based Intrusion Detection for Internet of Things Traffic
Alkadi, S., Al-Ahmadi, S. A., & Ismail, M. M. B. (2023) · Computers · doi:10.3390/computers12080148 · Cited by 45
“Examples of hyper-parameters are DT depth, the kernel type of the support vector machine (SVM), and others [40].”
BackgroundAdeniran, I. A., Efunniyi, C. P., Osundare, O. S., & Abhulimen, A. O. (2024) · Computer Science & IT Research Journal · doi:10.51594/csitrj.v5i8.1489 · Cited by 36
Machine Learning for Fraud Detection in E-Commerce: A Research Agenda
Tax, N., de Vries, K. J., de Jong, M., Dosoula, N., van den Akker, B., Smith, J., Thuong, O., & Bernardi, L. (2021) · Communications in computer and information science · doi:10.1007/978-3-030-87839-9_2 · Cited by 49
Machine learning applications in SEM-based pore analysis: a review
Papia, E.-M., Kondi, A., & Constantoudis, V. (2025) · Microporous and Mesoporous Materials · doi:10.1016/j.micromeso.2025.113675 · Cited by 44
Operationalizing machine learning models: a systematic literature review
Kolltveit, A. B., & Li, J. (2022) · Proceedings of the 1st Workshop on Software Engineering for Responsible AI · doi:10.1145/3526073.3527584 · Cited by 34
“A common characteristic of existing studies, e.g., [4, 18, 23, 24, 27, 31, 35–38, 41, 48, 49, 52], is that they tend to be quite broad in scope, covering many areas of SE for ML.”
BackgroundInfluentialA soft actor-critic reinforcement learning algorithm for network intrusion detection
Li, Z., Huang, C., Deng, S., Qiu, W., & Gao, X. (2023) · Computers & Security · doi:10.1016/j.cose.2023.103502 · Cited by 44
Babu, A., & Mandal, D. (2024) · ACS Applied Energy Materials · doi:10.1021/acsaem.3c02642 · Cited by 36
spade: Synthesizing Data Quality Assertions for Large Language Model Pipelines
Shankar, S., Li, H., Asawa, P., Hulsebos, M., Lin, Y., Zamfirescu-Pereira, J. D., Chase, H., Fu-Hinthorn, W., Parameswaran, A., & Wu, E. (2024) · Proceedings of the VLDB Endowment · doi:10.14778/3685800.3685835 · Cited by 72
Machine Learning With Computer Networks: Techniques, Datasets, and Models
Afifi, H., Pochaba, S., Boltres, A., Laniewski, D., Haberer, J., Paeleke, L., Poorzare, R., Stolpmann, D., Wehner, N., Redder, A., et al. (2024) · IEEE Access · doi:10.1109/access.2024.3384460 · Cited by 45
OECD (2023) · OECD digital economy papers · doi:10.1787/2448f04b-en · Cited by 95
Yuan, H. (2024) · Health Care Science · doi:10.1002/hcs2.114 · Cited by 37
“Compared with the first step, continual monitoring is used to identify data distribution drift, control model quality, and trigger system alarms when an ML model deviates from its normal behavior in the target setting [15].”
BackgroundChen, C., da Silva, B., Yang, C., Ma, C., Li, J., & Liu, C. (2023) · IEEE Transactions on Biomedical Circuits and Systems · doi:10.1109/tbcas.2023.3299084 · Cited by 41
“These tools are primarily aimed at software implementation [35].”
MethodMachine learning for healthcare that matters: Reorienting from technical novelty to equitable impact
Balagopalan, A., Baldini, I., Celi, L. A., Gichoya, J. W., McCoy, L. G., Naumann, T., Shalit, U., van der Schaar, M., & Wagstaff, K. L. (2024) · PLOS Digital Health · doi:10.1371/journal.pdig.0000474 · Cited by 42
“Machine learning models require significant work to be deployed in practice, especially in high-stakes domains such as healthcare [107,108].”
Backgroundmlpack 4: a fast, header-only C++ machine learning library
Curtin, R. R., Edel, M., Shrit, O., Agrawal, S., Basak, S., Balamuta, J., Birmingham, R., Dutt, K., Eddelbüttel, D., Garg, R., et al. (2023) · The Journal of Open Source Software · doi:10.21105/joss.05026 · Cited by 41
Aligning machine and human visual representations across abstraction levels
Muttenthaler, L., Greff, K., Born, F., Spitzer, B., Kornblith, S., Mozer, M. C., Müller, K. R., Unterthiner, T., & Lampinen, A. K. (2025) · Nature · doi:10.1038/s41586-025-09631-6 · Cited by 94
“However, these deployments lead to practical 49 and conceptual 50 concerns about trustworthiness and safety.”
Real-Time Semantic Segmentation: A brief survey and comparative study in remote sensing
Broni-Bediako, C., Xia, J., & Yokoya, N. (2023) · IEEE Geoscience and Remote Sensing Magazine · doi:10.1109/mgrs.2023.3321258 · Cited by 33
Torbarina, L., Ferkovic, T., Roguski, Ł., Mihelčić, V., Šarlija, B., & Kraljević, Ž. (2024) · Natural Language Processing Journal · doi:10.1016/j.nlp.2024.100076 · Cited by 33
Nutrition-Related Knowledge Graph Neural Network for Food Recommendation
Ma, W., Li, M., Dai, J., Ding, J., Chu, Z., & Chen, H. (2024) · Foods · doi:10.3390/foods13132144 · Cited by 39
“Therefore, leveraging technologies such as machine learning [7–9] and data mining [10–12] to construct personalized food recommendation models based on user preferences, historical behaviors, and food attributes has become a prevalent trend.”
BackgroundEnd-to-end predictive intelligence diagnosis in brain tumor using lightweight neural network
Ma, L., & Zhang, F. (2021) · Applied Soft Computing · doi:10.1016/j.asoc.2021.107666 · Cited by 29
Beyond Preserved Accuracy: Evaluating Loyalty and Robustness of BERT Compression
Xu, C., Zhou, W., Ge, T., Xu, K., McAuley, J., & Wei, F. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.832 · Cited by 53
“In real-world settings, training and deployment are often separate (Paleyes et al., 2020).”
BackgroundTowards a Better Understanding of Evaluating Trustworthiness in AI Systems
Kemmerzell, N., Schreiner, A., Khalid, H., Schalk, M., & Bordoli, L. (2025) · ACM Computing Surveys · doi:10.1145/3721976 · Cited by 45
Deepfake Media Forensics: State of the Art and Challenges Ahead
Amerini, I., Barni, M., Battiato, S., Bestagini, P., Boato, G., Bonaventura, T. S., Bruni, V., Caldelli, R., Natale, F. G. B. D., Nicola, R. D., et al. (2025) · Lecture notes in social networks · doi:10.1007/978-3-031-85386-9_3 · Cited by 46
SPADE: Synthesizing Data Quality Assertions for Large Language Model Pipelines
Shankar, S., Li, H., Asawa, P., Hulsebos, M., Lin, Y., Zamfirescu-Pereira, J. D., Chase, H., Fu-Hinthorn, W., Parameswaran, A. G., & Wu, E. (2024) · arXiv · doi:10.48550/arxiv.2401.03038 · Cited by 27
Alviz-Meza, A., Orozco-Agamez, J., Quinayá, D. C. P., & Amador, A. A. (2023) · ChemEngineering · doi:10.3390/chemengineering7010002 · Cited by 30
Chen, M., Park, Y., Mangalathu, S., & Jeon, J. (2024) · Earthquake Engineering & Structural Dynamics · doi:10.1002/eqe.4230 · Cited by 27
Marx, E., Benoit, W., Gunny, A., Omer, R., Chatterjee, D., Venterea, R. C., Wills, L., Saleem, M., Moreno, E. A., Raikman, R. J., et al. (2025) · Physical review. D/Physical review. D. · doi:10.1103/physrevd.111.042010 · Cited by 25
On the Understandability of MLOps System Architectures
Warnett, S. J., & Zdun, U. (2024) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2024.3367488 · Cited by 36
“In a survey of case studies, Paleyes et al. [30] identified significant practical challenges in deploying machine learning systems specific to ML architecture.”
BackgroundOptimizing Internet of Things Honeypots with Machine Learning: A Review
Lanz, S., Pignol, S. L.-R., Schmitt, P., Wang, H., Παπαϊωάννου, Μ., Choudhary, G., & Dragoni, N. (2025) · Applied Sciences · doi:10.3390/app15105251 · Cited by 24
Data and model bias in artificial intelligence for healthcare applications in New Zealand
Yogarajan, V., Dobbie, G., Leitch, S. V., Keegan, T. T., Bensemann, J., Witbrock, M., Asrani, V. M., & Reith, D. M. (2022) · Frontiers in Computer Science · doi:10.3389/fcomp.2022.1070493 · Cited by 36
“Ensuring equity requires identifying and mitigating fairness problems in data collection, data storage, data availability, model development, and model deployment (Mehrabi et al., 2021; Paleyes et al., 2022).”
BackgroundMLHOps: Machine Learning Health Operations
Khattak, F. K., Subasri, V., Krishnan, A., Pou-Prom, C., Koçak, S. A., Dolatabadi, E., Pandya, D., Seyyed-Kalantari, L., & Rudzicz, F. (2024) · IEEE Access · doi:10.1109/access.2024.3521279 · Cited by 38
“Prior to deployment, it is crucial to simulate the online learning procedure on retrospective data to assess robustness to data shifts [55], [219].”
A review of machine learning in scanpath analysis for passive gaze-based interaction
Selim, A. M., Barz, M., Bhatti, O. S., Alam, H., & Sonntag, D. (2024) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2024.1391745 · Cited by 34
Weber, M., Hein, A., Weking, J., & Krcmar, H. (2024) · Information Systems Journal · doi:10.1111/isj.12567 · Cited by 35
An empirical guide to MLOps adoption: Framework, maturity model and taxonomy
John, M. M., Olsson, H. H., & Bosch, J. (2025) · Information and Software Technology · doi:10.1016/j.infsof.2025.107725 · Cited by 36
Machine learning in X-ray diffraction for materials discovery and characterization
Davel, C. M., Bassiri‐Gharb, N., & Correa‐Baena, J. (2025) · Matter · doi:10.1016/j.matt.2025.102272 · Cited by 28
Data issues in industrial AI systems: A meta-review and research strategy
Li, X., Yang, C., Møller, C., & Lee, J. (2025) · Computers in Industry · doi:10.1016/j.compind.2025.104361 · Cited by 39
Pati, A. K., Tripathy, A. R., & Subudhi, S. (2025) · Measurement · doi:10.1016/j.measurement.2025.119122 · Cited by 28
Challenges in developing and deploying AI in the engineering, procurement and construction industry
Джусупова, Р., Bosch, J., & Olsson, H. H. (2022) · IEEE Annual International Computer Software and Applications Conference (COMPSAC) · doi:10.1109/compsac54236.2022.00167 · Cited by 31
Purcell, C. R., Walsh, A., Colefax, A. P., & Butcher, P. A. (2022) · Frontiers in Marine Science · doi:10.3389/fmars.2022.981897 · Cited by 23
Prediction of protein stability changes upon single-point variant using 3D structure profile
Gong, J., Wang, J., Zong, X., Ma, Z., & Xu, D. H. (2022) · Computational and Structural Biotechnology Journal · doi:10.1016/j.csbj.2022.12.008 · Cited by 22
“Moreover, the D G depends on several factors, such as experimental conditions, temperature, concentrations of salt, pH values, organic solvents, and other chemical agents, which may be difficult to control and may cause data dispersion [75].”
BackgroundMaintainability Challenges in ML: A Systematic Literature Review
Shivashankar, K., & Martini, A. (2022) · Euromicro Conference on Software Engineering and Advanced Applications (SEAA) · doi:10.1109/seaa56994.2022.00018 · Cited by 30
“Even the choice of model training techniques like incremental training and federated learning will add to the complexity of managing, integrating and deploying the training pipeline to other systems and applications [L18], [L19].”
BackgroundMethodInfluentialRobEns: Robust Ensemble Adversarial Machine Learning Framework for Securing IoT Traffic
Alkadi, S., Al-Ahmadi, S. A., & Ismail, M. M. B. (2024) · Sensors · doi:10.3390/s24082626 · Cited by 25
Adu, T. F., Mensah, L. D., Rockson, M. A. D., & Kemausuor, F. (2025) · Cleaner Waste Systems · doi:10.1016/j.clwas.2025.100218 · Cited by 33
González, R., Saha, A., Campbell, C. J. V., Nejat, P., Lokker, C., & Norgan, A. P. (2023) · Journal of Pathology Informatics · doi:10.1016/j.jpi.2023.100347 · Cited by 28
Text classification using NLP based machine learning approach
Mohapatra, S. K., Sarangi, P. K., Sarangi, P. K., Sahu, P., & Sahoo, B. K. (2022) · AIP conference proceedings · doi:10.1063/5.0080301 · Cited by 21
Continuous fake media detection: Adapting deepfake detectors to new generative techniques
Tassone, F., Maiano, L., & Amerini, I. (2024) · Computer Vision and Image Understanding · doi:10.1016/j.cviu.2024.104143 · Cited by 27
“Our goal is therefore to propose a simple yet effective design for a Machine Learning Model Operations ( MLOps [19, 25]) pipeline that would enable the end-to-end development of continuously trained and monitored intelligent detectors with a minimal set of components.”
BackgroundMethodShaban, M., Alateyah, A. I., Alsharekh, M. F., Alawad, M. O., BaQais, A., Kamel, M., Alsunaydih, F. N., El-Garaihy, W. H., & Salem, H. G. (2023) · Journal of Manufacturing and Materials Processing · doi:10.3390/jmmp7020052 · Cited by 21
An empirical study of challenges in machine learning asset management
Zhao, Z., Chen, Y., Bangash, A. A., Adams, B., & Hassan, A. E. (2024) · Empirical Software Engineering · doi:10.1007/s10664-024-10474-4 · Cited by 31
Insights into Artificial Intelligence Bias: Implications for Agriculture
Mayuravaani, M., Ramanan, A., Perera, M., Senanayake, D., & Vidanaarachchi, R. (2024) · Digital Society · doi:10.1007/s44206-024-00142-x · Cited by 23
“Thus, it’s recommended to involve researchers throughout the entire development process to ensure a cohesive approach (Paleyes et al., 2022 ).”
InfluentialTowards an AI Assistant for Power Grid Operators
Marot, A., Rozier, A., Dussartre, M., Crochepierre, L., & Donnot, B. (2022) · Frontiers in artificial intelligence and applications · doi:10.3233/faia220191 · Cited by 37
“These developments should create a necessary technical stack or data platform for an assistant, to overcome challenges in deploying AI [87,88].”
BackgroundLi, S., & Dib, O. (2024) · Journal of theoretical and applied electronic commerce research · doi:10.3390/jtaer19040141 · Cited by 18
Bhattacherjee, A., & Badhan, A. K. (2024) · Studies in big data · doi:10.1007/978-981-97-0448-4_15 · Cited by 18
ML-Enabled Systems Model Deployment and Monitoring: Status Quo and Problems
Zimelewicz, E., Kalinowski, M., Méndez, D., Giray, G., Alves, A. P. S., Lavesson, N., Azevedo, K., Villamizar, H., Escovedo, T., Lopes, H., et al. (2024) · Lecture notes in business information processing · doi:10.1007/978-3-031-56281-5_7 · Cited by 32
Network security AIOps for online stream data monitoring
Nguyen, G., Dlugolinský, Š., Tran, V., & García, Á. L. (2024) · Neural Computing and Applications · doi:10.1007/s00521-024-09863-z · Cited by 23
Metadata
- Authors
- Paleyes, A., Urma, R.-G., & Lawrence, N. D.
- Affiliations
- University of Cambridge (GB)
- Cambridge Spark, Cambridge, United Kingdom
- Venue
- ACM Computing Surveys
- Published
- 2022-12-07
- Volume
- 55
- Issue
- 6
- Pages
- 1-29
- ISSN
- 0360-0300 (ISSN-L)
- 1557-7341
- OpenAlex
- W3104128335
- Semantic Scholar
- b1f574c47d0b6e3032246770b9cbebb9c7bd0c7f
- Cited by
- 638
- References
- 128
- Citations by year
- 2026: 149 · 2025: 156 · 2024: 166 · 2023: 93 · 2022: 46
- FWCI
- 114.17
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- ACM Computing Surveys (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- White Rose Research Online (University of Leeds, The University of Sheffield, University of York) (submittedVersion)
- Apollo (University of Cambridge) (submittedVersion)
- arXiv (Cornell University)
- Apollo
- Funders
- Alan Turing Institute
- UK Research & Innovation
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
- Physical Sciences › Computer Science › Artificial Intelligence › Artificial Intelligence Applications
- Physical Sciences › Computer Science › Information Systems › Cloud Computing and Resource Management
- Keywords
- machine learning deployment, production systems, deployment challenges, industry case studies
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.