Challenges in Deploying Machine Learning: A Survey of Case Studies
Open accessThe DOI matches its Crossref record; 127 of the 128 references it cites were found in a registry or an index.
Abstract
In recent years, machine learning has transitioned from a field of academic research interest to a field capable of solving real-world business problems. However, the deployment of machine learning models in production systems can present a number of issues and concerns. This survey reviews published reports of deploying machine learning solutions in a variety of use cases, industries, and applications and extracts practical considerations corresponding to stages of the machine learning deployment workflow. By mapping found challenges to the steps of the machine learning deployment workflow, we show that practitioners face issues at each stage of the deployment process. The goal of this article is to lay out a research agenda to explore approaches addressing these challenges.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“Compared with the first step, continual monitoring is used to identify data distribution drift, control model quality, and trigger system alarms when an ML model deviates from its normal behavior in the target setting [15].”
From Toward real‐world deployment of machine learning for health care: External validation, continual monitoring, and randomized clinical trials (2024) · cites this paper as [15]
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“Our literature review was conducted using a systematic mapping protocol [54] in order to select a small set of relevant articles from the much larger collection of literature emerging at the intersection of data quality and ML.”
From A Survey of Data Quality Requirements That Matter in ML Development Pipelines (2023) · cites this paper as [54]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325 · cited 2×
“They took one of the state-of-the-art models in the field, BERT [47], and found out that depending on the chosen model size full training procedure can cost anywhere between $50k and $1.”
BackgroundQuinlan, J. R. (1986) · Machine Learning · doi:10.1007/bf00116251 · Cited by 21,439 · cited 1×
“For that reason decision trees (DT) [108], which can be considered a fairly basic ML algorithm, are widely used in practice.”
MethodArmstrong, J. S., & Georgoff, D. M. (1988) · International Journal of Forecasting · doi:10.1016/0169-2070(88)90019-2 · Cited by 17,894
Rudin, C. (2019) · Nature Machine Intelligence · doi:10.1038/s42256-019-0048-x · Cited by 10,561 · cited 1×
“Rudin [111] called the ML community to stop using black box models and explaining their behavior afterward, and instead design models that are inherently interpretable.”
BackgroundOn the Dangers of Stochastic Parrots: Can Language Models Be Too Big? 🦜
Bender, E. M., Gebru, T., McMillan-Major, A., & Shmitchell, S. (2021) · Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3442188.3445922 · Cited by 6,988
Fully homomorphic encryption using ideal lattices
Gentry, C. (2009) · Proceedings of the forty-first annual ACM symposium on Theory of computing · doi:10.1145/1536414.1536440 · Cited by 6,806 · cited 1×
“Some of the ways this trade-of is resolved in practice are diferential privacy that explicitly corrupts the data [42], homomorphic encryption that restricts the class of learning algorithm but allows for training on encrypted data [50], and federated learning that distributes training across personal devices to preser…”
BackgroundPractical Bayesian Optimization of Machine Learning Algorithms
Snoek, J., Larochelle, H., & Adams, R. P. (2012) · arXiv · doi:10.48550/arxiv.1206.2944 · Cited by 6,691 · cited 1×
“Even approaches like Hyperband [78] or Bayesian optimization [129], that are speciically designed to minimize the number of training cycles needed, are not yet able to deal with the high dimensional searches that emerge when many hyper-parameters are involved.”
Dwork, C. (2006) · Lecture Notes in Computer Science · doi:10.1007/11787006_1 · Cited by 5,291 · cited 1×
“Some of the ways this tradeoff is resolved in practice are differential privacy that explicitly corrupts the data [54], homomorphic encryption that restricts the class of learning algorithm but allows for training on encrypted data [55], and federated learning that distributes training across personal devices to prese…”
BackgroundMembership Inference Attacks Against Machine Learning Models
Shokri, R., Stronati, M., Song, C., & Shmatikov, V. (2017) · 2017 IEEE Symposium on Security and Privacy (SP) · doi:10.1109/sp.2017.41 · Cited by 4,754 · cited 1×
“They veriied it on models trained on leading ML-as-a-service providers [127], achieving anywhere from 70% up to 94% accuracy of membership inference.”
BackgroundA survey on concept drift adaptation
Gama, J., Žliobaitė, I., Bifet, A., Pechenizkiy, M., & Bouchachia, A. (2014) · ACM Computing Surveys · doi:10.1145/2523813 · Cited by 3,735 · cited 1×
“We refer interested readers to the works on using domain adaptation [48], meta-learning [46] and transfer learning [152, 157] as ways of addressing dataset shift.”
Method
Research works printed but not linked: 26
- Arif Cam, Michael Chui, and Bryce Hall. 2019. Global AI survey: AI proves its worth, but few scale impact. McKinsey Analytics (2019). https://www.mckinsey.com/featured-insights/artificial-intelligence/global-ai-survey-ai-proves-its-worth-but-few-scale-impact. Accessed from 13/06/2022.
- Royal Society (Great Britain). 2017. Machine Learning: The Power and Promise of Computers that Learn by Example: an Introduction. Royal Society.
- Ben Lorica and Nathan Paco. 2018. The State of Machine Learning Adoption in the Enterprise. O’Reilly Media.
- Colin Shearer. 2000. The CRISP-DM model: The new blueprint for data mining. J. Data Warehous. 5, 4 (2000), 13–22.
- Rajashree Y. Patil and R. V. Kulkarni. 2012. A review of data cleaning algorithms for data warehouse systems. Int. J. Comput. Sci. Inf. Technol. 3, 5 (2012), 5212–5214.
- 10.1109/TITS.2019.2962338 · DOI · Paper page
- Diego Charrez. 2019. NeurIPS 2019 Stats. Retrieved from https://medium.com/@dcharrezt/neurips-2019-stats-c91346d31c8f.
- 10.1023/A:1022643204877 · doi:10.1023/a:1022643204877 · Paper page
- 10.5555/3122009.3242042 · doi:10.5555/3122009.3242042
- Bobak Shahriari, Alexandre Bouchard-Cote, and Nando Freitas. 2016. Unbounded Bayesian optimization via regularization. In Proceedings of the 19th International Conference on Artificial Intelligence and Statistics (Proceedings of Machine Learning Research), Arthur Gretton and Christian C. Robert (Eds.), Vol. 51. PMLR, 1168–1176. Retrieved from http://proceedings.mlr.press/v51/shahriari16.html.
- Ram Ananth, Seph Mard, and Peter Simon. 2019. Opening the “Black Box”: The path to deployment of AI models in banking, white paper. DataRobot (2019). https://www.datarobot.com/resources/opening-the-black-box/. Accessed from 13/06/2022.
- Robert Soden Dennis Wagenaar Dave Luo and Annegien Tijssen. 2019. Taking Ethics Fairness and Bias Seriously in Machine Learning for Disaster Risk Management. arxiv:cs.CY/1912.05538.
- Nantheera Anantrasirichai and David Bull. 2021. Artificial intelligence in the creative industries: A review. Artif. Intell. Rev. (2021), 1–68.
- D. Malan et al. 2016. Values and the fourth industrial revolution: Connecting the dots between value, values, profit and purpose. In Proceedings of the World Economic Forum.
- Daniel Malan. 2018. The law can’t keep up with new tech. Here’s how to close the gap. In Proceedings of the World Economic Forum.
- Ramprakash Ramamoorthy P. Satya Madhuri and Malini Christina Raj. 2019. AI from labs to production—challenges and learnings. USENIX Association. https://www.usenix.org/conference/opml19/presentation/madhuri. Accessed from 13/06/2022.
- Oscar Schwartz. 2019. In 2016 Microsoft’s racist chatbot revealed the dangers of online conversation. IEEE Spectr. 11 (2019).
- Florian Tramèr, Fan Zhang, Ari Juels, Michael K. Reiter, and Thomas Ristenpart. 2016. Stealing machine learning models via prediction APIs. In Proceedings of the 25th USENIX Security Symposium (USENIX Security’16). 601–618.
- Jeremy Hermann and Mike Del Balso. 2017. Meet Michelangelo: Uber’s machine learning platform. Retrieved from https://eng.uber.com/michelangelo.
- Matei Zaharia, Andrew Chen, Aaron Davidson, Ali Ghodsi, Sue Ann Hong, Andy Konwinski, Siddharth Murching, Tomas Nykodym, Paul Ogilvie, Mani Parkhe, et al. 2018. Accelerating the machine learning lifecycle with MLflow.IEEE Data Eng. Bull. 41, 4 (2018), 39–45.
- Nick Hynes, D. Sculley, and Michael Terry. 2017. The Data Linter: Lightweight, automated sanity checking for ML data sets. In Proceedings of the NIPS MLSys Workshop.
- Matthias Feurer, Aaron Klein, Katharina Eggensperger, Jost Springenberg, Manuel Blum, and Frank Hutter. 2015. Efficient and robust automated machine learning. In Advances in Neural Information Processing Systems, C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Garnett (Eds.), Vol. 28. Curran Associates, Inc. Retrieved from https://proceedings.neurips.cc/paper/2015/file/11d0e6287202fced83f79975ec59a3a6-Paper.pdf.
- Randal S. Olson and Jason H. Moore. 2016. TPOT: A tree-based pipeline optimization tool for automating machine learning. In Proceedings of the Workshop on Automatic Machine Learning. PMLR, 66–74.
- Stephan Rabanser, Stephan Günnemann, and Zachary Chase Lipton. 2019. Failing loudly: An empirical study of methods for detecting dataset shift. In Proceedings of the Neural Information Processing Systems Foundation Conference.
- Chelsea Finn, Pieter Abbeel, and Sergey Levine. 2017. Model-agnostic meta-learning for fast adaptation of deep networks. In Proceedings of the International Conference on Machine Learning. PMLR, 1126–1135.
- A. Zujus. 2018. AI project development–how project managers should prepare. Tratto il giorno Agosto 31 (2018), 2019.
Cited by
Items 201–300 of 737
Zhang, X., Zhao, P., Jaskolka, J., Li, H., & Lu, R. (2026) · Empirical Software Engineering · doi:10.1007/s10664-025-10795-y · Cited by 10
“Its importance is particularly evident in applications requiring the processing and analysis of vast amounts of data, such as autonomous vehicles, healthcare diagnostics, financial fraud detection, and industrial automation (Sarker 2021; Paleyes et al. 2022).”
Ogundairo, K. M. (2026) · IIARD INTERNATIONAL JOURNAL OF ECONOMICS AND BUSINESS MANAGEMENT · doi:10.56201/ijebm.vol.11.no12.2025.pg441.478 · Cited by 14
Glass, A., & Noennig, J. R. (2022) · Procedia Computer Science · doi:10.1016/j.procs.2022.09.395 · Cited by 8
Some aspects of software engineering for AI-based systems
Любченко, В. В. (2022) · PROBLEMS IN PROGRAMMING · doi:10.15407/pp2022.03-04.099 · Cited by 8
Data augmentation strategies to improve text classification: a use case in smart cities
Bencke, L. R., & Moreira, V. P. (2023) · Language Resources and Evaluation · doi:10.1007/s10579-023-09685-w · Cited by 14
“The lack of labeled data can be a consequence of the limited access to experts, absence of high-variance data, or environments that tend to generate large volumes of data making manual label assignment unfeasible ( e.g., network traffic analysis) (Paleyes et al., 2022).”
Muttaqin, R., & Sudiana, D. (2025) · Jurnal Penelitian Pendidikan IPA · doi:10.29303/jppipa.v10i12.8346 · Cited by 9
Джусупова, Р., Bosch, J., & Olsson, H. H. (2022) · IEEE/ACM International Conference on AI Engineering – Software Engineering for AI (CAIN) · doi:10.1145/3522664.3528595 · Cited by 10
Kurzekar, A. S., Waghe, U. P., & Waghe, P. (2025) · Iranian Journal of Science and Technology Transactions of Civil Engineering · doi:10.1007/s40996-025-01979-z · Cited by 10
“This work presents a new framework that facilitates the use of recycled materials in real-world applications, to the bene�t of the environment and performance under harsh service conditions.”
Sethi, S. R., & Mahadik, D. A. (2025) · Journal of Chinese Economic and Business Studies · doi:10.1080/14765284.2025.2549237 · Cited by 9
Centralized Two-Tiered Tree-Based Intrusion-Detection System (C2T-IDS)
Yassine, H. A. K., Saleh, M. A., Nakhal, B., & Chakik, A. E. (2025) · IoT · doi:10.3390/iot6040067 · Cited by 9
“Nevertheless, the centralized methodology persists as a dominant and effective solution, especially in scenarios with lower data sensitivity or where strong, encrypted communication pathways are established [12].”
An Analysis of the Barriers Preventing the Implementation of MLOps
Narayanappa, A. K., & Amrit, C. (2023) · IFIP advances in information and communication technology · doi:10.1007/978-3-031-50188-3_10 · Cited by 7
Common Errors in Machine Learning Projects: A Second Look
Zimmermann, R. M., Allin, S. J., & Zhang, L. (2023) · Koli Calling International Conference on Computing Education Research · doi:10.1145/3631802.3631808 · Cited by 10
Speeding up Explorative BPM with Lightweight IT: the Case of Machine Learning
Bojer, C. S., Bygstad, B., & Øvrelid, E. (2024) · Information Systems Frontiers · doi:10.1007/s10796-024-10474-1 · Cited by 7
“ML is different from traditional software in that 1) its output is probabilistic, with the affordances offered depending on the particular dataset used for learning, and 2) it has models and datasets as key artefacts (Paleyes et al., 2022).”
Catching Silent Failures: A Machine Learning Model Monitoring and Explainability Survey
Karval, R., & Singh, K. N. (2023) · OITS International Conference on Information Technology (OCIT) · doi:10.1109/ocit59427.2023.10431343 · Cited by 12
“ML models can malfunction and deteriorate in real-world activity, thus monitoring them after deployment is essential [9].”
BackgroundA Model for Estimating the Economic Costs of Computer Vision Systems That Use Deep Learning
Thompson, N., Fleming, M., Tang, B. J., Pastwa, A. M., Borge, N., Goehring, B., & Das, S. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i21.30343 · Cited by 14
“Prior literature has often considered the various challenges in deploying deep learning (Paleyes, Urma, and Lawrence 2022) associated with typical stages of a machine learning project life cycle: data management, model learning, model verification, and model deployment (Ashmore, Calinescu, and…”
BackgroundTowards a Comprehensive Ontology for Requirements Engineering for AI-Powered Systems
Sadovski, E., Aviv, I., & Hadar, I. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-57327-9_14 · Cited by 14
Just-in-Time crash prediction for mobile apps
Wimalasooriya, C., Licorish, S. A., da Costa, D. A., & MacDonell, S. G. (2024) · Empirical Software Engineering · doi:10.1007/s10664-024-10455-7 · Cited by 11
TriFusion hybrid model for human activity recognition
Ahmed, M. F., He, G., & Wang, S. (2024) · Signal Image and Video Processing · doi:10.1007/s11760-024-03487-5 · Cited by 7
Bainomugisha, E., & Nabende, J. N. (2024) · Applied AI Letters · doi:10.1002/ail2.100 · Cited by 7
FeedbackLogs: Recording and Incorporating Stakeholder Feedback into Machine Learning Pipelines
Barker, M., Kallina, E., Ashok, D., Collins, K. M., Casovan, A., Weller, A., Talwalkar, A., Chen, V., & Bhatt, U. (2023) · ACM Conference on Equity and Access in Algorithms, Mechanisms, and Optimization (EAAMO) · doi:10.1145/3617694.3623239 · Cited by 15
“In addition to documenting model development, previous work has argued for a comprehensive understanding of the usage of a system, including algorithmic auditing [21, 56] and critical refusal [29].”
MethodNguyen, Q. T., Tran, M. P., Prabhakaran, V., Liu, A., & Hai, G. N. (2024) · Frontiers in Medicine · doi:10.3389/fmed.2024.1398565 · Cited by 7
Bootstrapping Recommendations at Chrome Web Store
Qin, Z., Zhuang, H., Jagerman, R., Qian, X., Hu, P., Chen, D. C., Wang, X., Bendersky, M., & Najork, M. (2021) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3447548.3467099 · Cited by 16
“[23] survey challenges at every step of the ML deployment workflow due to practical considerations of deploying ML in production.”
BackgroundRequirements and Reference Architecture for MLOps:Insights from IndustryPreprint
Kumara, I., Arts, R., Nucci, D. D., Heuvel, W. J. V. D., & Tamburri, D. A. (2022) · doi:10.36227/techrxiv.21397413 · Cited by 10
Multi-source data joint processing framework for DEM calibration and fusion
Yu, C., Wang, Q., Zhang, Z., Zhong, Z., Ding, Y., Lai, T., Huang, H., & Shen, P. (2025) · International Journal of Applied Earth Observation and Geoinformation · doi:10.1016/j.jag.2025.104484 · Cited by 12
PestDet: A unified detection framework for accurate and efficient stored-grain pest detection
Tian, J., Sun, M., Zhou, H., & Li, J. (2025) · Ecological Informatics · doi:10.1016/j.ecoinf.2025.103145 · Cited by 7
Scalable earthquake magnitude prediction using spatio-temporal data and model versioning
Singh, R., & Roy, B. (2025) · Scientific Reports · doi:10.1038/s41598-025-00804-x · Cited by 8
“MLflow is an open-source platform for managing the machine learning lifecycle 45 , including experimentation, reproducibility, and deployment 46 .”
Integrating AI into healthcare systems: A multivocal literature review
Mallardi, G., Calefato, F., Quaranta, L., & Lanubile, F. (2025) · Smart Health · doi:10.1016/j.smhl.2025.100631 · Cited by 8
BERTau: Itau BERT for digital customer service
Caridá, V. F., Finardi, P., Viegas, J., Ferreira, G., & Fernandes, A. (2023) · LatinX in AI at Neural Information Processing Systems Conference (LXAI) · doi:10.52591/lxai202312107 · Cited by 15
Srinivasan, S., Agrahari, A., & Kumar, A. (2022) · Journal of Decision System · doi:10.1080/12460125.2022.2043576 · Cited by 11
“Execution approach Waterfall approach Success criteria not defined, no MVP orientation Team anchored in IT organization Approaching analytics like IS projects (Paleyes et al., 2020) (Olszak, 2016) (Weiss, 2009) (Peppard & Marchand, 2013)”
BackgroundChallenges in machine learning application development: an industrial experience report
Rahman, M. S., Khomh, F., Rivera, E., Guéhéneuc, Y., & Lehnert, B. (2022) · doi:10.1145/3526073.3527593 · Cited by 11
“We review existing literature to explore software engineering methodology [1, 29] for ML applications, common challenges [16, 21, 25] and best practices [30] for ML application development.”
MethodChallenges and Experiences of Iranian Developers with MLOps at Enterprise
Heydari, M., & Rezvani, Z. (2023) · Iranian Conference on Advances in Enterprise Architecture (ICAEA) · doi:10.1109/icaea60387.2023.10414442 · Cited by 7
“Paleyes et al. [10] reviews published studies of implementing machine learning solutions across diverse use cases, industries, and applications.”
BackgroundFramework for Platform Independent Machine Learning (ML) Model Execution
Ananthi, S., Lokesh, B., Sanjeev Chandran, M., & Sakthi, S. (2024) · International Conference on Intelligent Data Communication Technologies and Internet of Things (IDCIoT) · doi:10.1109/idciot59759.2024.10467931 · Cited by 7
Predicting Weather Conditions for Improving Crop Productivity Using Machine Learning Approaches
Anand, V., Rajput, V. D., Minkina, T., Манджиева, С. С., Sharma, A., Kumar, S., & Konstantinova, E. (2024) · Advances in environmental engineering and green technologies book series · doi:10.4018/979-8-3693-1471-5.ch008 · Cited by 7
Password Strength Classification Using Machine Learning Methods
Rehman, H., Ahmed, A., Hussain, A., Umar, M., Siraj, M., Ahmed, W., Muhamnad, M., & Rehman, E. (2024) · Global Conference on Wireless and Optical Technologies (GCWOT) · doi:10.1109/gcwot63882.2024.10805622 · Cited by 6
Modyn: Data-Centric Machine Learning Pipeline Orchestration
Böther, M., Robroek, T., Gsteiger, V., Holzinger, R., Ma, X., Tözün, P., & Klimovic, A. (2025) · Proceedings of the ACM on Management of Data · doi:10.1145/3709705 · Cited by 19
“While a few (often commercial) platforms like NeptuneAI [70], Amazon SageMaker [5], Continuum [107], or Tensorflow TFX [65] have partial retraining support, deploying continuous retraining still requires a lot of manual plumbing [10, 75, 95, 108].”
Handling imperfection: A taxonomy for machine learning on data with data quality defects
Hagn, M., Heinrich, B., Krapf, T., & Schiller, A. (2025) · Decision Support Systems · doi:10.1016/j.dss.2025.114493 · Cited by 9
“Paleyes et al. [35] examine DQ issues appearing during the process of data collection and data (pre)processing for ML-based analysis.”
Tian, J. (2025) · AI · doi:10.3390/ai6070140 · Cited by 7
Automated algorithm selection for Predictive Maintenance: Advances and challenges
Engbers, H., & Freitag, M. (2025) · Journal of Manufacturing Systems · doi:10.1016/j.jmsy.2025.06.023 · Cited by 6
Evaluating machine learning techniques to define the factors related to boar taint
Makridis, G., Heyrman, E., Kotios, D., Mavrepis, P., Callens, B., Vijver, R. V. D., Maselyne, J., Aluwé, M., & Kyriazis, D. (2022) · Livestock Science · doi:10.1016/j.livsci.2022.105045 · Cited by 8
Suryadevara, M., Rangineni, S., & Venkata, S. (2023) · International Journal of Science and Research (IJSR) · doi:10.21275/sr23719211528 · Cited by 18
Out-of-Distribution Detection in Deep Learning Models: A Feature Space-Based Approach
Carvalho, T. M., Vellasco, M. M. B. R., & Amaral, J. F. (2023) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn54540.2023.10191711 · Cited by 6
“When deploying DL in practical applications, several challenges must be addressed [1].”
Automation and Development Effort in Continuous AI Development: A Practitioners’ Survey
Steidl, M., Golendukhina, V., Felderer, M., & Ramler, R. (2023) · Euromicro Conference on Software Engineering and Advanced Applications (SEAA) · doi:10.1109/seaa60479.2023.00027 · Cited by 9
“In summary, employing fully automated stages and tasks for the continuous development of AI minimizes the effort during task execution, accelerates model deployment in production, and ensures consistency while improving reliability and quality [14], [20].”
BackgroundMethodAtalan, A., & Dönmez, C. Ç. (2024) · Healthcare · doi:10.3390/healthcare12131272 · Cited by 6
“Generally, ML consists of the following phases: data collection, data preprocessing, model selection, model training, model evaluation, and model deployment [31].”
BackgroundAn empirical evaluation of flow based programming in the machine learning deployment context
Paleyes, A., Cabrera, C., & Lawrence, N. D. (2022) · IEEE/ACM International Conference on AI Engineering – Software Engineering for AI (CAIN) · doi:10.1145/3522664.3528601 · Cited by 12
“Deployment of ML in production faces a variety of challenges that affect every step of the workflow [41].”
BackgroundMethodInfluentialSelf-citationOn the Democratization of Machine Learning Pipelines
Carqueja, A., Cabral, B., Fernandes, J. P., & Lourenço, N. (2022) · IEEE Symposium Series on Computational Intelligence (SSCI) · doi:10.1109/ssci51031.2022.10022107 · Cited by 6
“In another study [11], Paleyes et al. performed a review on published reports of ML deployment solutions.”
BackgroundInfluentialDataflow graphs as complete causal graphs
Paleyes, A., Guo, S., Schölkopf, B., & Lawrence, N. D. (2023) · IEEE/ACM International Conference on AI Engineering – Software Engineering for AI (CAIN) · doi:10.1109/cain58948.2023.00010 · Cited by 6
“First, businesses and research groups seek to deploy more data driven software solutions, including those powered by machine learning (ML), which means software engineers are more often faced with data availability and analysis requirements [23], [32].”
BackgroundSelf-citationTest & Evaluation Best Practices for Machine Learning-Enabled SystemsPreprint
Chandrasekaran, J., Cody, T., McCarthy, N., Lanus, E., & Freeman, L. E. B. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.4862589 · Cited by 14
SpaceFusion: Advanced Deep Learning Operator Fusion via Space-Mapping Graph
Zhu, L., Yao, J., & Guan, H. (2025) · European Conference on Computer Systems · doi:10.1145/3689031.3696087 · Cited by 6
“…learning models manifest two trends: increasing computational scale and heightened computational complexity[27, 28, 37, 40, 51, 52], making it more expensive and challenging to deploy them as efficient, rapid-response inference services, whether on cloud infrastructures or local devices[43, 44].”
An algorithmic perspective on deciphering cell–cell interactions with spatial omics data
van Santvoort, M., & Eduati, F. (2025) · Briefings in Bioinformatics · doi:10.1093/bib/bbaf236 · Cited by 6
“However, their flexibility comes at the cost of interpretability [45]: these black-box models provide limited insight into how predictions are made, potentially hindering their generalizability to different datasets or biological settings.”
A fairness scale for real-time recidivism forecasts using a national database of convicted offenders
Verrey, J., Neyroud, P., Sherman, L. W., & Ariel, B. (2025) · Neural Computing and Applications · doi:10.1007/s00521-025-11478-x · Cited by 7
Wang, X., Chen, Q., He, N., Wang, D., Nie, B., Zhang, Y., & Liu, W. (2025) · Journal of Energy Storage · doi:10.1016/j.est.2025.119594 · Cited by 5
Awad, S. S., Hamza, A. A., Sobh, M. A., & Bahaa-Eldin, A. M. (2026) · Scientific Reports · doi:10.1038/s41598-026-49939-5 · Cited by 5
“All of this is aimed at making sure our models are fully prepared for real-world production settings 15,16 .”
InfluentialData Cleansing With Minimum Distortion for ML-Based Equipment Anomaly Detection
Hsieh, Y.-C., Chen, C., Liao, D.-Y., Lin, K.-C., & Chang, S. (2023) · IEEE Transactions on Semiconductor Manufacturing · doi:10.1109/tsm.2023.3262957 · Cited by 5
“Data cleansing (or data cleaning) is thus used in the preprocessing step of data-driven machine learning [3], [4], [5], [6] to help improve the quality of collected data by defining, identifying, and correcting errors to ensure the integrity of the data [7].”
MethodSupervised Learning Models for Tomato Quality Prediction Using Electronic Nose Data
Affognon, L., Diallo, A., Diallo, C., & Ezin, E. C. (2023) · International Conference on Electrical, Computer and Energy Technologies (ICECET) · doi:10.1109/icecet58911.2023.10389485 · Cited by 5
“As stated by A. Paleyes et al. in [3], ”Machine learning (ML) has evolved from being an area of academic research to an applied field”.”
BackgroundResource Democratization: Is Compute the Binding Constraint on AI Research?
Gelles, R., Kinoshita, V., Musser, M., & Dunham, J. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i18.29959 · Cited by 14
“Previous literature review (Paleyes, Urma, and Lawrence 2022) and interview studies (Merhi 2023; Westenberger, Schuler, and Schlegel 2022; Baier, J¨ohren, and Seebacher 2019; Shankar et al. 2022) have focused on AI project’s success or failure in practice.”
BackgroundAn Exploratory Study on Machine Learning Model Management
Latendresse, J., Abedu, S., Abdellatif, A., & Shihab, E. (2024) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3688841 · Cited by 12
Pacheco, V. M. G., de Paiva, J. P. Q., Silva, B. C. R., Santos, P. V., Ferreira, J. R., Reis, M. R. C., Tornieri, D., Ribeiro, G., da Silva, L. O., Nogueira, S. A., et al. (2024) · PLoS ONE · doi:10.1371/journal.pone.0307022 · Cited by 7
Habib, F., Ali, Z., Azam, A., Kamran, K., & Pasha, F. M. (2024) · Frontiers in Big Data · doi:10.3389/fdata.2024.1387325 · Cited by 7
Automatic Data Labeling for Software Vulnerability Prediction Models: How Far Are We?
Le, T. H. M., & Babar, M. A. (2024) · Proceedings of the ... ACM-IEEE International Symposium on Empirical Software Engineering and Measurement/International Symposium on Empirical Software Engineering and Measurement · doi:10.1145/3674805.3686675 · Cited by 14
“CL [64] is the state-of-the-art method for removing data noise in many domains such as audio/text processing, and recently Software Engineering [18, 66].”
DWFL: Enhancing Federated Learning Through Dynamic Weighted Averaging
Chourasia, P., Ali, T. E., Ali, S., & Pattersn, M. (2025) · Communications in computer and information science · doi:10.1007/978-3-031-91428-7_3 · Cited by 5
Maturity Framework for Enhancing Machine Learning Quality
Castelli, A., Chouliaras, G. C., & Goldenberg, D. (2025) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3711896.3737246 · Cited by 8
Computationally Assisted Quality Control for Public Health Data Streams
Joshi, A., Mazaitis, K., Rosenfeld, R., & Wilder, B. (2023) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2023/666 · Cited by 6
Dual-Step Transfer Learning-Based Prediction Model for Next-Generation Intelligent Cellular Networks
Aziz, W. A., Ioannou, I. I., Lestas, M., & Vassiliou, V. (2023) · IEEE International Conference on Communications workshops · doi:10.1109/iccworkshops57953.2023.10283745 · Cited by 5
Anonymization-as-a-Service: The Service Center Transcripts Industrial Case
Borovits, N., Bardelloni, G., Tamburri, D. A., & van den Heuvel, W. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-48424-7_19 · Cited by 6
Liao, Y., Bian, J., Zhao, J., Zhang, Z., Xu, S., Qin, Y., Luo, X., Miao, S., Liu, R., Liu, R., et al. (2024) · iMeta · doi:10.1002/imt2.162 · Cited by 5
Sabrigiriraj, M., & Manoharan, K. (2024) · WSEAS TRANSACTIONS ON ADVANCES in ENGINEERING EDUCATION · doi:10.37394/232010.2024.21.8 · Cited by 5
“However, researchers face challenges during the deployment of the ML model at every stage, [1] and this provides a wide opportunity for the research community to exploration.”
BackgroundLSPR23: A novel IDS dataset from the largest live-fire cybersecurity exercise
Dijk, A., Halisdemir, M. E., Melella, C., Schu, A., Pihelgas, M., & Meier, R. (2024) · Journal of Information Security and Applications · doi:10.1016/j.jisa.2024.103847 · Cited by 7
Machine Learning Operations: A Mapping Study
Chakraborty, A., Das, S., & Gary, K. A. (2025) · Communications in computer and information science · doi:10.1007/978-3-031-86644-9_1 · Cited by 8
AI Education in a Mirror: Challenges Faced by Academic and Industry Experts
Akgün, M., & Hosseini, H. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-98417-4_24 · Cited by 9
Identification of North American softwoods via machine-learning
Lopes, D. J. V., Bobadilha, G. D. S., Burgreen, G. W., & Entsminger, E. D. (2021) · Canadian Journal of Forest Research · doi:10.1139/cjfr-2020-0416 · Cited by 5
The Significance of Machine Learning in the Manufacturing Sector: An ISM Approach
Lakra, A., Gupta, S., Ranjan, R., Tripathy, S., & Singhal, D. (2022) · Logistics · doi:10.3390/logistics6040076 · Cited by 7
A Risk Predictive Model for Primary Tumor using Machine Learning with Initial Missing Values
Bajaj, R., Shandilya, D., Gagneja, S., Gupta, K., & Rawat, D. (2022) · International Conference on Data Science and Information System (ICDSIS) · doi:10.1109/icdsis55133.2022.9915957 · Cited by 5
“The outcomes are analyzed which proved effective [11].”
BackgroundAnalysis and Benchmarking of feature reduction for classification under computational constraints
Subasi, O., Ghosh, S., Manzano, J., Palmer, B. J., & Márquez, A. (2024) · Machine Learning Science and Technology · doi:10.1088/2632-2153/ad3726 · Cited by 7
“However, several research problems remain open [6].”
Automated search of process control limits for fault detection in time series data
Schlegl, T., Tomaselli, D., Schlegl, S., West, N., & Deuse, J. (2022) · Journal of Process Control · doi:10.1016/j.jprocont.2022.07.002 · Cited by 8
Towards semantically enhanced detection of emerging quality-related concerns in source code
Krasniqi, R., & Do, H. (2023) · Software Quality Journal · doi:10.1007/s11219-023-09614-8 · Cited by 5
A Unified System for Molecular Property Predictions: Oloren ChemEngine and its ApplicationsPreprint
Huang, D. Z., Chowdhuri, S., Li, A. C., Li, A., Agrawal, A., Gano, K., & Zhu, A. (2022) · ChemRxiv · doi:10.26434/chemrxiv-2022-zz776 · Cited by 6
Onboard AI: Constraints and Limitations
Miller, K., & Lohn, A. J. (2023) · doi:10.51593/2022ca008 · Cited by 9
Kristiana, I., Prabowo, H., Gaol, F. L., & Qomariyah, N. N. (2025) · IEEE Access · doi:10.1109/access.2025.3584648 · Cited by 5
Challenges and Opportunities for Unikernels in Machine Learning Inference
Ahuja, A., & Jain, V. (2021) · International Conference on Reliability, Infocom Technologies and Optimization (Trends and Future Directions) (ICRITO) · doi:10.1109/icrito51393.2021.9596080 · Cited by 4
Hildesheim, W., Holoyad, T., & Schmid, T. (2023) · it - Information Technology · doi:10.1515/itit-2023-0028 · Cited by 4
Unlocking Deeper Understanding: Leveraging Explainable AI for API Anomaly Detection Insights
Jones, M., Bayesh, M., & Jahan, S. (2024) · International Conference on Machine Learning and Computing (ICMLC) · doi:10.1145/3651671.3651738 · Cited by 7
Дроменко, В. Б., & Lisovets, S. (2024) · Scientific notes of Taurida National V I Vernadsky University Series Technical Sciences · doi:10.32782/2663-5941/2024.4/12 · Cited by 4
A Formal Model for Artificial Intelligence Applications in Automation Systems
Schieseck, M., Topalis, P., Reinpold, L. M., Gehlhoff, F., & Fay, A. (2024) · IEEE International Conference on Emerging Technologies and Factory Automation (ETFA) · doi:10.1109/etfa61755.2024.10710890 · Cited by 4
“[5]–[8] While some of these challenging aspects, such as data quality or trustworthiness, have received extensive research attention in the past years, the software documentation of AI application and its impact to the overall system in various scenarios is often disregarded [7], [8].”
BackgroundKapalaga, G., Kivunike, F. N., Kerfua, S. D., Jjingo, D., Biryomumaisho, S., Rutaisire, J., Ssajjakambwe, P., Mugerwa, S., Abbey, S., Aaron, M. H., et al. (2024) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2024.1455331 · Cited by 5
Analysis of AutoML Tools in the World of Automated Deep Learning
Connor, L., Kelly, R., Murphy, S., McCarthy, E., & Murray, E. (2024) · Fusion of Multidisciplinary Research An International Journal · doi:10.63995/fxpc8243 · Cited by 5
“…It’s worth noting that the choice of an AutoML platform or tool depends on various factors, such as the specific requirements of the project, the level of expertise of the user, the type of data and problem being addressed, and the desired level of control and customization [23, 24, 25, 26, 27].”
Naser, M. Z. (2025) · Technology in Society · doi:10.1016/j.techsoc.2025.103039 · Cited by 5
MLOps in Practice: Requirements and a Reference Architecture from Industry
Kumara, I., Arts, R., Ferreira, R. C., Nucci, D. D., Kazman, R., Tamburri, D. A., & van den Heuvel, W. (2025) · Lecture notes in computer science · doi:10.1007/978-3-032-02138-0_2 · Cited by 7
Hunsicker, T., König, C. J., & Langer, M. (2025) · Computers in Human Behavior Artificial Humans · doi:10.1016/j.chbah.2025.100216 · Cited by 10
Cheimarios, N. (2025) · Frontiers in Physics · doi:10.3389/fphy.2025.1711356 · Cited by 7
“Unlike classical simulation codes, which once validated can remain useful for decades, AI-driven systems demand constant monitoring, retraining, and governance [68, 70, 75].”
Machine Learning for Medical Image Analysis
Bouamrane, A., Derdour, M., Bennour, A., Benmachiche, A., & Gasmi, M. (2025) · Springer eBooks · doi:10.1007/978-3-032-02963-8_4 · Cited by 4
A sampling-based approach for information-theoretic inspection management
Bull, L. A., Dervilis, N., Worden, K., Cross, E. J., & Rogers, T. J. (2022) · Proceedings of the Royal Society A Mathematical Physical and Engineering Sciences · doi:10.1098/rspa.2021.0790 · Cited by 7
“1An established problem in practical applications of machine learning, often referred to as ‘high variance’ data [18].”
BackgroundIntegrating Machine Learning in Urban Pedagogy: Addressing Homelessness in Skid Row
Meshkani, T. (2024) · Architecture · doi:10.3390/architecture4010008 · Cited by 4
“This scenario is analogous to challenges faced in deploying ML models in resource-limited environments, such as the Europa Clipper spacecraft project, where computational resource scarcity necessitated the use of less demanding ML techniques [23].”
AutoMHS-GPT: Automated Model and Hyperparameter Selection with Generative Pre-Trained Model
de Souza, L. A. C., Sammarco, M., Achir, N., Campista, M. E. M., & Costa, L. H. M. K. (2024) · IEEE International Conference on Cloud Networking · doi:10.1109/cloudnet62863.2024.10815898 · Cited by 4
“However, deploying artificial intelligence models in production environments takes days or even months [1].”
BackgroundSemi-Supervised Learning for Deep Causal Generative Models
Ibrahim, Y., Warr, H., & Kamnitsas, K. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-72390-2_28 · Cited by 5
Human-Centered AI (Also) for Humanistic Management
Facchini, A., & Mangili, F. (2024) · Humanism in business series · doi:10.1007/978-3-031-67155-5_11 · Cited by 5
Industrial Applications of AI in Aircraft Manufacturing: A PRISMA Systematic Literature Review
Bougault, P., Haddad, R., & Ma, L. (2026) · Production Engineering Archives · doi:10.30657/pea.2026.32.5 · Cited by 6
“Moreover, the model choice is made based on: (1) the problem type, whether linear or non-linear, and its complexity; (2) data type: continuous, categorical variables, etc.; (3) the quantity of data to be processed (Gzar et al., 2022); (4) the desired complexity of the models (Paleyes et al., 2023).”
AssistML: A Concept to Recommend ML Solutions for Predictive Use Cases
Zacarias, A. G. V., Weber, C., Reimann, P., & Mitschang, B. (2021) · IEEE International Conference on Data Science and Advanced Analytics (DSAA) · doi:10.1109/dsaa53316.2021.9564168 · Cited by 4
Active Learning with Combinatorial Coverage
Katragadda, S. P., Cody, T., Beling, P. A., & Freeman, L. (2022) · International Conference on Machine Learning and Applications · doi:10.1109/icmla55696.2022.00281 · Cited by 6
“As pointed out by Paleyes and Urma, model selection is often an iterative process in machine learning deployment [11].”
BackgroundAhmad, M. F. (2023) · International Journal of Information Technology · doi:10.1007/s41870-023-01507-0 · Cited by 6
Artificial intelligence in analogue circuit design
Chen, A. (2024) · Applied and Computational Engineering · doi:10.54254/2755-2721/48/20241365 · Cited by 4
Metadata
- Authors
- Paleyes, A., Urma, R.-G., & Lawrence, N. D.
- Affiliations
- University of Cambridge (GB)
- Cambridge Spark, Cambridge, United Kingdom
- Venue
- ACM Computing Surveys
- Published
- 2022-12-07
- Volume
- 55
- Issue
- 6
- Pages
- 1-29
- ISSN
- 0360-0300 (ISSN-L)
- 1557-7341
- OpenAlex
- W3104128335
- Semantic Scholar
- b1f574c47d0b6e3032246770b9cbebb9c7bd0c7f
- Cited by
- 638
- References
- 128
- Citations by year
- 2026: 149 · 2025: 156 · 2024: 166 · 2023: 93 · 2022: 46
- FWCI
- 114.17
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- ACM Computing Surveys (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- White Rose Research Online (University of Leeds, The University of Sheffield, University of York) (submittedVersion)
- Apollo (University of Cambridge) (submittedVersion)
- arXiv (Cornell University)
- Apollo
- Funders
- Alan Turing Institute
- UK Research & Innovation
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
- Physical Sciences › Computer Science › Artificial Intelligence › Artificial Intelligence Applications
- Physical Sciences › Computer Science › Information Systems › Cloud Computing and Resource Management
- Keywords
- machine learning deployment, production systems, deployment challenges, industry case studies
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.