Challenges in Deploying Machine Learning: A Survey of Case Studies
Open accessThe DOI matches its Crossref record; 127 of the 128 references it cites were found in a registry or an index.
Abstract
In recent years, machine learning has transitioned from a field of academic research interest to a field capable of solving real-world business problems. However, the deployment of machine learning models in production systems can present a number of issues and concerns. This survey reviews published reports of deploying machine learning solutions in a variety of use cases, industries, and applications and extracts practical considerations corresponding to stages of the machine learning deployment workflow. By mapping found challenges to the steps of the machine learning deployment workflow, we show that practitioners face issues at each stage of the deployment process. The goal of this article is to lay out a research agenda to explore approaches addressing these challenges.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“Compared with the first step, continual monitoring is used to identify data distribution drift, control model quality, and trigger system alarms when an ML model deviates from its normal behavior in the target setting [15].”
From Toward real‐world deployment of machine learning for health care: External validation, continual monitoring, and randomized clinical trials (2024) · cites this paper as [15]
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“Our literature review was conducted using a systematic mapping protocol [54] in order to select a small set of relevant articles from the much larger collection of literature emerging at the intersection of data quality and ML.”
From A Survey of Data Quality Requirements That Matter in ML Development Pipelines (2023) · cites this paper as [54]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325 · cited 2×
“They took one of the state-of-the-art models in the field, BERT [47], and found out that depending on the chosen model size full training procedure can cost anywhere between $50k and $1.”
BackgroundQuinlan, J. R. (1986) · Machine Learning · doi:10.1007/bf00116251 · Cited by 21,439 · cited 1×
“For that reason decision trees (DT) [108], which can be considered a fairly basic ML algorithm, are widely used in practice.”
MethodArmstrong, J. S., & Georgoff, D. M. (1988) · International Journal of Forecasting · doi:10.1016/0169-2070(88)90019-2 · Cited by 17,894
Rudin, C. (2019) · Nature Machine Intelligence · doi:10.1038/s42256-019-0048-x · Cited by 10,561 · cited 1×
“Rudin [111] called the ML community to stop using black box models and explaining their behavior afterward, and instead design models that are inherently interpretable.”
BackgroundOn the Dangers of Stochastic Parrots: Can Language Models Be Too Big? 🦜
Bender, E. M., Gebru, T., McMillan-Major, A., & Shmitchell, S. (2021) · Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3442188.3445922 · Cited by 6,988
Fully homomorphic encryption using ideal lattices
Gentry, C. (2009) · Proceedings of the forty-first annual ACM symposium on Theory of computing · doi:10.1145/1536414.1536440 · Cited by 6,806 · cited 1×
“Some of the ways this trade-of is resolved in practice are diferential privacy that explicitly corrupts the data [42], homomorphic encryption that restricts the class of learning algorithm but allows for training on encrypted data [50], and federated learning that distributes training across personal devices to preser…”
BackgroundPractical Bayesian Optimization of Machine Learning Algorithms
Snoek, J., Larochelle, H., & Adams, R. P. (2012) · arXiv · doi:10.48550/arxiv.1206.2944 · Cited by 6,691 · cited 1×
“Even approaches like Hyperband [78] or Bayesian optimization [129], that are speciically designed to minimize the number of training cycles needed, are not yet able to deal with the high dimensional searches that emerge when many hyper-parameters are involved.”
Dwork, C. (2006) · Lecture Notes in Computer Science · doi:10.1007/11787006_1 · Cited by 5,291 · cited 1×
“Some of the ways this tradeoff is resolved in practice are differential privacy that explicitly corrupts the data [54], homomorphic encryption that restricts the class of learning algorithm but allows for training on encrypted data [55], and federated learning that distributes training across personal devices to prese…”
BackgroundMembership Inference Attacks Against Machine Learning Models
Shokri, R., Stronati, M., Song, C., & Shmatikov, V. (2017) · 2017 IEEE Symposium on Security and Privacy (SP) · doi:10.1109/sp.2017.41 · Cited by 4,754 · cited 1×
“They veriied it on models trained on leading ML-as-a-service providers [127], achieving anywhere from 70% up to 94% accuracy of membership inference.”
BackgroundA survey on concept drift adaptation
Gama, J., Žliobaitė, I., Bifet, A., Pechenizkiy, M., & Bouchachia, A. (2014) · ACM Computing Surveys · doi:10.1145/2523813 · Cited by 3,735 · cited 1×
“We refer interested readers to the works on using domain adaptation [48], meta-learning [46] and transfer learning [152, 157] as ways of addressing dataset shift.”
Method
Research works printed but not linked: 26
- Arif Cam, Michael Chui, and Bryce Hall. 2019. Global AI survey: AI proves its worth, but few scale impact. McKinsey Analytics (2019). https://www.mckinsey.com/featured-insights/artificial-intelligence/global-ai-survey-ai-proves-its-worth-but-few-scale-impact. Accessed from 13/06/2022.
- Royal Society (Great Britain). 2017. Machine Learning: The Power and Promise of Computers that Learn by Example: an Introduction. Royal Society.
- Ben Lorica and Nathan Paco. 2018. The State of Machine Learning Adoption in the Enterprise. O’Reilly Media.
- Colin Shearer. 2000. The CRISP-DM model: The new blueprint for data mining. J. Data Warehous. 5, 4 (2000), 13–22.
- Rajashree Y. Patil and R. V. Kulkarni. 2012. A review of data cleaning algorithms for data warehouse systems. Int. J. Comput. Sci. Inf. Technol. 3, 5 (2012), 5212–5214.
- 10.1109/TITS.2019.2962338 · DOI · Paper page
- Diego Charrez. 2019. NeurIPS 2019 Stats. Retrieved from https://medium.com/@dcharrezt/neurips-2019-stats-c91346d31c8f.
- 10.1023/A:1022643204877 · doi:10.1023/a:1022643204877 · Paper page
- 10.5555/3122009.3242042 · doi:10.5555/3122009.3242042
- Bobak Shahriari, Alexandre Bouchard-Cote, and Nando Freitas. 2016. Unbounded Bayesian optimization via regularization. In Proceedings of the 19th International Conference on Artificial Intelligence and Statistics (Proceedings of Machine Learning Research), Arthur Gretton and Christian C. Robert (Eds.), Vol. 51. PMLR, 1168–1176. Retrieved from http://proceedings.mlr.press/v51/shahriari16.html.
- Ram Ananth, Seph Mard, and Peter Simon. 2019. Opening the “Black Box”: The path to deployment of AI models in banking, white paper. DataRobot (2019). https://www.datarobot.com/resources/opening-the-black-box/. Accessed from 13/06/2022.
- Robert Soden Dennis Wagenaar Dave Luo and Annegien Tijssen. 2019. Taking Ethics Fairness and Bias Seriously in Machine Learning for Disaster Risk Management. arxiv:cs.CY/1912.05538.
- Nantheera Anantrasirichai and David Bull. 2021. Artificial intelligence in the creative industries: A review. Artif. Intell. Rev. (2021), 1–68.
- D. Malan et al. 2016. Values and the fourth industrial revolution: Connecting the dots between value, values, profit and purpose. In Proceedings of the World Economic Forum.
- Daniel Malan. 2018. The law can’t keep up with new tech. Here’s how to close the gap. In Proceedings of the World Economic Forum.
- Ramprakash Ramamoorthy P. Satya Madhuri and Malini Christina Raj. 2019. AI from labs to production—challenges and learnings. USENIX Association. https://www.usenix.org/conference/opml19/presentation/madhuri. Accessed from 13/06/2022.
- Oscar Schwartz. 2019. In 2016 Microsoft’s racist chatbot revealed the dangers of online conversation. IEEE Spectr. 11 (2019).
- Florian Tramèr, Fan Zhang, Ari Juels, Michael K. Reiter, and Thomas Ristenpart. 2016. Stealing machine learning models via prediction APIs. In Proceedings of the 25th USENIX Security Symposium (USENIX Security’16). 601–618.
- Jeremy Hermann and Mike Del Balso. 2017. Meet Michelangelo: Uber’s machine learning platform. Retrieved from https://eng.uber.com/michelangelo.
- Matei Zaharia, Andrew Chen, Aaron Davidson, Ali Ghodsi, Sue Ann Hong, Andy Konwinski, Siddharth Murching, Tomas Nykodym, Paul Ogilvie, Mani Parkhe, et al. 2018. Accelerating the machine learning lifecycle with MLflow.IEEE Data Eng. Bull. 41, 4 (2018), 39–45.
- Nick Hynes, D. Sculley, and Michael Terry. 2017. The Data Linter: Lightweight, automated sanity checking for ML data sets. In Proceedings of the NIPS MLSys Workshop.
- Matthias Feurer, Aaron Klein, Katharina Eggensperger, Jost Springenberg, Manuel Blum, and Frank Hutter. 2015. Efficient and robust automated machine learning. In Advances in Neural Information Processing Systems, C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Garnett (Eds.), Vol. 28. Curran Associates, Inc. Retrieved from https://proceedings.neurips.cc/paper/2015/file/11d0e6287202fced83f79975ec59a3a6-Paper.pdf.
- Randal S. Olson and Jason H. Moore. 2016. TPOT: A tree-based pipeline optimization tool for automating machine learning. In Proceedings of the Workshop on Automatic Machine Learning. PMLR, 66–74.
- Stephan Rabanser, Stephan Günnemann, and Zachary Chase Lipton. 2019. Failing loudly: An empirical study of methods for detecting dataset shift. In Proceedings of the Neural Information Processing Systems Foundation Conference.
- Chelsea Finn, Pieter Abbeel, and Sergey Levine. 2017. Model-agnostic meta-learning for fast adaptation of deep networks. In Proceedings of the International Conference on Machine Learning. PMLR, 1126–1135.
- A. Zujus. 2018. AI project development–how project managers should prepare. Tratto il giorno Agosto 31 (2018), 2019.
Cited by
Items 701–737 of 737
Towards Task-Agnostic Evaluation Framework for LLM-Embedded Applications
Widanapathiranage, D. (2026) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3774748.3787633
“Despite these challenges, practitioners continue to deploy LLM-embedded applications, often resorting to ad-hoc evaluation strategies that are time-consuming and inconsistent [23, 30].”
Cost-Aware Healing Optimization Policy for Self-Healing Machine Learning Pipelines
Santhosh Kumar, S., ARS, S. P., Kashyap, S. R., Reddy, Y. B., & Sudharsan, B. (2026) · International Conference on Emerging Technologies in Computing and Communication (ETCC) · doi:10.1109/etcc69750.2026.11681955
“Recent studies have also highlighted deployment and maintenance challenges in production machine learning systems [16].”
Leveraging Machine Learning Algorithms in Decision Support Systems for Business Enterprises
Hamada, M. A., Abiche, A. E., & Hamada, G. M. (2026) · International Conference on Machine Learning Technologies · doi:10.1109/icmlt69916.2026.11688625
“They assure that the implementation of machine learning extends beyond technical considerations, requiring interdisciplinary strategies to develop systems that are ethical, scalable, and practically effective [10].”
Bisht, S., Jangwal, T., Gurjar, S., Thakur, S., Mishra, N., & Hossain, S. (2026) · IEEE International Conference on Instrumentation (INSTCon) · doi:10.1109/instcon69741.2026.11691936
“Recent surveys of case literature [16] refers to the dataset shift, distribution mismatch and label bias [17] as the main causes of ML failures in manufacturing settings.”
A knowledge-driven framework for HEP model deployment on heterogeneous hardware
Mustofa, A., & Cheng, Y. (2026) · Radiation Detection Technology and Methods · doi:10.1007/s41605-026-00734-w
Sun, T., & Li, Y. (2026) · Mathematics · doi:10.3390/math14193523
A LAYERED REFERENCE FRAMEWORK FOR SCALABLE DATA ENGINEERING AND AI APPLICATIONS
Maddali, S. K. (2026) · World Journal of Advanced Research and Reviews · doi:10.30574/wjarr.2026.31.3.2489
Ramsøy, T. Z. (2026) · Frontiers in Human Neuroscience · doi:10.3389/fnhum.2026.1837923
“…episodes are an acute instance of a chronic condition: deployed systems accumulate maintenance obligations that are easy to defer and costly to ignore (Sculley et al., 2015), and the operational difficulty of detecting and correcting post-deployment decay is well documented (Paleyes et al., 2022).”
Structurally Causal Data Augmentation via Physics-Informed Neural Networks
Bidin, A. H. B. H., & Othman, N. A. B. (2026) · International Conference on Computing Communication Control and automation · doi:10.1109/iccubea69821.2026.11701974
“Process plants are engineered to operate within narrow, safe and optimal steady-state region [2], [5], [6].”
ARCHITECTURAL PATTERNS FOR SOFTWARE DESIGN OF INTELLIGENT SYSTEMS
Panasiuk, O., & Novak, D. S. (2026) · Computer Systems and Information Technologies · doi:10.31891/csit-2026-3-20
Ling, J. (2026) · ARAI Journal of Mobility Technology · doi:10.37285/ajmt.6.4.1
Asthma Risk Classification in Pediatrics Using Gaussian Naive Bayes
Othman, D. K. B. A., & Azman, A. (2026) · Journal of Telecommunication Electronic and Computer Engineering (JTEC) · doi:10.54554/jtec.2026.18.03.003
“However, it has also introduced new challenges associated with the limited RAM, processing capacity, and flash memory of embedded systems [21], [22].”
Zong, R., Zhao, Y., He, S., & Rahim, M. A. B. U. (2026) · Communications in computer and information science · doi:10.1007/978-3-032-39763-8_6
Vytla, S. K. (2023) · International Journal of Advances in Signal and Image Sciences · doi:10.29284/tkn8wh18
When 99.8% is Not Enough: Closed-Loop Evaluation Exposes the Metric Gap in Learned Microgrid Control
White, M., & Nadarajah, M. (2026) · IEEE International Conference on Automation Science and Engineering (CASE) · doi:10.1109/case69030.2026.11704193
“Deployment surveys show that failures can emerge through data pipelines, monitoring and feedback effects rather than model accuracy alone [2], motivating operational metrics that measure system-level quality under dynamic or uncertain conditions [9].”
Hidri, A., Alsaif, S. A., Alahmari, M., Alshehri, E., & Hidri, M. S. (2026) · IEEE Access · doi:10.1109/access.2026.3741402
Govindaraj, V., Krishnammal, P. M., Devi, S. S., Nettem, A. V., Vinoth Kumar, V., & Yizengaw, A. W. (2026) · Discover Computing · doi:10.1007/s10791-026-10502-5 · Cited by 1
A Portable Governed MLOps Pipeline for Day-Ahead District-Heating Demand Forecasting
Zhao, X., Ma, Z. G., & Jôrgensen, B. N. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-41864-7_2
AICybOps: A Framework for Operationalizing Machine-Learning Models in Cybersecurity
Leal, R., & Figueiredo, S. (2026) · Journal of Cybersecurity and Privacy · doi:10.3390/jcp6050175
Ranjan, A., Khan, T., Dohare, I., Alrslani, F. A. F., Alzahrani, S. M., & Ahmadian, A. (2026) · Future Generation Computer Systems · doi:10.1016/j.future.2026.108871
(2026) · Ìnformatika, kulʹtura, tehnìka. · doi:10.15276/ict.03.2026.44
Operationalising CAPA for AI: Evidence Requirements and Limits from a Production Case StudyPreprint
Kabir-Chisty, M. (2026) · Research Square · doi:10.21203/rs.3.rs-10893823/v1
Structural Barriers to the Deployment of AI Foundation Models
Hristov, H., Somova, E., & Fuchedzhiev, E. (2026) · Journal of Computer Science · doi:10.3844/jcssp.2026.3068.3081
Chen, Y., Tootkaboni, M. G., Kesminas, A., & Rajabifard, A. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.7615086
Ilyas, R., & Bijaksana, M. A. (2026) · Jurnal Teknik Informatika (Jutif) · doi:10.52436/1.jutif.2026.7.5.6297
From AI Pilot to Sustainable Value: A Prudential Framework for AI Commercialisation
Godbole, A. (2026) · International Journal of Research and Innovation in Applied Science · doi:10.51584/ijrias.2026.11090112
Research on AI Agent Maturity Evaluation Based on BP Neural Networks
Jia, Y., Shang, Y., Ma, B., Li, S., Gao, S., Li, X., & Li, J. (2026) · IEEE Information Technology and Mechatronics Engineering Conference (ITOEC) · doi:10.1109/itoec69754.2026.11732715
Li, C. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.7674874
Tanwar, G., Kumawat, S., Kumari, P., Prasad, M., Rai, A. K., & Agrawal, A. (2026) · Genetics and Molecular Research · doi:10.4238/2axf3427
Sharma, A., & Sharma, G. (2026) · Research Square · doi:10.21203/rs.3.rs-11530197/v1
Architecting an Automated Containerized MLOps Pipeline for Scalable Machine Learning Systems
Abdullah, Garg, S., & Kaur, K. (2026) · IEEE International Conference on Advances in Computing Research on Science Engineering and Technology (ACROSET) · doi:10.1109/acroset70960.2026.11742353
Speech Enhancement Using the Demucs Model for Noise Removal in Voice Recordings
Wicaksana, L. M., & Bianto, M. A. (2026) · Journal of Artificial Intelligence and Engineering Applications (JAIEA) · doi:10.59934/jaiea.v6i1.2889
TRINDADE, A., Ribeiro, M. H. D. M., & Barbosa, M. A. D. C. (2026) · Anais ... Encontro Nacional de Engenharia de Produção/Anais do Encontro Nacional de Engenharia de Produção · doi:10.14488/enegep2026_tn_wpg_441_2184_54104
Jenkins, M., Baker, T., & Hernandez, T. (2026) · Academic Journal of Applied Sciences · doi:10.54097/m33p3g15
From Navigation to Intent: A Capability-Exposure Framework for Agent-Ready Enterprise Portals
Gusain, S. (2026) · International Journal of Engineering Science and Information Technology · doi:10.52088/ijesty.v6i1.2058
Abuhamdah, M., Kchaou, H., & Ammar, B. (2026) · Procedia Computer Science · doi:10.1016/j.procs.2026.10.114
Metadata
- Authors
- Paleyes, A., Urma, R.-G., & Lawrence, N. D.
- Affiliations
- University of Cambridge (GB)
- Cambridge Spark, Cambridge, United Kingdom
- Venue
- ACM Computing Surveys
- Published
- 2022-12-07
- Volume
- 55
- Issue
- 6
- Pages
- 1-29
- ISSN
- 0360-0300 (ISSN-L)
- 1557-7341
- OpenAlex
- W3104128335
- Semantic Scholar
- b1f574c47d0b6e3032246770b9cbebb9c7bd0c7f
- Cited by
- 638
- References
- 128
- Citations by year
- 2026: 149 · 2025: 156 · 2024: 166 · 2023: 93 · 2022: 46
- FWCI
- 114.17
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- ACM Computing Surveys (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- White Rose Research Online (University of Leeds, The University of Sheffield, University of York) (submittedVersion)
- Apollo (University of Cambridge) (submittedVersion)
- arXiv (Cornell University)
- Apollo
- Funders
- Alan Turing Institute
- UK Research & Innovation
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
- Physical Sciences › Computer Science › Artificial Intelligence › Artificial Intelligence Applications
- Physical Sciences › Computer Science › Information Systems › Cloud Computing and Resource Management
- Keywords
- machine learning deployment, production systems, deployment challenges, industry case studies
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.