Challenges in Deploying Machine Learning: A Survey of Case Studies
Open accessThe DOI matches its Crossref record; 127 of the 128 references it cites were found in a registry or an index.
Abstract
In recent years, machine learning has transitioned from a field of academic research interest to a field capable of solving real-world business problems. However, the deployment of machine learning models in production systems can present a number of issues and concerns. This survey reviews published reports of deploying machine learning solutions in a variety of use cases, industries, and applications and extracts practical considerations corresponding to stages of the machine learning deployment workflow. By mapping found challenges to the steps of the machine learning deployment workflow, we show that practitioners face issues at each stage of the deployment process. The goal of this article is to lay out a research agenda to explore approaches addressing these challenges.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“Compared with the first step, continual monitoring is used to identify data distribution drift, control model quality, and trigger system alarms when an ML model deviates from its normal behavior in the target setting [15].”
From Toward real‐world deployment of machine learning for health care: External validation, continual monitoring, and randomized clinical trials (2024) · cites this paper as [15]
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“Our literature review was conducted using a systematic mapping protocol [54] in order to select a small set of relevant articles from the much larger collection of literature emerging at the intersection of data quality and ML.”
From A Survey of Data Quality Requirements That Matter in ML Development Pipelines (2023) · cites this paper as [54]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325 · cited 2×
“They took one of the state-of-the-art models in the field, BERT [47], and found out that depending on the chosen model size full training procedure can cost anywhere between $50k and $1.”
BackgroundQuinlan, J. R. (1986) · Machine Learning · doi:10.1007/bf00116251 · Cited by 21,439 · cited 1×
“For that reason decision trees (DT) [108], which can be considered a fairly basic ML algorithm, are widely used in practice.”
MethodArmstrong, J. S., & Georgoff, D. M. (1988) · International Journal of Forecasting · doi:10.1016/0169-2070(88)90019-2 · Cited by 17,894
Rudin, C. (2019) · Nature Machine Intelligence · doi:10.1038/s42256-019-0048-x · Cited by 10,561 · cited 1×
“Rudin [111] called the ML community to stop using black box models and explaining their behavior afterward, and instead design models that are inherently interpretable.”
BackgroundOn the Dangers of Stochastic Parrots: Can Language Models Be Too Big? 🦜
Bender, E. M., Gebru, T., McMillan-Major, A., & Shmitchell, S. (2021) · Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3442188.3445922 · Cited by 6,988
Fully homomorphic encryption using ideal lattices
Gentry, C. (2009) · Proceedings of the forty-first annual ACM symposium on Theory of computing · doi:10.1145/1536414.1536440 · Cited by 6,806 · cited 1×
“Some of the ways this trade-of is resolved in practice are diferential privacy that explicitly corrupts the data [42], homomorphic encryption that restricts the class of learning algorithm but allows for training on encrypted data [50], and federated learning that distributes training across personal devices to preser…”
BackgroundPractical Bayesian Optimization of Machine Learning Algorithms
Snoek, J., Larochelle, H., & Adams, R. P. (2012) · arXiv · doi:10.48550/arxiv.1206.2944 · Cited by 6,691 · cited 1×
“Even approaches like Hyperband [78] or Bayesian optimization [129], that are speciically designed to minimize the number of training cycles needed, are not yet able to deal with the high dimensional searches that emerge when many hyper-parameters are involved.”
Dwork, C. (2006) · Lecture Notes in Computer Science · doi:10.1007/11787006_1 · Cited by 5,291 · cited 1×
“Some of the ways this tradeoff is resolved in practice are differential privacy that explicitly corrupts the data [54], homomorphic encryption that restricts the class of learning algorithm but allows for training on encrypted data [55], and federated learning that distributes training across personal devices to prese…”
BackgroundMembership Inference Attacks Against Machine Learning Models
Shokri, R., Stronati, M., Song, C., & Shmatikov, V. (2017) · 2017 IEEE Symposium on Security and Privacy (SP) · doi:10.1109/sp.2017.41 · Cited by 4,754 · cited 1×
“They veriied it on models trained on leading ML-as-a-service providers [127], achieving anywhere from 70% up to 94% accuracy of membership inference.”
BackgroundA survey on concept drift adaptation
Gama, J., Žliobaitė, I., Bifet, A., Pechenizkiy, M., & Bouchachia, A. (2014) · ACM Computing Surveys · doi:10.1145/2523813 · Cited by 3,735 · cited 1×
“We refer interested readers to the works on using domain adaptation [48], meta-learning [46] and transfer learning [152, 157] as ways of addressing dataset shift.”
Method
Research works printed but not linked: 26
- Arif Cam, Michael Chui, and Bryce Hall. 2019. Global AI survey: AI proves its worth, but few scale impact. McKinsey Analytics (2019). https://www.mckinsey.com/featured-insights/artificial-intelligence/global-ai-survey-ai-proves-its-worth-but-few-scale-impact. Accessed from 13/06/2022.
- Royal Society (Great Britain). 2017. Machine Learning: The Power and Promise of Computers that Learn by Example: an Introduction. Royal Society.
- Ben Lorica and Nathan Paco. 2018. The State of Machine Learning Adoption in the Enterprise. O’Reilly Media.
- Colin Shearer. 2000. The CRISP-DM model: The new blueprint for data mining. J. Data Warehous. 5, 4 (2000), 13–22.
- Rajashree Y. Patil and R. V. Kulkarni. 2012. A review of data cleaning algorithms for data warehouse systems. Int. J. Comput. Sci. Inf. Technol. 3, 5 (2012), 5212–5214.
- 10.1109/TITS.2019.2962338 · DOI · Paper page
- Diego Charrez. 2019. NeurIPS 2019 Stats. Retrieved from https://medium.com/@dcharrezt/neurips-2019-stats-c91346d31c8f.
- 10.1023/A:1022643204877 · doi:10.1023/a:1022643204877 · Paper page
- 10.5555/3122009.3242042 · doi:10.5555/3122009.3242042
- Bobak Shahriari, Alexandre Bouchard-Cote, and Nando Freitas. 2016. Unbounded Bayesian optimization via regularization. In Proceedings of the 19th International Conference on Artificial Intelligence and Statistics (Proceedings of Machine Learning Research), Arthur Gretton and Christian C. Robert (Eds.), Vol. 51. PMLR, 1168–1176. Retrieved from http://proceedings.mlr.press/v51/shahriari16.html.
- Ram Ananth, Seph Mard, and Peter Simon. 2019. Opening the “Black Box”: The path to deployment of AI models in banking, white paper. DataRobot (2019). https://www.datarobot.com/resources/opening-the-black-box/. Accessed from 13/06/2022.
- Robert Soden Dennis Wagenaar Dave Luo and Annegien Tijssen. 2019. Taking Ethics Fairness and Bias Seriously in Machine Learning for Disaster Risk Management. arxiv:cs.CY/1912.05538.
- Nantheera Anantrasirichai and David Bull. 2021. Artificial intelligence in the creative industries: A review. Artif. Intell. Rev. (2021), 1–68.
- D. Malan et al. 2016. Values and the fourth industrial revolution: Connecting the dots between value, values, profit and purpose. In Proceedings of the World Economic Forum.
- Daniel Malan. 2018. The law can’t keep up with new tech. Here’s how to close the gap. In Proceedings of the World Economic Forum.
- Ramprakash Ramamoorthy P. Satya Madhuri and Malini Christina Raj. 2019. AI from labs to production—challenges and learnings. USENIX Association. https://www.usenix.org/conference/opml19/presentation/madhuri. Accessed from 13/06/2022.
- Oscar Schwartz. 2019. In 2016 Microsoft’s racist chatbot revealed the dangers of online conversation. IEEE Spectr. 11 (2019).
- Florian Tramèr, Fan Zhang, Ari Juels, Michael K. Reiter, and Thomas Ristenpart. 2016. Stealing machine learning models via prediction APIs. In Proceedings of the 25th USENIX Security Symposium (USENIX Security’16). 601–618.
- Jeremy Hermann and Mike Del Balso. 2017. Meet Michelangelo: Uber’s machine learning platform. Retrieved from https://eng.uber.com/michelangelo.
- Matei Zaharia, Andrew Chen, Aaron Davidson, Ali Ghodsi, Sue Ann Hong, Andy Konwinski, Siddharth Murching, Tomas Nykodym, Paul Ogilvie, Mani Parkhe, et al. 2018. Accelerating the machine learning lifecycle with MLflow.IEEE Data Eng. Bull. 41, 4 (2018), 39–45.
- Nick Hynes, D. Sculley, and Michael Terry. 2017. The Data Linter: Lightweight, automated sanity checking for ML data sets. In Proceedings of the NIPS MLSys Workshop.
- Matthias Feurer, Aaron Klein, Katharina Eggensperger, Jost Springenberg, Manuel Blum, and Frank Hutter. 2015. Efficient and robust automated machine learning. In Advances in Neural Information Processing Systems, C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Garnett (Eds.), Vol. 28. Curran Associates, Inc. Retrieved from https://proceedings.neurips.cc/paper/2015/file/11d0e6287202fced83f79975ec59a3a6-Paper.pdf.
- Randal S. Olson and Jason H. Moore. 2016. TPOT: A tree-based pipeline optimization tool for automating machine learning. In Proceedings of the Workshop on Automatic Machine Learning. PMLR, 66–74.
- Stephan Rabanser, Stephan Günnemann, and Zachary Chase Lipton. 2019. Failing loudly: An empirical study of methods for detecting dataset shift. In Proceedings of the Neural Information Processing Systems Foundation Conference.
- Chelsea Finn, Pieter Abbeel, and Sergey Levine. 2017. Model-agnostic meta-learning for fast adaptation of deep networks. In Proceedings of the International Conference on Machine Learning. PMLR, 1126–1135.
- A. Zujus. 2018. AI project development–how project managers should prepare. Tratto il giorno Agosto 31 (2018), 2019.
Cited by
Items 101–200 of 737
AI Assistants: A Framework for Semi-Automated Data Wrangling
Petříček, T., van den Burg, G. J. J., Nazábal, A., Ceritli, T., Jiménez-Ruiz, E., & Williams, C. K. I. (2022) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2022.3222538 · Cited by 21
“The problem of data wrangling has been studied by both practitioners [1], [2], [37] and academics [12], [38], [39], [40].”
BackgroundGenerating psychological analysis tables for children's drawings using deep learning
Lee, M., Kim, Y., & Kim, Y.-K. (2023) · Data & Knowledge Engineering · doi:10.1016/j.datak.2023.102266 · Cited by 17
Development in times of hype: How freelancers explore Generative AI?
Dolata, M., Lange, N., & Schwabe, G. (2024) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3597503.3639111 · Cited by 35
Accelerating AI for science: open data science for science
Lawrence, N. D., & Montgomery, J. (2024) · Royal Society Open Science · doi:10.1098/rsos.231130 · Cited by 26
“Failure modes vary, arising at each stage of the AI development pipeline, from understanding user needs, to managing data quality, to maintaining performance levels in changeable environments or anticipating user interactions [11].”
BackgroundSelf-citationWarnett, S. J., Ntentos, E., & Zdun, U. (2024) · Journal of Systems and Software · doi:10.1016/j.jss.2024.112257 · Cited by 18
“Paleyes et al.’s [5] survey of case studies identifies practical challenges in ML system architecture, encompassing tools, services, and architectures within the deployment structure, as well as potential disparities between ML and data science, and software engineering practices.”
Burggräf, P., Wagner, J., Heinbach, B., Steinberg, F., Martínez, A. R. P., Schmallenbach, L., Garcke, J., Steffes-lai, D., & Wolter, M. (2021) · Procedia CIRP · doi:10.1016/j.procir.2021.11.108 · Cited by 17
Alwabisi, S. O., Ouni, R., & Saleem, K. (2022) · Electronics · doi:10.3390/electronics11234065 · Cited by 17
Safety Is Our Friend: A Federated Learning Framework Toward Driver’s State and Behavior Detection
Khoa, T. A., Trac, N. D., Tinh, V. P., Nam, N. H., Dang, D. N. M., Son, H. H., & Lam, P. D. (2023) · IEEE Transactions on Computational Social Systems · doi:10.1109/tcss.2023.3273727 · Cited by 21
“Nevertheless, DL networks encounter some interesting challenges [12].”
BackgroundSaleh, D. S., & Othman, M. S. (2023) · Baghdad Science Journal · doi:10.21123/bsj.2023.8544 · Cited by 21
“This could be due to patient-specific data, such as symptoms or blood test findings 64 .”
BackgroundPrompt2Model: Generating Deployable Models from Natural Language Instructions
Viswanathan, V., Zhao, C., Bertsch, A., Wu, T., & Neubig, G. (2023) · Conference on Empirical Methods in Natural Language Processing: System Demonstrations (EMNLP) · doi:10.18653/v1/2023.emnlp-demo.38 · Cited by 59
Identifying Citizen-Related Issues from Social Media Using LLM-Based Data Augmentation
dos Santos, V. G., Santos, G. L., Lynn, T., & Benatallah, B. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-61057-8_31 · Cited by 30
Tran, T.-A., Ruppert, T., & Abonyi, J. (2024) · Computers · doi:10.3390/computers13100252 · Cited by 19
Ogundairo, K. M., & Ayivi-Donkor, S. S. (2026) · IIARD INTERNATIONAL JOURNAL OF ECONOMICS AND BUSINESS MANAGEMENT · doi:10.56201/ijebm.v9.no10.2023.pg210.246 · Cited by 21
Ogundairo, K. M., & Ayivi-Donkor, S. S. (2026) · INTERNATIONAL JOURNAL OF ECONOMICS AND FINANCIAL MANAGEMENT · doi:10.56201/ijefm.v8.no8.2023.pg195.232 · Cited by 21
Scalable anomaly detection in manufacturing systems using an interpretable deep learning approach
Schlegl, T., Schlegl, S., West, N., & Deuse, J. (2021) · Procedia CIRP · doi:10.1016/j.procir.2021.11.261 · Cited by 15
State-of-the-Art and Challenges of Engineering ML- Enabled Software Systems in the Deep Learning Era
Assres, G. M., Bhandari, G. P., Shalaginov, A., Grønli, T., & Ghinea, G. (2025) · ACM Computing Surveys · doi:10.1145/3731597 · Cited by 21
“Version-based feedback, and incentives [74, 82].”
Cost-Optimised Machine Learning Model Comparison for Predictive Maintenance
Yang, Y., & Iqbal, M. Z. (2025) · Electronics · doi:10.3390/electronics14122497 · Cited by 24
Making Sense of AI Systems Development
Dolata, M., & Crowston, K. (2023) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2023.3338857 · Cited by 22
“Generative language models trained a year or two ago may underperform compared to recent LLMs like GPT or PaLM.”
Quality issues in machine learning software systems
Côté, P.-O., Nikanjam, A., Bouchoucha, R., Basta, I., Abidi, M., & Khomh, F. (2024) · Empirical Software Engineering · doi:10.1007/s10664-024-10536-7 · Cited by 27
“While there are research works describing the challenges in MLOps (Paleyes et al., 2022; Polyzotis et al., 2017, 2018; Schelter et al., 2015; Shankar et al., 2022; Steidl et al., 2023) and (anti-) patterns in MLSSs (Alahdab and Çalıklı, 2019; Bogner et al.”
BackgroundCausal inference in AI education: A primer
Forney, A., & Mueller, S. (2022) · Journal of Causal Inference · doi:10.1515/jci-2021-0048 · Cited by 26
Teaching Machine Learning as Part of Agile Software Engineering
Chenoweth, S., & Linos, P. K. (2023) · IEEE Transactions on Education · doi:10.1109/te.2023.3337343 · Cited by 14
“In2020,Paleyes et al. [11] summarized surveys of the deployment of ML systems.”
BackgroundMachine learning data practices through a data curation lens: An evaluation framework
Bhardwaj, E., Gujral, H., Wu, S., Zogheib, C., Maharaj, T., & Becker, C. (2024) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3630106.3658955 · Cited by 44
MLOps: A Multiple Case Study in Industry 4.0
Faubel, L., & Schmid, K. (2024) · IEEE International Conference on Emerging Technologies and Factory Automation (ETFA) · doi:10.1109/etfa61755.2024.10711136 · Cited by 24
“In this context, MLOps tools, frameworks, and architectural considerations play an important role [4], [8]–[10].”
SynDEc: A Synthetic Data Ecosystem
Karst, F., Li, M. M., & Leimeister, J. M. (2025) · Electronic Markets · doi:10.1007/s12525-024-00746-8 · Cited by 16
“However, even with automatic updates, the dataset may still be dominated by outdated fraud patterns, posing a risk to the algorithms (Paleyes et al., 2023).”
SensiX++: Bringing MLOps and Multi-tenant Model Serving to Sensory Edge Devices
Min, C., Mathur, A., Acer, U. G., Montanari, A., & Kawsar, F. (2023) · ACM Transactions on Embedded Computing Systems · doi:10.1145/3617507 · Cited by 23
“MLOps are a relatively new area of data engineering, and so far, most of the attention is focused on automated model training and primarily driven by the industry due to the challenges in running ML systems in the real world [54, 60].”
BackgroundMohammed, N. I., Jarde, A., Mackenzie, G. A., D’Alessandro, U., & Jeffries, D. J. (2022) · Frontiers in Public Health · doi:10.3389/fpubh.2021.772620 · Cited by 14
Towards containerized, reuse-oriented AI deployment platforms for cognitive IoT applications
Veiga, T., Asad, H. A., Kraemer, F. A., & Bach, K. (2022) · Future Generation Computer Systems · doi:10.1016/j.future.2022.12.029 · Cited by 20
Knowledge-Adaptation PriorsPreprint
Khan, M. E., & Swaroop, S. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.08769 · Cited by 18
Requirements and Reference Architecture for MLOps:Insights from IndustryPreprint
Kumara, I., Arts, R., Nucci, D. D., Kazman, R., Heuvel, W. J. V. D., & Tamburri, D. A. (2022) · doi:10.36227/techrxiv.21397413.v1 · Cited by 28
DroMOD: A Drone-Based Multi-Scope Object Detection System
Abdellatif, T., Sedrine, M. A., & Gacha, Y. (2023) · IEEE Access · doi:10.1109/access.2023.3253767 · Cited by 16
“Integration, monitoring and updating tasks are the three main concerns when deploying DL models [37].”
End-to-end lifecycle machine learning framework for predictive maintenance of critical equipment
Marchand, J., Laval, J., Sekhari, A., Cheutet, V., & Danielou, J.-B. (2025) · Enterprise Information Systems · doi:10.1080/17517575.2024.2448008 · Cited by 15
“ML introduces unique artefacts with unique characteristics (Paleyes, Urma, and Lawrence 2022): datasets, ML models and the derivated code.”
Design of a Machine Learning Approach to Anomaly Detection in Tyre-Road Interaction
Sakhnevych, A., Pasquino, N., & Sperlí, G. (2025) · IEEE Access · doi:10.1109/access.2025.3539923 · Cited by 13
A MLOps architecture for near real-time distributed Stream Learning operation deployment
Rodrigues, M. G., Viegas, E. K., Santin, A. O., & Enembreck, F. (2025) · Journal of Network and Computer Applications · doi:10.1016/j.jnca.2025.104169 · Cited by 19
Adversarial Attacks and Defenses for Deployed AI Models
Gupta, K. D., & Dasgupta, D. (2022) · IT Professional · doi:10.1109/mitp.2022.3180330 · Cited by 13
Detecting of a Patient's Condition From Clinical Narratives Using Natural Language Representation
Le, T.-D., Noumeir, R., Rambaud, J., Sans, G., & Jouvet, P. A. (2022) · IEEE Open Journal of Engineering in Medicine and Biology · doi:10.1109/ojemb.2022.3209900 · Cited by 22
Federated Learning Operations (FLOps): Challenges, Lifecycle and Approaches
Cheng, Q., & Long, G. (2022) · Conference on Technologies and Applications of Artificial Intelligence (TAAI) · doi:10.1109/taai57707.2022.00012 · Cited by 15
“Paleyes et al. [8] addresses the challenges in deploying machine learning models in production systems.”
BackgroundRequirements Are All You Need: The Final Frontier for End-User Software Engineering
Robinson, D., Cabrera, C., Gordon, A. D., Lawrence, N. D., & Mennen, L. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3708524 · Cited by 12
Self-citationOptimal selection of benchmarking datasets for unbiased machine learning algorithm evaluation
Pereira, J. L. J., Smith‐Miles, K., Muñoz, M. A., & Lorena, A. C. (2023) · Data Mining and Knowledge Discovery · doi:10.1007/s10618-023-00957-1 · Cited by 12
“But it is still common to face challenges when deploying ML-based systems in practice (Paleyes et al. 2022), making it not uncommon that results become disappointing.”
Achatz, J., Luković, M., Hilt, S., Lädrach, T., & Schubert, M. (2024) · Expert Systems with Applications · doi:10.1016/j.eswa.2023.123117 · Cited by 15
The State of Documentation Practices of Third-Party Machine Learning Models and Datasets
Oreamuno, E. L., Khan, R. M., Bangash, A. A., Stinson, C., & Adams, B. (2024) · IEEE Software · doi:10.1109/ms.2024.3366111 · Cited by 25
“Even large companies can fail to attend to the legalities of the code and data they acquire, as demonstrated by Google DeepMind, who faced fines for deploying a model that used NHS dataset [13].”
MLOps critical success factors - A systematic literature review
Mehmood, Y., Sabahat, N., & Ijaz, M. A. (2024) · VFAST Transactions on Software Engineering · doi:10.21015/vtse.v12i1.1747 · Cited by 20
Pourkamali‐Anaraki, F., Husseini, J. F., Pineda, E. J., Bednarcyk, B. A., & Stapleton, S. E. (2024) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2024.109436 · Cited by 14
Cai, C., Hong, S., Ma, M., Feng, H., Du, S., Chow, M., Teo, W. L.-L., Liu, S., & Fan, X. (2025) · Education and Information Technologies · doi:10.1007/s10639-025-13633-2 · Cited by 16
“Second, they need extensive fine-tuning on pre-existing labeled data (data that human annotators have classified), which is resource-intensive (Orescanin et al., 2023; Paleyes et al., 2022).”
Machine Learning Systems: A Survey from a Data-Oriented Perspective
Cabrera, C., Paleyes, A., Thodoroff, P., & Lawrence, N. D. (2025) · ACM Computing Surveys · doi:10.1145/3769292 · Cited by 19
“Cloud deployments imply the change of data ownership from end users to service providers.”
Self-citationBridging The "Adoption Gap": Why Supply Chain AI Fails Without Product Marketing Principles
Ogundairo, K. M., & Ayivi-Donkor, S. S. (2026) · INTERNATIONAL JOURNAL OF MARKETING AND COMMUNICATION STUDIES · doi:10.56201/ijmcs.v8.no5.2024.pg169.205 · Cited by 12
Hybrid modelling approaches in process intensification: A thorough review
Esposito, F., Caprio, U. D., Buzzi, S., Vermeire, F. H., & Leblebici, M. E. (2025) · Chemical Engineering and Processing - Process Intensification · doi:10.1016/j.cep.2025.110496 · Cited by 11
Prediction of ATFM impact for individual flights: A machine learning approach
Mas-Pujol, S., & Delgado, L. (2024) · Expert Systems with Applications · doi:10.1016/j.eswa.2024.124146 · Cited by 11
Muhsin, Z. J., Qahwaji, R., AlShawabkeh, M., AlRyalat, S. A., Bdour, M. D. A., & Al-Taee, M. A. (2024) · Eye and Vision · doi:10.1186/s40662-024-00394-1 · Cited by 11
“Moreover, most of these studies were conducted in an academic research setting [41], rather than being applied in clinical practice [42, 43].”
BackgroundMethodSecuring Federated Learning: Approaches, Mechanisms and Opportunities
Moshawrab, M., Adda, M., Bouzouane, A., Ibrahim, H., & Raad, A. (2024) · Electronics · doi:10.3390/electronics13183675 · Cited by 14
Security Challenges in Autonomous Systems Design
Hamad, M., & Steinhorst, S. (2025) · Communications in computer and information science · doi:10.1007/978-3-031-81981-0_13 · Cited by 12
Video enabled crowd monitoring system and analysis
Mehra, G., Joshi, A., Singh, A., Shuaib, M., Diwakar, M., & Singh, P. (2025) · International Conference on Intelligent and Innovative Technologies in Computing, Electrical and Electronics (IITCEE) · doi:10.1109/iitcee64140.2025.10915289 · Cited by 11
“This system proposes a technique for human detection using computer vision, which modifies the objective and interpretation of the task [6–8].”
Looper: An End-to-End ML Platform for Product Decisions
Markov, I. L., Wang, H., Kasturi, N., Singh, S., Garrard, M. R., Huang, Y., Yuen, S. W., Tran, S., Wang, Z., Glotov, I., et al. (2022) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3534678.3539059 · Cited by 27
“Improving user experience and system performance with ML remains challenging (Paleyes et al., 2020) as correlations in data found by ML models might not lead to causal improvements.”
BackgroundALDI++: Automatic and parameter-less discord and outlier detection for building energy load profiles
Quintana, M., Stoeckmann, T., Park, J. Y., Turowski, M., Hagenmeyer, V., & Miller, C. (2022) · Energy and Buildings · doi:10.1016/j.enbuild.2022.112096 · Cited by 12
“This new “data-centric” paradigm focuses more on the issues of data collection, quality, and cleaning, particularly for real-world deployed systems [44], which also becomes apparent through its first ever competition held in September 2021 called Data-Centric AI7.”
BackgroundArtificial intelligence in pharmacovigilance: Do we need explainability?
Hauben, M. (2022) · Pharmacoepidemiology and Drug Safety · doi:10.1002/pds.5501 · Cited by 11
“While explainabil-ity may placate clinicians, it is subservient to other measures, such as early and ongoing engagement with clinicians 33 and validation in independent data sets to mitigate lack of generalizability due to potential selection bias, and overfitting of smaller clinical data sets.”
BackgroundNaser, M. Z. (2024) · Journal of Structural Engineering · doi:10.1061/jsendh.steng-12934 · Cited by 13
Improving Fairness using Vision-Language Driven Image Augmentation
Moreno D’Incà, Tzelepis, C., Patras, I., & Sebe, N. (2024) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv57701.2024.00463 · Cited by 15
“In this context, studying the behavior of deep learning models is crucial to avoid unwanted situations at inference time [26].”
Assessment Framework for Deployability of Machine Learning Models in Production
Heymann, H., Mende, H., Frye, M., & Schmitt, R. H. (2023) · Procedia CIRP · doi:10.1016/j.procir.2023.06.007 · Cited by 12
Koh, V. J. W., Lai, W. X., Tan, K. Z., Singh, N. B., Matchar, D. B., & Chan, A. W.-M. (2024) · GeroScience · doi:10.1007/s11357-024-01098-z · Cited by 13
“While ML models may demonstrate high accuracies, we should be cautious about interpreting such data as ML models are dependent on factors such as computational power, data quality and availability, expertise, and cost [69, 70].”
The Role of AI Enablers in Overcoming Impairments in 6G Networks
Saimler, M., İckin, S., Bernini, G., Toumi, N., Diamanti, M., Papavassiliou, S., Zivkovic, M., Akgül, Ö. U., & Khorsandi, B. M. (2024) · Joint European Conference on Networks and Communications & 6G Summit (EuCNC/6G Summit) · doi:10.1109/eucnc/6gsummit60053.2024.10597102 · Cited by 10
“Operational challenges in Machine Learning Operations (MLOps) [5] further complicate efficient training and inference, while 6G evolution emphasizes resource allocation and interoperability.”
BackgroundConcept-drift-adaptive anomaly detector for marine sensor data streams
Nguyen, N.-T., Heldal, R., & Pelliccione, P. (2024) · Internet of Things · doi:10.1016/j.iot.2024.101414 · Cited by 12
Ziyad, F., Alemayehu, H., Wolla, D. W., & Dadi, F. (2024) · The International Journal of Advanced Manufacturing Technology · doi:10.1007/s00170-024-14952-3 · Cited by 12
“Integrating super learning models into existing industrial systems and workflows can be difficult and may necessitate significant changes [67].”
Towards Visual Analytics for Explainable AI in Industrial Applications
Kucher, K., Zohrevandi, E., & Westin, C. (2025) · Analytics · doi:10.3390/analytics4010007 · Cited by 17
“…Ground The prior work accumulated within visualization and HCI can certainly be valuable in this regard [28,74,101,125,126], but even further sources should be considered, e.g., with respect to the challenges of deploying ML solutions [127] or integrating data scientists in software teams [128].”
Li, L., Sun, W., Gómez-Zamorano, L. Y., Liu, Z., Zhang, W., & Ma, H. (2025) · Polymers · doi:10.3390/polym17182541 · Cited by 14
“This procedure provides a comprehensive assessment of the model’s ability to generalize to new data [207,208].”
Wei, H., Pañeda, X. G., & Salvachúa, J. (2025) · Cluster Computing · doi:10.1007/s10586-025-05584-7 · Cited by 12
Kong, S. C., & Hou, C. (2025) · Computers and Education Artificial Intelligence · doi:10.1016/j.caeai.2025.100503 · Cited by 11
Ning, W., Tang, F., Zhao, M., & Kato, N. (2025) · IEEE Communications Surveys & Tutorials · doi:10.1109/comst.2025.3646200 · Cited by 15
“Furthermore, fault samples required for UHR scenarios (e.g., sudden deep fading) are extremely scarce in reality [143].”
Model Provenance Management in MLOps Pipeline
Mei, S., Liu, C., Wang, Q., & Su, H. (2022) · doi:10.1145/3512850.3512861 · Cited by 12
“In [5], the survey shows that practitioners face challenges at each stage of the deployment.”
BackgroundMiyajiwala, A., Ladkat, A., Jagadale, S., & Joshi, R. B. (2022) · Lecture notes in networks and systems · doi:10.1007/978-3-031-10464-0_42 · Cited by 16
“In this work, we aim to highlight the limitations of deep learning based text classification models in different practical scenarios [14][15].”
BackgroundMLTEing Models: Negotiating, Evaluating, and Documenting Model and System Qualities
Maffey, K. R., Dotterrer, K., Niemann, J., Cruickshank, I. J., Lewis, G. A., & Kästner, C. (2023) · IEEE/ACM International Conference on Software Engineering: New Ideas and Emerging Results (ICSE-NIER) · doi:10.1109/icse-nier58687.2023.00012 · Cited by 16
“Research into applications of ML systems and organizational processes used to implement them reveals that communication is a critical yet often neglected part of product development [20], [21].”
BackgroundMethodNaming the Pain in machine learning-enabled systems engineering
Kalinowski, M., Méndez, D., Giray, G., Alves, A. P. S., Azevedo, K., Escovedo, T., Villamizar, H., Lopes, H., Baldassarre, M. T., Wagner, S., et al. (2025) · Information and Software Technology · doi:10.1016/j.infsof.2025.107866 · Cited by 20
Alamin, M. A. A., & Uddin, G. (2021) · IEEE International Conference on Autonomous Systems (ICAS) · doi:10.1109/icas49788.2021.9551151 · Cited by 22
Does Data Repair Lead to Fair Models? Curating Contextually Fair Data To Reduce Model Bias
Agarwal, S., Muku, S., Anand, S., & Arora, C. (2022) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv51458.2022.00395 · Cited by 19
Yao, D., & de Soto, B. G. (2022) · IOP Conference Series Materials Science and Engineering · doi:10.1088/1757-899x/1218/1/012017 · Cited by 10
“…and PyTorch packages were launched in 2015 and 2017, respectively), which might not be compatible with the older versions of computer configuration; thus, the deployment of ML might necessitate the redesign of the relevant business processes, which is expensive for construction companies [33].”
BackgroundAlerTiger: Deep Learning for AI Model Health Monitoring at LinkedIn
Xu, Z., Wang, R., Balaji, G., Bundele, M., Liu, X., Liu, L., & Wang, T. (2023) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3580305.3599802 · Cited by 17
“In recent years, machine learning has evolved from a research area into productionized solutions for business problems [20].”
BackgroundValue of Analytics for Decision-Making: Role of Managers and Analysts
Srivastava, S., & Dixit, G. (2023) · Journal of Computer Information Systems · doi:10.1080/08874417.2023.2255557 · Cited by 15
“39”
InfluentialZürn, M., Kienzlen, A., Klingel, L., Lechler, A., Verl, A. W., Ren, S., & Xu, W. (2023) · IEEE International Conference on Automation Science and Engineering (CASE) · doi:10.1109/case56687.2023.10260646 · Cited by 9
“Synthetic data is especially helpful as annotations and creating data can be very time-consuming [15].”
Uncertainty quantification in neural network classifiers—A local linear approach
Malmström, M., Skog, I., Axehill, D., & Gustafsson, F. (2024) · Automatica · doi:10.1016/j.automatica.2024.111563 · Cited by 11
Navigating the Ethical Landscape: Considerations in Implementing AI-ML Systems in Human Resources
Basnet, S. (2024) · International Journal of Research Publication and Reviews · doi:10.55248/gengpi.5.0324.0755 · Cited by 14
Volkova, A., Hatamian, M., Anapyanova, A., & de Meer, H. (2024) · International Conference on Information Technology for Social Good (GoodIT) · doi:10.1145/3677525.3678651 · Cited by 13
Zhukova, A., von Sperl, L., Matt, C. E., & Gipp, B. (2024) · Knowledge and Information Systems · doi:10.1007/s10115-024-02212-5 · Cited by 14
“Therefore, integrating user experience (UX) and human-computer interaction (HCI) research into ML/NLP research addresses users’ needs, fuses their expertise, and increases intuitiveness, trans-parency, simplicity, and trust for the system users (Boukhelifa et al, 2018; Paleyes et al, 2022).”
Addressing challenges when adopting AI-driven Zero Defect Manufacturing: Insights from industry
Leberruyer, N., Ahlskog, M., & Bruch, J. (2024) · Procedia CIRP · doi:10.1016/j.procir.2024.10.064 · Cited by 12
Protschky, D., Lämmermann, L., Hofmann, P., & Urbach, N. (2025) · IEEE Access · doi:10.1109/access.2025.3534628 · Cited by 15
“However, the research into monitoring ML applications is just beginning to understand what is important when monitoring ML applications [47].”
A comparative analysis of ML techniques for bug report classification
Laiq, M., bin Ali, N., Börstler, J., & Engström, E. (2025) · Journal of Systems and Software · doi:10.1016/j.jss.2025.112457 · Cited by 14
Cloud-Native Scheduling and Resource Orchestration: A Deep Dive into AI-Driven Approaches
Dias, T., Ferreira, L., Fevereiro, D., Rosa, L., Cordeiro, L., & Fernandes, J. (2025) · IFIP advances in information and communication technology · doi:10.1007/978-3-031-97317-8_8 · Cited by 11
Automatic classification of construction accident reports using BERTopic-GLDA approach
Wang, J., Qu, Z., Wu, S., Skitmore, M., & Ma, L. (2025) · Engineering Construction & Architectural Management · doi:10.1108/ecam-04-2025-0692 · Cited by 11
Vytla, S. K. (2024) · International Journal of Computational and Experimental Science and Engineering · doi:10.22399/ijcesen.5393 · Cited by 10
“The evidence for this pattern is well documented across enterprise AI deployments [1].”
Makoni, T., & Chikobvu, D. (2023) · Data · doi:10.3390/data8050078 · Cited by 9
“However, DNN models can be more challenging to use due to their high cost and the need for optimal architecture and hyper-parameter tuning, as noted by [31,32].”
BackgroundIndustrial adoption of machine learning techniques for early identification of invalid bug reports
Laiq, M., bin Ali, N., Börstler, J., & Engström, E. (2024) · Empirical Software Engineering · doi:10.1007/s10664-024-10502-3 · Cited by 18
“– Regular model updates: After the model is deployed, it is necessary to update it to ensure that it reflects recent trends in the data (Paleyes et al. 2022), that is, the ML model used for predicting the validity of bug reports should be updated at regular intervals to adapt to changing data…”
BackgroundMethodInfluentialAdapting to Change: The Effects of Case Study Approaches on Problem-Solving Skills
Ahmar, D. S., Azzajjad, M. F., & Ahmar, A. S. (2023) · ARRUS Journal of Mathematics and Applied Science · doi:10.35877/mathscience2206 · Cited by 13
“Exercises using case studies give practice in dealing with complicated situations, recognizing crucial aspects, and proposing integrated solutions (Paleyes et al., 2023).”
Lin, C.-T., & Huang, S.-J. (2024) · Information and Software Technology · doi:10.1016/j.infsof.2024.107449 · Cited by 13
Sumarlin, T., & Qosidah, N. (2024) · Journal of Management and Informatics · doi:10.51903/jmi.v3i2.35 · Cited by 10
Analysis of Sector-Specific Responses of AI Generative
Mohapatra, H., & Mishra, S. R. (2024) · Studies in computational intelligence · doi:10.1007/978-981-97-8460-8_9 · Cited by 11
Song, Y., Ding, L., Peng, J., Song, L., & Zhang, X. (2025) · Biomedical Signal Processing and Control · doi:10.1016/j.bspc.2024.107472 · Cited by 9
HLS-compiled PYNQ-based cardiac arrhythmia detection system leveraging quantized ECG beat images
Mangaraj, S., Mahapatra, K., & Ari, S. (2025) · Biomedical Signal Processing and Control · doi:10.1016/j.bspc.2025.108063 · Cited by 10
Sklavenitis, D., & Kalles, D. (2025) · Applied Sciences · doi:10.3390/app15137165 · Cited by 15
Kazim, M. Z., Pirim, H., Le, C., Le, T., & Yadav, O. P. (2025) · Sustainable Energy Grids and Networks · doi:10.1016/j.segan.2025.101969 · Cited by 8
Babu, A., Šutka, A., & Mandal, D. (2025) · Advanced Materials Technologies · doi:10.1002/admt.202501635 · Cited by 10
Bozovic, A., Jovanović, L., Dobrojević, M., Antonijević, M., Bačanin, N., Desnica, E., Simic, V., & Zivkovic, M. Z. (2025) · The Journal of Supercomputing · doi:10.1007/s11227-025-08082-0 · Cited by 8
“Although AI and ML techniques have become increasingly valuable for predicting and monitoring tool wear, their effectiveness is often constrained by several inherent challenges [16, 17].”
Chen, C., Tan, Y., Liu, Y., Sun, M., Wang, Z. L., Zheng, X., Shen, Z., & Zhao, M. (2026) · Environmental Research · doi:10.1016/j.envres.2026.123853 · Cited by 11
Metadata
- Authors
- Paleyes, A., Urma, R.-G., & Lawrence, N. D.
- Affiliations
- University of Cambridge (GB)
- Cambridge Spark, Cambridge, United Kingdom
- Venue
- ACM Computing Surveys
- Published
- 2022-12-07
- Volume
- 55
- Issue
- 6
- Pages
- 1-29
- ISSN
- 0360-0300 (ISSN-L)
- 1557-7341
- OpenAlex
- W3104128335
- Semantic Scholar
- b1f574c47d0b6e3032246770b9cbebb9c7bd0c7f
- Cited by
- 638
- References
- 128
- Citations by year
- 2026: 149 · 2025: 156 · 2024: 166 · 2023: 93 · 2022: 46
- FWCI
- 114.17
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- ACM Computing Surveys (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- White Rose Research Online (University of Leeds, The University of Sheffield, University of York) (submittedVersion)
- Apollo (University of Cambridge) (submittedVersion)
- arXiv (Cornell University)
- Apollo
- Funders
- Alan Turing Institute
- UK Research & Innovation
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
- Physical Sciences › Computer Science › Artificial Intelligence › Artificial Intelligence Applications
- Physical Sciences › Computer Science › Information Systems › Cloud Computing and Resource Management
- Keywords
- machine learning deployment, production systems, deployment challenges, industry case studies
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.