Challenges in Deploying Machine Learning: A Survey of Case Studies
Open accessThe DOI matches its Crossref record; 127 of the 128 references it cites were found in a registry or an index.
Abstract
In recent years, machine learning has transitioned from a field of academic research interest to a field capable of solving real-world business problems. However, the deployment of machine learning models in production systems can present a number of issues and concerns. This survey reviews published reports of deploying machine learning solutions in a variety of use cases, industries, and applications and extracts practical considerations corresponding to stages of the machine learning deployment workflow. By mapping found challenges to the steps of the machine learning deployment workflow, we show that practitioners face issues at each stage of the deployment process. The goal of this article is to lay out a research agenda to explore approaches addressing these challenges.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“Compared with the first step, continual monitoring is used to identify data distribution drift, control model quality, and trigger system alarms when an ML model deviates from its normal behavior in the target setting [15].”
From Toward real‐world deployment of machine learning for health care: External validation, continual monitoring, and randomized clinical trials (2024) · cites this paper as [15]
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“Our literature review was conducted using a systematic mapping protocol [54] in order to select a small set of relevant articles from the much larger collection of literature emerging at the intersection of data quality and ML.”
From A Survey of Data Quality Requirements That Matter in ML Development Pipelines (2023) · cites this paper as [54]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325 · cited 2×
“They took one of the state-of-the-art models in the field, BERT [47], and found out that depending on the chosen model size full training procedure can cost anywhere between $50k and $1.”
BackgroundQuinlan, J. R. (1986) · Machine Learning · doi:10.1007/bf00116251 · Cited by 21,439 · cited 1×
“For that reason decision trees (DT) [108], which can be considered a fairly basic ML algorithm, are widely used in practice.”
MethodArmstrong, J. S., & Georgoff, D. M. (1988) · International Journal of Forecasting · doi:10.1016/0169-2070(88)90019-2 · Cited by 17,894
Rudin, C. (2019) · Nature Machine Intelligence · doi:10.1038/s42256-019-0048-x · Cited by 10,561 · cited 1×
“Rudin [111] called the ML community to stop using black box models and explaining their behavior afterward, and instead design models that are inherently interpretable.”
BackgroundOn the Dangers of Stochastic Parrots: Can Language Models Be Too Big? 🦜
Bender, E. M., Gebru, T., McMillan-Major, A., & Shmitchell, S. (2021) · Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3442188.3445922 · Cited by 6,988
Fully homomorphic encryption using ideal lattices
Gentry, C. (2009) · Proceedings of the forty-first annual ACM symposium on Theory of computing · doi:10.1145/1536414.1536440 · Cited by 6,806 · cited 1×
“Some of the ways this trade-of is resolved in practice are diferential privacy that explicitly corrupts the data [42], homomorphic encryption that restricts the class of learning algorithm but allows for training on encrypted data [50], and federated learning that distributes training across personal devices to preser…”
BackgroundPractical Bayesian Optimization of Machine Learning Algorithms
Snoek, J., Larochelle, H., & Adams, R. P. (2012) · arXiv · doi:10.48550/arxiv.1206.2944 · Cited by 6,691 · cited 1×
“Even approaches like Hyperband [78] or Bayesian optimization [129], that are speciically designed to minimize the number of training cycles needed, are not yet able to deal with the high dimensional searches that emerge when many hyper-parameters are involved.”
Dwork, C. (2006) · Lecture Notes in Computer Science · doi:10.1007/11787006_1 · Cited by 5,291 · cited 1×
“Some of the ways this tradeoff is resolved in practice are differential privacy that explicitly corrupts the data [54], homomorphic encryption that restricts the class of learning algorithm but allows for training on encrypted data [55], and federated learning that distributes training across personal devices to prese…”
BackgroundMembership Inference Attacks Against Machine Learning Models
Shokri, R., Stronati, M., Song, C., & Shmatikov, V. (2017) · 2017 IEEE Symposium on Security and Privacy (SP) · doi:10.1109/sp.2017.41 · Cited by 4,754 · cited 1×
“They veriied it on models trained on leading ML-as-a-service providers [127], achieving anywhere from 70% up to 94% accuracy of membership inference.”
BackgroundA survey on concept drift adaptation
Gama, J., Žliobaitė, I., Bifet, A., Pechenizkiy, M., & Bouchachia, A. (2014) · ACM Computing Surveys · doi:10.1145/2523813 · Cited by 3,735 · cited 1×
“We refer interested readers to the works on using domain adaptation [48], meta-learning [46] and transfer learning [152, 157] as ways of addressing dataset shift.”
MethodEnergy and Policy Considerations for Deep Learning in NLP
Strubell, E., Ganesh, A., & McCallum, A. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1355 · Cited by 3,684 · cited 1×
“[49] regarding the impact the training of ML models has on the environment.”
BackgroundOn hyperparameter optimization of machine learning algorithms: Theory and practice
Yang, L., & Shami, A. (2020) · Neurocomputing · doi:10.1016/j.neucom.2020.07.061 · Cited by 3,232
Doyle, T. (2017) · The Information Society · doi:10.1080/01972243.2017.1354593 · Cited by 3,071
Model Inversion Attacks that Exploit Confidence Information and Basic Countermeasures
Fredrikson, M., Jha, S., & Ristenpart, T. (2015) · Proceedings of the 22nd ACM SIGSAC Conference on Computer and Communications Security · doi:10.1145/2810103.2813677 · Cited by 2,961 · cited 1×
“have shown that they could recover training data by exploiting models that report conidence values along with their predictions [47].”
BackgroundModel Cards for Model Reporting
Mitchell, M., Wu, S., Zaldivar, A., Barnes, P., Vasserman, L., Hutchinson, B., Spitzer, E., Raji, I. D., & Gebru, T. (2019) · Proceedings of the Conference on Fairness, Accountability, and Transparency · doi:10.1145/3287560.3287596 · Cited by 2,681 · cited 1×
“[91] proposed model cards, short documents that accompany trained models detailing their performance, intended use context, and other information that might be relevant for model’s application.”
BackgroundGebru, T., Morgenstern, J., Vecchione, B., Vaughan, J. W., Wallach, H., Daumé, H., & Crawford, K. (2018) · arXiv · doi:10.48550/arxiv.1803.09010 · Cited by 2,474
Gebru, T., Morgenstern, J., Vecchione, B., Vaughan, J. W., Wallach, H., III, H. D., & Crawford, K. (2021) · Communications of the ACM · doi:10.1145/3458723 · Cited by 2,469
(2016) · Cited by 2,334
Adversarial machine learning at scale
Kurakin, A., Goodfellow, I., & Bengio, S. (2016) · arXiv · doi:10.48550/arxiv.1611.01236 · Cited by 2,325 · cited 1×
“The field of adversarial machine learning studies the effect of such attacks against ML models and how to protect against them [115, 116].”
BackgroundARTIFICIAL INTELLIGENCE FOR THE REAL WORLD
(2023) · International Research Journal of Modernization in Engineering Technology and Science · doi:10.56726/irjmets42512 · Cited by 1,434 · cited 1×
“According to a recent global survey conducted by McKinsey, machine learning is increasingly adopted in standard business processes with nearly 25 percent year-over-year growth [30] and with growing interest from the general public, business leaders [36] and governments [26].”
MethodWild patterns: Ten years after the rise of adversarial machine learning
Biggio, B., & Roli, F. (2018) · Pattern Recognition · doi:10.1016/j.patcog.2018.07.023 · Cited by 1,409 · cited 1×
“The field of adversarial machine learning studies the effect of such attacks against ML models and how to protect against them [115, 116].”
BackgroundWachter, S., Mittelstadt, B., & Floridi, L. (2017) · International Data Privacy Law · doi:10.1093/idpl/ipx005 · Cited by 1,308
Software Engineering for Machine Learning: A Case Study
Amershi, S., Begel, A., Bird, C., DeLine, R., Gall, H., Kamar, E., Nagappan, N., Nushi, B., & Zimmermann, T. (2019) · 2019 IEEE/ACM 41st International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP) · doi:10.1109/icse-seip.2019.00042 · Cited by 1,135 · cited 2×
“Despite obvious onboarding and slow-start challenges, this approach was seen to bring long term benefits in terms of speed and quality of product delivery [12].”
BackgroundHyperband: A Novel Bandit-Based Approach to Hyperparameter Optimization
Li, L., Jamieson, K., DeSalvo, G., Rostamizadeh, A., & Talwalkar, A. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1603.06560 · Cited by 1,018 · cited 1×
“Even approaches like Hyperband [41] or Bayesian optimization [42], that are specifically designed to minimize the number of training cycles needed, are still not able to deal with certain problems due to the complexity of the models or the size of the datasets.”
Background(2015) · neural information processing systems · Cited by 912
Auto-Keras: An Efficient Neural Architecture Search System
Jin, H., Song, Q., & Hu, X. (2019) · Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining · doi:10.1145/3292500.3330648 · Cited by 891 · cited 2×
“There is a large variety of tools that provide general-purpose implementations of AutoML algorithms, such as Auto-keras [137], Auto-sklearn [138] or TPOT [139].”
MethodBeyond accuracy: behavioral testing of NLP models with CheckList
Ribeiro, M. T., Wu, T., Guestrin, C., & Singh, S. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.442 · Cited by 890
The Creation and Detection of Deepfakes: A Survey
Mirsky, Y., & Lee, W. (2022) · ACM Computing Surveys · doi:10.1145/3425780 · Cited by 863 · cited 1×
“Closely related to this question is the growing concern of fake content being generated with ML, such as deepfake images and video, which can be easily used for the wrong purposes [95].”
BackgroundThe Creation and Detection of Deepfakes: A Survey
Mirsky, Y., & Lee, W. (2020) · arXiv · doi:10.48550/arxiv.2004.11138 · Cited by 863 · cited 1×
“Closely related to this question is the growing concern of fake content being generated with ML, such as deepfake images and video, which can be easily used for the wrong purposes [95].”
BackgroundManipulating Machine Learning: Poisoning Attacks and Countermeasures for Regression Learning
Jagielski, M., Oprea, A., Biggio, B., Liu, C., Nita-Rotaru, C., & Li, B. (2018) · 2018 IEEE Symposium on Security and Privacy (SP) · doi:10.1109/sp.2018.00057 · Cited by 861
Confident Learning: Estimating Uncertainty in Dataset Labels
Northcutt, C., Jiang, L., & Chuang, I. (2021) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.12125 · Cited by 795
Energy and Policy Considerations for Modern Deep Learning Research
Strubell, E., Ganesh, A., & McCallum, A. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i09.7123 · Cited by 774 · cited 1×
“[49] regarding the impact the training of ML models has on the environment.”
BackgroundDataset Shift in Machine Learning
Adams, N. (2010) · Journal of the Royal Statistical Society Series A: Statistics in Society · doi:10.1111/j.1467-985x.2009.00624_10.x · Cited by 694
Incremental learning from noisy data
Schlimmer, J. C., & Granger, R. H. (1986) · Machine Learning · doi:10.1007/bf00116895 · Cited by 653 · cited 1×
“evenwhenoccurringatmicroscopicscale, asZeniseket al. [60]showintheir research on predictive maintenance for wear and tear of industrial machinery. Even though concept drift has been known for decades [61], these examples show that it remains a critical problem for applications of ML today. On top of the question of wh…”
BackgroundChallenges of real-world reinforcement learning: definitions, benchmarks and analysis
Dulac-Arnold, G., Levine, N., Mankowitz, D. J., Li, J., Paduraru, C., Gowal, S., & Hester, T. (2021) · Machine Learning · doi:10.1007/s10994-021-05961-4 · Cited by 645 · cited 2×
“This point is emphasized further by the experiences from the ield of reinforcement learning [40], where use of simulations is a de-facto standard for training agents.”
BackgroundExplainable machine learning in deployment
Bhatt, U., Xiang, A., Sharma, S., Weller, A., Taly, A., Jia, Y., Ghosh, J., Puri, R., Moura, J. M. F., & Eckersley, P. (2020) · Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency · doi:10.1145/3351095.3375624 · Cited by 633 · cited 2×
“[11] analyzed explainability as a feature of machine learning models deployed within enterprises, and found that it is a must-have requirement for most stakeholders, including executives, ML engineers, regulators, and others.”
BackgroundA survey on active learning and human-in-the-loop deep learning for medical image analysis
Budd, S., Robinson, E. C., & Kainz, B. (2021) · Medical Image Analysis · doi:10.1016/j.media.2021.102062 · Cited by 606 · cited 2×
“[30] show that interface design directly impacts the quality of applications built to collect annotations for unlabeled data.”
BackgroundIoT-Based Big Data Storage Systems in Cloud Computing: Perspectives and Challenges
Cai, H., Xu, B., Jiang, L., & Vasilakos, A. V. (2017) · IEEE Internet of Things Journal · doi:10.1109/jiot.2016.2619369 · Cited by 588 · cited 1×
“[19] for further discussion of big data storage.”
BackgroundFederated Optimization:Distributed Optimization Beyond the Datacenter
Konečný, J., McMahan, B., & Ramage, D. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1511.03575 · Cited by 544 · cited 1×
“Some of the ways this trade-of is resolved in practice are diferential privacy that explicitly corrupts the data [42], homomorphic encryption that restricts the class of learning algorithm but allows for training on encrypted data [50], and federated learning that distributes training across personal devices to preser…”
BackgroundSoK: Security and Privacy in Machine Learning
Papernot, N., McDaniel, P., Sinha, A., & Wellman, M. P. (2018) · 2018 IEEE European Symposium on Security and Privacy (EuroS&P) · doi:10.1109/eurosp.2018.00035 · Cited by 502 · cited 1×
“As more businesses start using ML techniques on their users’ data, concerns are being raised over the privacy of data and how well individuals’ sensitive information is preserved over the course of the model training process [102].”
BackgroundExploiting machine learning to subvert your spam filter
Nelson, B. A., Barreno, M., Chi, F. J., Joseph, A. D., Rubinstein, B. I. P., Saini, U., Sutton, C. A., Tygar, J. D., & Xia, K. (2008) · Edinburgh Research Explorer (University of Edinburgh) · Cited by 481
Enterprise Data Analysis and Visualization: An Interview Study
Kandel, S., Paepcke, A., Hellerstein, J. M., & Heer, J. (2012) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2012.219 · Cited by 439 · cited 1×
“t. Availability of high quality tools is essential for conducting any kind of data analysis. One area that practitioners find particularly challenging in that regard is visualization for data profiling [27]. Data profiling refers to all activities associated with troubleshooting data quality, such as missing values, inco…”
BackgroundAssessing gender bias in machine translation: a case study with Google Translate
Prates, M. O. R., Avelar, P. H., & Lamb, L. C. (2020) · Neural Computing and Applications · doi:10.1007/s00521-019-04144-6 · Cited by 410 · cited 1×
“[89] show a strong tendency towards male defaults in popular online translation service in particular for fields typically associated with unbalanced gender distribution, such as STEM.”
BackgroundAssessing Gender Bias in Machine Translation -- A Case Study with Google Translate
Prates, M. O. R., Avelar, P. H. C., & Lamb, L. (2018) · arXiv · doi:10.48550/arxiv.1809.02208 · Cited by 409 · cited 1×
“[89] show a strong tendency towards male defaults in popular online translation service in particular for fields typically associated with unbalanced gender distribution, such as STEM.”
BackgroundUnderstanding artificial intelligence ethics and safetyPreprint
Leslie, D. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1906.05684 · Cited by 381 · cited 3×
“As stated in the report on ethical AI produced by the Alan Turing Institute [87], “it is essential to establish a continuous chain of human responsibility across the whole AI project delivery workflow”.”
BackgroundPacheco, F., Exposito, E., Gineste, M., Baudoin, C., & Aguilar, J. (2019) · IEEE Communications Surveys & Tutorials · doi:10.1109/comst.2018.2883147 · Cited by 368 · cited 1×
“[100], which surveys applications of machine learning to network traic classiication, with tasks such as protocol identiication or attack detection.”
BackgroundA Review of Deep Learning Methods and Applications for Unmanned Aerial Vehicles
Carrio, A., Sampedro, C., Rodriguez-Ramos, A., & Campoy, P. (2017) · Journal of Sensors · doi:10.1155/2017/3296874 · Cited by 356 · cited 1×
“This notion is exemplified by the field of unmanned aerial vehicles (UAV) [35].”
BackgroundAdversarial Machine Learning-Industry Perspectives
Kumar, R. S. S., Nystrom, M., Lambert, J., Marshall, A. D., Goertzel, M., Comissoneru, A., Swann, M., & Xia, S. (2020) · IEEE Security and Privacy Workshops (SPW) · doi:10.1109/spw50608.2020.00028 · Cited by 354 · cited 1×
“found that industry practitioners are not equipped to protect, detect and respond to attacks on their ML systems [117].”
BackgroundFaes, L., Wagner, S. K., Fu, D. J., Liu, X., Korot, E., Ledsam, J. R., Back, T., Chopra, R., Pontikos, N., Kern, C., et al. (2019) · The Lancet Digital Health · doi:10.1016/s2589-7500(19)30108-6 · Cited by 308
Laï, M.-C., Brian, M., & Mamzer, M.-F. (2020) · Journal of Translational Medicine · doi:10.1186/s12967-019-02204-y · Cited by 294 · cited 1×
“ML is often met cautiously by the end users [72], [26], [111].”
BackgroundOneto, L., & Chiappa, S. (2020) · Studies in Computational Intelligence · doi:10.1007/978-3-030-43883-8_7 · Cited by 291 · cited 1×
“Such questions about fairness and accountability of ML algorithms and data are studied by the branch of machine learning known as Fairness in ML [16].”
BackgroundUpdates in Human-AI Teams: Understanding and Addressing the Performance/Compatibility Tradeoff
Bansal, G., Nushi, B., Kamar, E., Weld, D. S., Lasecki, W. S., & Horvitz, E. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.33012429 · Cited by 284 · cited 1×
“[15] introduced the notion of compatibility of an AI update.”
BackgroundAlgorithms that remember: model inversion attacks and data protection law
Veale, M., Binns, R., & Edwards, L. (2018) · Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences · doi:10.1098/rsta.2018.0083 · Cited by 278 · cited 1×
“[123] emphasize the importance of protecting against model inversion attacks as a critical step towards compliance with data protection laws such as GDPR.”
BackgroundChallenges in the Deployment and Operation of Machine Learning in Practice
Baier, L., Jöhren, F., & Seebacher, S. (2019) · Repository KITopen (Karlsruhe Institute of Technology) · doi:10.5445/ir/1000095028 · Cited by 261
Assuring the Machine Learning Lifecycle: Desiderata, Methods, and Challenges
Ashmore, R., Calinescu, R., & Paterson, C. (2022) · ACM Computing Surveys · doi:10.1145/3453444 · Cited by 256 · cited 3×
“[10], however it is possible to conduct a similar review with any other ML pipeline description, such as CRISP-DM [126] or TDSP [123].”
BackgroundDetecting data errors: where are we and what needs to be done?
Abedjan, Z., Chu, X., Deng, D., Fernandez, R. C., Ilyas, I. F., Ouzzani, M., Papotti, P., Stonebraker, M., & Tang, N. (2016) · Proceedings of the VLDB Endowment · doi:10.14778/2994509.2994518 · Cited by 251 · cited 2×
“As mentioned in Section 3, data integrity plays a big part in quality control, and is an active branch of research [18].”
BackgroundIndustrial deployment of multi-agent technologies: review and selected case studies
Pěchouček, M., & Mařík, V. (2008) · Autonomous Agents and Multi-Agent Systems · doi:10.1007/s10458-008-9050-0 · Cited by 241
Challenges of Real-World Reinforcement Learning
Dulac-Arnold, G., Mankowitz, D. J., & Hester, T. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1904.12901 · Cited by 238
AIOps: Real-World Challenges and Research Innovations
Dang, Y., Lin, Q., & Huang, P. (2019) · 2019 IEEE/ACM 41st International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse-companion.2019.00023 · Cited by 238 · cited 1×
“[50] which uses the term AIOps for DevOps tasks for ML systems.”
MethodA Review on Data Cleansing Methods for Big Data
Ridzuan, F., & Wan Zainon, W. M. N. (2019) · Procedia Computer Science · doi:10.1016/j.procs.2019.11.177 · Cited by 223 · cited 1×
“[20].”
Data Lifecycle Challenges in Production Machine Learning: A Survey
Polyzotis, N., Roy, S., Whang, S. E., & Zinkevich, M. (2018) · ACM SIGMOD Record · doi:10.1145/3299887.3299891 · Cited by 214 · cited 1×
“[15].”
Wang, D., Wang, L., Zhang, Z., Wang, D., Zhu, H., Gao, Y., Fan, X., & Tian, F. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445432 · Cited by 209 · cited 1×
“One of the main reasons quoted was the UX design that did not take into account particularities of the environment (screen sizes, interaction with other software in the clinic) where it was installed, often resulting in an unusable software [147].”
BackgroundTechnology readiness levels for machine learning systems
Lavin, A., Gilligan-Lee, C. M., Visnjic, A., Ganju, S., Newman, D., Ganguly, S., Lange, D., Baydin, A. G., Sharma, A., Gibson, A., et al. (2022) · Nature Communications · doi:10.1038/s41467-022-33128-9 · Cited by 209
Zhao, P., Hoi, S. C. H., Wang, J., & Li, B. (2014) · Artificial Intelligence · doi:10.1016/j.artint.2014.06.003 · Cited by 199 · cited 1×
“We refer interested readers to the works on using domain adaptation [48], meta-learning [46] and transfer learning [152, 157] as ways of addressing dataset shift.”
MethodData scientists in software teams: State of the art and challenges
Kim, M., Zimmermann, T., DeLine, R., & Begel, A. (2018) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2017.2754374 · Cited by 199
"The human body is a black box": supporting clinical decision-making with deep learning
Sendak, M., Elish, M. C., Gao, M., Futoma, J., Ratliff, W., Nichols, M., Bedoya, A., Balu, S., & O'Brien, C. (2020) · Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency · doi:10.1145/3351095.3372827 · Cited by 196 · cited 3×
“One example is the project called Sepsis Watch [122].”
BackgroundInfluentialMachine learning based concept drift detection for predictive maintenance
Zenisek, J., Holzinger, F., & Affenzeller, M. (2019) · Computers & Industrial Engineering · doi:10.1016/j.cie.2019.106031 · Cited by 188 · cited 1×
“[80] show in their research on predictive maintenance for wear and tear of industrial machinery.”
BackgroundSnuba: automating weak supervision to label training data
Varma, P., & Ré, C. (2018) · Proceedings of the VLDB Endowment · doi:10.14778/3291264.3291268 · Cited by 184 · cited 1×
“Some of the most popular tools include Snorkel [93], Snuba [94] and cleanlab [95].”
MethodData Validation for Machine Learning
Breck, E. J., Zinkevich, M., Polyzotis, N., Whang, S. E., & Roy, S. (2019) · Cited by 182
The effect of the general data protection regulation on medical research
Rumbold, J. M. M., & Pierscionek, B. (2017) · Journal of Medical Internet Research · doi:10.2196/jmir.7108 · Cited by 179 · cited 2×
“Examples of such regulations include the General Data Protection Regulation in the European Union [116] and ethical screening laws in a range of Asian countries [5].”
BackgroundRegulatory responses to medical machine learning
Minssen, T., Gerke, S., Aboy, M., Price, N., & Cohen, G. (2020) · Journal of Law and the Biosciences · doi:10.1093/jlb/lsaa002 · Cited by 173 · cited 1×
“analyze the current regulatory approaches to medical ML in the US and Europe, and discuss how existing laws evaluate ML applications [103].”
BackgroundWeakly-Supervised Learning of Visual Relations
Peyre, J., Laptev, I., Schmid, C., & Sivic, J. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.554 · Cited by 172 · cited 1×
“A possible option here is to use noisy label oracles [39] or weak annotations [105], however these approaches provide imprecise labels, which can lead to a loss in quality of the model [112].”
BackgroundScaling big data mining infrastructure: the twitter experience
Lin, J., & Ryaboy, D. (2013) · ACM SIGKDD Explorations Newsletter · doi:10.1145/2481244.2481247 · Cited by 165 · cited 1×
“As explained by Lin and Ryaboy [16], at Twitter this situation often happened due to the fact that the same entity (e.”
BackgroundProceedings of the Conference on Fairness, Accountability, and Transparency
(2019) · doi:10.1145/3287560 · Cited by 159
Developing a prediction model for customer churn from electronic banking services using data mining
Keramati, A., Ghaneei, H., & Mirmohammadi, S. M. (2016) · Financial Innovation · doi:10.1186/s40854-016-0029-6 · Cited by 150 · cited 1×
“[44] where the primary goal of the ML application is predicting churn in an interpretable way through if-then rules.”
BackgroundThe Growing Gap Between Emerging Technologies and the Law
Marchant, G. E. (2011) · The International library of ethics, law and technology · doi:10.1007/978-94-007-1356-7_2 · Cited by 138
Applying Deep Learning to Airbnb Search
Haldar, M., Abdool, M., Ramanathan, P., Xu, T., Yang, S., Duan, H., Zhang, Q., Barrow-Williams, N., Turnbull, B. C., Collins, B. M., et al. (2019) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3292500.3330658 · Cited by 124 · cited 1×
“e. This approach accelerates the time to get a deployed solution, allows the collection of important feedback and also helps avoid overcomplicated designs. This was the case reported by Haldar et al. [30]. In the process of applying machine learning to AirBnB search, the team started with a complex deep learning model…”
MethodSnorkel DryBell: A Case Study in Deploying Weak Supervision at Industrial Scale
Bach, S. H., Rodriguez, D., Liu, Y., Luo, C., Shao, H., Xia, C., Sen, S., Ratner, A., Hancock, B., Alborzi, H., et al. (2019) · Proceedings of the 2019 International Conference on Management of Data · doi:10.1145/3299869.3314036 · Cited by 118 · cited 1×
“Some of the most popular tools include Snorkel [93], Snuba [94] and cleanlab [95].”
MethodThe Cost of Training NLP Models: A Concise Overview
Sharir, O., Peleg, B., & Shoham, Y. (2020) · arXiv · doi:10.48550/arxiv.2004.08900 · Cited by 113
Provenance Analysis for Missing Answers and Integrity Repairs.
Zaharia, M., Chen, A., Davidson, A., Ghodsi, A., Hong, S. A., Konwinski, A., Murching, S., Nykodym, T., Ogilvie, P., Parkhe, M., et al. (2018) · IEEE Data(base) Engineering Bulletin · Cited by 112
Monitoring and explainability of models in productionPreprint
Klaise, J., Looveren, A. V., Cox, C., Vacanti, G., & Coca, A. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2007.06299 · Cited by 105 · cited 1×
“[72] point out the importance of outlier detection as a key instrument to flag model predictions that cannot be used in a production setting.”
Background150 Successful Machine Learning Models: 6 Lessons Learned at Booking.com
Bernardi, L., Mavridis, T., & Estevez, P. (2019) · Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining · doi:10.1145/3292500.3330744 · Cited by 95 · cited 1×
“com discovered after deploying 150 models into production [21].”
BackgroundContinual Learning in PracticePreprint
Diethe, T., Borchert, T., Thereska, E., Balle, B., & Lawrence, N. D. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1903.05202 · Cited by 91
Self-citationWhen Machine Learning Meets Wireless Cellular Networks: Deployment, Challenges, and Applications
Challita, U., Rydén, H., & Tullberg, H. (2020) · IEEE Communications Magazine · doi:10.1109/mcom.001.1900664 · Cited by 87 · cited 1×
“Most advanced techniques, such as deep learning, are not considered yet for practical deployment, despite being able to handle high dimensional mobile network data [41].”
BackgroundAssuring the Machine Learning Lifecycle: Desiderata, Methods, and ChallengesPreprint
Ashmore, R., Călinescu, R., & Paterson, C. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1905.04223 · Cited by 86 · cited 3×
“[10], however it is possible to conduct a similar review with any other ML pipeline description, such as CRISP-DM [126] or TDSP [123].”
BackgroundFailure Modes in Machine Learning SystemsPreprint
Kumar, R. S. S., O'Brien, D., Albert, K., Viljöen, S., & Snover, J. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1911.11034 · Cited by 86 · cited 1×
“[78].”
An empirical investigation of the challenges of real-world reinforcement learning
Dulac-Arnold, G., Levine, N., Mankowitz, D. J., Li, J., Paduraru, C., Gowal, S., & Hester, T. (2020) · arXiv · doi:10.48550/arxiv.2003.11881 · Cited by 78
Denoising Multi-Source Weak Supervision for Neural Text Classification
Ren, W., Li, Y., Su, H., Kartchner, D., Mitchell, C. S., & Zhang, C. (2020) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2020.findings-emnlp.334 · Cited by 77
Adaptation Strategies for Automated Machine Learning on Evolving Data
Celik, B., & Vanschoren, J. (2020) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2021.3062900 · Cited by 75 · cited 2×
“Undetected, this phenomenon can have major adverse effects on model performance, as is shown by Jameel et al. [76] for classification problems or by Celik and Vanschoren [77] in the AutoML context.”
BackgroundMuthiah, S., Butler, P. J., Khandpur, R. P., Saraf, P., Self, N., Rozovskaya, A., Zhao, L., Cadena, J., Lu, C., Vullikanti, A. K. S., et al. (2016) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/2939672.2939709 · Cited by 70 · cited 2×
“[73].”
MethodAljunid, S. M., Srithamrongsawat, S., Chen, W. L., Bae, S., Pwu, R., Ikeda, S., & Xu, L. (2012) · Value in Health · doi:10.1016/j.jval.2011.11.004 · Cited by 69
An Empirical Analysis of Backward Compatibility in Machine Learning Systems
Srivastava, M., Nushi, B., Kamar, E. S., Shah, S., & Horvitz, E. (2020) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3394486.3403379 · Cited by 65 · cited 1×
“[132], who provide a detailed empirical study of backward compatibility issues in ML systems.”
BackgroundHan, C., Rundo, L., Murao, K., Nemoto, T., & Nakayama, H. (2020) · IFIP advances in information and communication technology · doi:10.1007/978-3-030-49186-4_27 · Cited by 64 · cited 3×
“[54], many countries have strict laws in place to protect the data of patients, which makes the adoption of ML in healthcare particularly di cult.”
BackgroundInfluentialFirebird: Predicting Fire Risk and Prioritizing Fire Inspections in Atlanta
Madaio, M., Chen, S.-T., Haimson, O. L., Zhang, W., Cheng, X. L., Hinds-Aldrich, M., Chau, D. H., & Dilkina, B. (2016) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/2939672.2939682 · Cited by 63 · cited 2×
“[22].”
MethodOn the Co-evolution of ML Pipelines and Source Code - Empirical Study of DVC Projects
Barrak, A., Eghan, E. E., & Adams, B. (2021) · IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER) · doi:10.1109/saner50967.2021.00046 · Cited by 61 · cited 1×
“Regular SE tools for such common operations as source control, branching and merging, review, cannot be applied to these new artifacts "as is" [17].”
BackgroundQuantifying the accuracy of the ground truth associated with Internet traffic traces
Dusi, M., Gringoli, F., & Salgarelli, L. (2010) · Computer Networks · doi:10.1016/j.comnet.2010.11.006 · Cited by 56
Hardware-aware machine learning: modeling and optimization
Marculescu, D., Stamoulis, D. S., & Cai, E. (2018) · doi:10.1145/3240765.3243479 · Cited by 55 · cited 1×
“[43] in the context of hardware-aware ML.”
Adversarial Machine Learning -- Industry PerspectivesPreprint
Kumar, R. S. S., Nyström, M., Lambert, J., Marshall, A. D., Goertzel, M., Comissoneru, A., Swann, M., & Xia, S. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2002.05646 · Cited by 49 · cited 1×
“found that industry practitioners are not equipped to protect, detect and respond to attacks on their ML systems [117].”
BackgroundDVC: Data Version Control - Git for Data & Models
Kuprieiev, R., Petrov, D., Outis, M., Redzyński, P., da Costa-Luis, C. O., Schepanovski, A., Щеклеин, И., Orpinel, J., Щеклеин, И., Santos, F., et al. (2020) · Zenodo (CERN European Organization for Nuclear Research) · doi:10.5281/zenodo.3677553 · Cited by 49
Machine Learning Algorithms in Heavy Process Manufacturing
Hansson, K., Yella, S., Dougherty, M. S., & Fleyeh, H. A. (2016) · DiVA (Digitala Vetenskapliga Arkivet) · doi:10.5923/j.ajis.20160601.01 · Cited by 48 · cited 1×
“[43] describe several cases in manufacturing that adopt DT because of their interpretability.”
BackgroundSoemers, D. J. N. J., Brys, T., Driessens, K., Winands, M. H. M., & Nowé, A. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.11411 · Cited by 48 · cited 1×
“Consequently, detection of concept drift becomes a growing concern for teams that maintain ML models in production [131, 136], which makes this challenge directly related to monitoring discussed in the previous section.”
BackgroundActive Learning with Human-Like Noisy Oracle
Du, J., & Ling, C. X. (2010) · Proceedings · doi:10.1109/icdm.2010.114 · Cited by 41 · cited 1×
“A possible option here is to use noisy label oracles [39] or weak annotations [105], however these approaches provide imprecise labels, which can lead to a loss in quality of the model [112].”
BackgroundLearning a Unified Embedding for Visual Search at Pinterest
Zhai, A., Wu, H., Tzeng, E., Park, D. H., & Rosenberg, C. (2019) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3292500.3330739 · Cited by 41 · cited 1×
“An illustrative case is an approach Pinterest took towards learning image embeddings [156].”
MethodData Engineering for Data Analytics: A Classification of the Issues, and Case StudiesPreprint
Nazábal, A., Williams, C. K. I., Colavizza, G., Smith, C. R., & Williams, A. A. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2004.12929 · Cited by 38 · cited 1×
“Joining this information into a single dataset suitable for machine learning can be a complicated task in its own right, known as the data integration process [94].”
BackgroundGear-Induced Concept Drift in Marine Images and Its Effect on Deep Learning Classification
Langenkämper, D., van Kevelaer, R., Purser, A., & Nattkemper, T. W. (2020) · Frontiers in Marine Science · doi:10.3389/fmars.2020.00506 · Cited by 37 · cited 1×
“[73] which studies the efects of slight changes in marine images on deep learning models’ performance.”
BackgroundAzureML: Anatomy of a machine learning service
AzureML Team (2015) · Cited by 29
Technology Readiness Levels for Machine Learning SystemsPreprint
Lavin, A., Lee, C., Visnjic, A., et al. (2021) · Research Square · doi:10.21203/rs.3.rs-133138/v1 · Cited by 27
A Critical Review on Adverse Effects of Concept Drift over Machine Learning Classification Models
Jameel, S. M., Ahmed, M., Alhussain, H., Rehman, M., & Budiman, A. (2020) · International Journal of Advanced Computer Science and Applications · doi:10.14569/ijacsa.2020.0110127 · Cited by 24 · cited 1×
“[61] for classiication problems or by Celik and Vanschoren [32] in the AutoML context.”
BackgroundUsing Small Business Banking Data for Explainable Credit Risk Scoring
Wang, W., Lesner, C., Ran, A., Rukonic, M., Xue, J., & Shiu, E. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i08.7055 · Cited by 23 · cited 1×
“[148] describe such a requirement in the context of credit risk scoring.”
BackgroundDeploying Machine Learning Models for Public Policy: A Framework
Ackermann, K., Walsh, J., De Unánue, A., Naveed, H., Navarrete Rivera, A., Lee, S.-J., Bennett, J., Defoe, M., Cody, C., Haynes, L., et al. (2018) · Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining · doi:10.1145/3219819.3219911 · Cited by 22 · cited 2×
“abeled outlier data is scarce, and the detector training often becomes a semi-supervised or even an unsupervisedproblem. Additional insight on monitoringof ML systems can be found in Ackermann et al. [54]. This paper describes an early intervention system (EIS) for two police departments in the US. On the surface thei…”
BackgroundMethodEnabling Onboard Detection of Events of Scientific Interest for the Europa Clipper Spacecraft
Wagstaff, K. L., Doran, G. B., Davies, A. G., Anwar, S., Chakraborty, S., Cameron, M. E., Daubar, I. J., & Phillips, C. B. (2019) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3292500.3330656 · Cited by 22 · cited 1×
“[146] controlling for consumption of spacecraft resources is crucial.”
BackgroundMutembesa, D., Omongo, C. A., & Mwebaze, E. (2018) · Proceedings of the AAAI Conference on Human Computation and Crowdsourcing · doi:10.1609/hcomp.v6i1.13322 · Cited by 21
Business Analysis Method for Constructing Business–AI Alignment Model
Takeuchi, H., & Yamamoto, S. (2020) · Procedia Computer Science · doi:10.1016/j.procs.2020.09.140 · Cited by 20 · cited 2×
“We refer interested readers to the work by Takeuchi and Yamamoto [17].”
ResultAnalyzing concept drift: A case study in the financial sector
Masegosa, A. R., Martínez, A. M., Ramos-López, D., Helge Langseth, A., Nielsen, T. D., & Salmerón, A. (2020) · Intelligent Data Analysis · doi:10.3233/ida-194515 · Cited by 19
The Single Responsibility Principle
Noback, M. (2018) · Apress eBooks · doi:10.1007/978-1-4842-4119-6_1 · Cited by 17
Continuous Training for Production ML in the TensorFlow Extended (TFX) Platform
Baylor, D., Haas, K., Katsiapis, K., Leong, S., Liu, R., Mewald, C., Miao, H., Polyzotis, N., Trott, M. D., & Zinkevich, M. (2019) · Cited by 17
Data Readiness LevelsPreprint
Lawrence, N. D. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1705.02245 · Cited by 14 · cited 1×
“Lawrence [150] proposes an approach toward classifying the readiness of data for ML tasks, which Royal Society DELVE applied in one of their reports in the context of COVID-19 pandemic [20].”
MethodSelf-citationHow can Automated Machine Learning Help Business Data Science Teams?
Ebadi, A., Gauthier, Y., Tremblay, S., & Paul, P. (2019) · International Conference on Machine Learning and Applications · doi:10.1109/icmla.2019.00196 · Cited by 14 · cited 1×
“However practical reports of applying AutoML to real world problems indicate that practitioners need to exercise extreme caution, as AutoML methods might not be ready for decision making in high-stakes areas yet [43, 44].”
BackgroundImplementation and On-Orbit Testing Results of a Space Communications Cognitive Engine
Hackett, T. M., Bilén, S. G., Ferreira, P. V. R., Wyglinski, A. M., Reinhart, R. C., & Mortensen, D. J. (2018) · IEEE Transactions on Cognitive Communications and Networking · doi:10.1109/tccn.2018.2878202 · Cited by 13 · cited 1×
“presented an instructive use case of how limited simulation-based testing can be [67].”
BackgroundA Selective Transfer Learning Method for Concept Drift Adaptation
Xie, G., Sun, Y., Lin, M., & Tang, K. (2017) · Lecture notes in computer science · doi:10.1007/978-3-319-59081-3_42 · Cited by 10 · cited 1×
“We refer interested readers to the works on using domain adaptation [48], meta-learning [46] and transfer learning [152, 157] as ways of addressing dataset shift.”
MethodReview of Concept Drift Detection Method for Industrial Process Modeling
Sun, Z., Tang, J., Qiao, J., & Cui, C. (2020) · Chinese Control Conference (CCC) · doi:10.23919/ccc50068.2020.9189106 · Cited by 10 · cited 1×
“Consequently, detection of concept drift becomes a growing concern for teams that maintain ML models in production [131, 136], which makes this challenge directly related to monitoring discussed in the previous section.”
BackgroundLi, Y., Wang, Z., Ding, B., & Zhang, C. (2021) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3447548.3470827 · Cited by 10 · cited 5×
“However practical reports of applying AutoML to real world problems indicate that practitioners need to exercise extreme caution, as AutoML methods might not be ready for decision making in high-stakes areas yet [140, 141].”
InfluentialTowards better data discovery and collection with flow-based programmingPreprint
Paleyes, A., Cabrera, C., & Lawrence, N. D. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2108.04105 · Cited by 5
Self-citationAutomatic Discovery of Privacy–Utility Pareto Fronts
Avent, B., González, J., Diethe, T., Paleyes, A., & Balle, B. (2020) · Proceedings on Privacy Enhancing Technologies · doi:10.2478/popets-2020-0060 · Cited by 3
Self-citationUsing Amazon SageMaker to Operationalize Machine Learning
Venkateswar, K. (2019) · Cited by 1
DVC: Data Version Control - Git for Data & Models
Kuprieiev, R., Pachhai, S., Petrov, D., Redzyński, P., da Costa-Luis, C. O., (변기호), P. R., Schepanovski, A., Щеклеин, И., Taskaya, B., Orpinel, J., et al. (2022) · Zenodo (CERN European Organization for Nuclear Research) · doi:10.5281/zenodo.6195393 · Cited by 1
Research works printed but not linked: 26
- Arif Cam, Michael Chui, and Bryce Hall. 2019. Global AI survey: AI proves its worth, but few scale impact. McKinsey Analytics (2019). https://www.mckinsey.com/featured-insights/artificial-intelligence/global-ai-survey-ai-proves-its-worth-but-few-scale-impact. Accessed from 13/06/2022.
- Royal Society (Great Britain). 2017. Machine Learning: The Power and Promise of Computers that Learn by Example: an Introduction. Royal Society.
- Ben Lorica and Nathan Paco. 2018. The State of Machine Learning Adoption in the Enterprise. O’Reilly Media.
- Colin Shearer. 2000. The CRISP-DM model: The new blueprint for data mining. J. Data Warehous. 5, 4 (2000), 13–22.
- Rajashree Y. Patil and R. V. Kulkarni. 2012. A review of data cleaning algorithms for data warehouse systems. Int. J. Comput. Sci. Inf. Technol. 3, 5 (2012), 5212–5214.
- 10.1109/TITS.2019.2962338 · DOI · Paper page
- Diego Charrez. 2019. NeurIPS 2019 Stats. Retrieved from https://medium.com/@dcharrezt/neurips-2019-stats-c91346d31c8f.
- 10.1023/A:1022643204877 · doi:10.1023/a:1022643204877 · Paper page
- 10.5555/3122009.3242042 · doi:10.5555/3122009.3242042
- Bobak Shahriari, Alexandre Bouchard-Cote, and Nando Freitas. 2016. Unbounded Bayesian optimization via regularization. In Proceedings of the 19th International Conference on Artificial Intelligence and Statistics (Proceedings of Machine Learning Research), Arthur Gretton and Christian C. Robert (Eds.), Vol. 51. PMLR, 1168–1176. Retrieved from http://proceedings.mlr.press/v51/shahriari16.html.
- Ram Ananth, Seph Mard, and Peter Simon. 2019. Opening the “Black Box”: The path to deployment of AI models in banking, white paper. DataRobot (2019). https://www.datarobot.com/resources/opening-the-black-box/. Accessed from 13/06/2022.
- Robert Soden Dennis Wagenaar Dave Luo and Annegien Tijssen. 2019. Taking Ethics Fairness and Bias Seriously in Machine Learning for Disaster Risk Management. arxiv:cs.CY/1912.05538.
- Nantheera Anantrasirichai and David Bull. 2021. Artificial intelligence in the creative industries: A review. Artif. Intell. Rev. (2021), 1–68.
- D. Malan et al. 2016. Values and the fourth industrial revolution: Connecting the dots between value, values, profit and purpose. In Proceedings of the World Economic Forum.
- Daniel Malan. 2018. The law can’t keep up with new tech. Here’s how to close the gap. In Proceedings of the World Economic Forum.
- Ramprakash Ramamoorthy P. Satya Madhuri and Malini Christina Raj. 2019. AI from labs to production—challenges and learnings. USENIX Association. https://www.usenix.org/conference/opml19/presentation/madhuri. Accessed from 13/06/2022.
- Oscar Schwartz. 2019. In 2016 Microsoft’s racist chatbot revealed the dangers of online conversation. IEEE Spectr. 11 (2019).
- Florian Tramèr, Fan Zhang, Ari Juels, Michael K. Reiter, and Thomas Ristenpart. 2016. Stealing machine learning models via prediction APIs. In Proceedings of the 25th USENIX Security Symposium (USENIX Security’16). 601–618.
- Jeremy Hermann and Mike Del Balso. 2017. Meet Michelangelo: Uber’s machine learning platform. Retrieved from https://eng.uber.com/michelangelo.
- Matei Zaharia, Andrew Chen, Aaron Davidson, Ali Ghodsi, Sue Ann Hong, Andy Konwinski, Siddharth Murching, Tomas Nykodym, Paul Ogilvie, Mani Parkhe, et al. 2018. Accelerating the machine learning lifecycle with MLflow.IEEE Data Eng. Bull. 41, 4 (2018), 39–45.
- Nick Hynes, D. Sculley, and Michael Terry. 2017. The Data Linter: Lightweight, automated sanity checking for ML data sets. In Proceedings of the NIPS MLSys Workshop.
- Matthias Feurer, Aaron Klein, Katharina Eggensperger, Jost Springenberg, Manuel Blum, and Frank Hutter. 2015. Efficient and robust automated machine learning. In Advances in Neural Information Processing Systems, C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Garnett (Eds.), Vol. 28. Curran Associates, Inc. Retrieved from https://proceedings.neurips.cc/paper/2015/file/11d0e6287202fced83f79975ec59a3a6-Paper.pdf.
- Randal S. Olson and Jason H. Moore. 2016. TPOT: A tree-based pipeline optimization tool for automating machine learning. In Proceedings of the Workshop on Automatic Machine Learning. PMLR, 66–74.
- Stephan Rabanser, Stephan Günnemann, and Zachary Chase Lipton. 2019. Failing loudly: An empirical study of methods for detecting dataset shift. In Proceedings of the Neural Information Processing Systems Foundation Conference.
- Chelsea Finn, Pieter Abbeel, and Sergey Levine. 2017. Model-agnostic meta-learning for fast adaptation of deep networks. In Proceedings of the International Conference on Machine Learning. PMLR, 1126–1135.
- A. Zujus. 2018. AI project development–how project managers should prepare. Tratto il giorno Agosto 31 (2018), 2019.
Cited by
On Predictive Maintenance in Industry 4.0: Overview, Models, and Challenges
Achouch, M., Dimitrova, M., Ziane, K., Sattarpanah Karganroudi, S., Dhouib, R., Ibrahim, H., & Adda, M. (2022) · Applied Sciences · doi:10.3390/app12168081 · Cited by 528
“Compared to traditional enterprise software updates, this cycle of improving predictive maintenance models is much more complex [54].”
MethodModel aggregation techniques in federated learning: A comprehensive survey
Qi, P., Chiaro, D., Guzzo, A., Ianni, M., Fortino, G., & Piccialli, F. (2024) · Future Generation Computer Systems · doi:10.1016/j.future.2023.09.008 · Cited by 452
Technology readiness levels for machine learning systems
Lavin, A., Gilligan-Lee, C. M., Visnjic, A., Ganju, S., Newman, D., Ganguly, S., Lange, D., Baydin, A. G., Sharma, A., Gibson, A., et al. (2022) · Nature Communications · doi:10.1038/s41467-022-33128-9 · Cited by 209
Exploiting BERT for Multimodal Target Sentiment Classification through Input Space Translation
Khan, Z., & Fu, Y. (2021) · Proceedings of the 29th ACM International Conference on Multimedia · doi:10.1145/3474085.3475692 · Cited by 193
“However, the remarkable ability of transformers to adapt without architectural modifications [18], as well as the practical desirability of deploying [26] popular baseline architectures without modifications, has motivated research into exploiting large language models for target/aspect sentiment classification withou…”
Cross-Validation Visualized: A Narrative Guide to Advanced Methods
Allgaier, J., & Pryss, R. (2024) · Machine Learning and Knowledge Extraction · doi:10.3390/make6020065 · Cited by 176
“It is critical to acknowledge, as [20] emphasizes, that reliance on simulation-based testing is constrained by the underlying assumptions of the simulations, and thus cannot fully substitute for empirical testing in real-world scenarios.”
Background
Metadata
- Authors
- Paleyes, A., Urma, R.-G., & Lawrence, N. D.
- Affiliations
- University of Cambridge (GB)
- Cambridge Spark, Cambridge, United Kingdom
- Venue
- ACM Computing Surveys
- Published
- 2022-12-07
- Volume
- 55
- Issue
- 6
- Pages
- 1-29
- ISSN
- 0360-0300 (ISSN-L)
- 1557-7341
- OpenAlex
- W3104128335
- Semantic Scholar
- b1f574c47d0b6e3032246770b9cbebb9c7bd0c7f
- Cited by
- 638
- References
- 128
- Citations by year
- 2026: 149 · 2025: 156 · 2024: 166 · 2023: 93 · 2022: 46
- FWCI
- 114.17
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- ACM Computing Surveys (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- White Rose Research Online (University of Leeds, The University of Sheffield, University of York) (submittedVersion)
- Apollo (University of Cambridge) (submittedVersion)
- arXiv (Cornell University)
- Apollo
- Funders
- Alan Turing Institute
- UK Research & Innovation
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
- Physical Sciences › Computer Science › Artificial Intelligence › Artificial Intelligence Applications
- Physical Sciences › Computer Science › Information Systems › Cloud Computing and Resource Management
- Keywords
- machine learning deployment, production systems, deployment challenges, industry case studies
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.