MLOps: A Taxonomy and a Methodology
Open accessThe DOI matches its Crossref record; 25 of the 25 references it cites were found in a registry or an index.
Abstract
Over the past few decades, the substantial growth in enterprise-data availability and the advancements in Artificial Intelligence (AI) have allowed companies to solve real-world problems using Machine Learning (ML). ML Operations (MLOps) represents an effective strategy for bringing ML models from academic resources to useful tools for solving problems in the corporate world. The current literature on MLOps is still mostly disconnected and sporadic. In this work, we review the existing scientific literature and we propose a taxonomy for clustering research papers on MLOps. In addition, we present methodologies and operations aimed at defining a ML pipeline to simplify the release of ML applications in industry. The pipeline is based on ten steps: business problem understanding, data acquisition, ML methodology, ML training & testing, continuous integration, continuous delivery, continuous training, continuous monitoring, explainability, and sustainability. The scientific and business interest and the impact of MLOps have grown significantly over the past years. The definition of a clear and standardized methodology for conducting MLOps projects is the main contribution of this paper.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
LeCun, Y., Bengio, Y., & Hinton, G. (2015) · Nature · doi:10.1038/nature14539 · Cited by 84,126
Larian Studios Dev Team (2024) · doi:10.1145/3292500.3330701 · Cited by 8,618
Peeking Inside the Black-Box: A Survey on Explainable Artificial Intelligence (XAI)
Adadi, A., & Berrada, M. (2018) · IEEE Access · doi:10.1109/access.2018.2870052 · Cited by 6,294
Explanation in Artificial Intelligence: Insights from the Social Sciences
Miller, T. (2017) · arXiv · doi:10.48550/arxiv.1706.07269 · Cited by 5,338
Explanation in artificial intelligence: Insights from the social sciences
Miller, T. (2019) · Artificial Intelligence · doi:10.1016/j.artint.2018.07.007 · Cited by 5,302
Energy and Policy Considerations for Deep Learning in NLP
Strubell, E., Ganesh, A., & McCallum, A. (2019) · arXiv · doi:10.48550/arxiv.1906.02243 · Cited by 3,735
Energy and Policy Considerations for Deep Learning in NLP
Strubell, E., Ganesh, A., & McCallum, A. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1355 · Cited by 3,684 · cited 1×
“Strubell et al. selectively focused in carbon footprint analysis on AI models for natural language processing [94].”
BackgroundExplainable AI: a review of machine learning interpretability methods
Linardatos, P., Papastefanopoulos, V., & Kotsiantis, S. (2020) · Entropy · doi:10.3390/e23010018 · Cited by 2,922 · cited 1×
“An explainable model does not necessarily translate into one that humans are able to understand (internal logic or underlying processes) [90].”
BackgroundEmpirical studies of agile software development: A systematic review
Dybå, T., & Dingsøyr, T. (2008) · Information and Software Technology · doi:10.1016/j.infsof.2008.01.006 · Cited by 2,668 · cited 1×
“This implies that ML models need to be retrained or, in the worst-case scenario, the entire ML pipeline has to be rebuilt to tackle feature drift [8], [9].”
BackgroundBenchmarking Neural Network Robustness to Common Corruptions and Perturbations
Hendrycks, D., & Dietterich, T. (2019) · arXiv · doi:10.48550/arxiv.1903.12261 · Cited by 1,335
Samek, W., Wiegand, T., & Müller, K.-R. (2017) · arXiv · doi:10.48550/arxiv.1708.08296 · Cited by 1,080
A unifying view on dataset shift in classification
Moreno-Torres, J. G., Raeder, T., Alaiz-Rodríguez, R., Chawla, N. V., & Herrera, F. (2012) · Pattern Recognition · doi:10.1016/j.patcog.2011.06.019 · Cited by 1,029
Preparing Medical Imaging Data for Machine Learning
Willemink, M. J., Koszek, W. A., Hardell, C., Wu, J., Fleischmann, D., Harvey, H., Folio, L. R., Summers, R. M., Rubin, D. L., & Lungren, M. P. (2020) · Radiology · doi:10.1148/radiol.2020192224 · Cited by 1,020 · cited 1×
“Data labeling tools (Table 1) are used to help the data science team to label large datasets such as texts, images, etc. [56], [57].”
MethodDalzochio, J., Kunst, R., Pignaton, E., Binotto, A., Sanyal, S., Favilla, J., & Barbosa, J. (2020) · Computers in Industry · doi:10.1016/j.compind.2020.103298 · Cited by 621
Artificial Intelligence and COVID-19: Deep Learning Approaches for Diagnosis and Treatment
Jamshidi, M., Lalbakhsh, A., Talla, J., Peroutka, Z., Hadjilooei, F., Lalbakhsh, P., Jamshidi, M., Spada, L. L., Mirmozafari, M., Dehghani, M., et al. (2020) · IEEE Access · doi:10.1109/access.2020.3001973 · Cited by 551
Syafrudin, M., Alfian, G., Fitriyani, N. L., & Rhee, J. (2018) · Sensors · doi:10.3390/s18092946 · Cited by 401 · cited 1×
“Some tools with advantages and disadvantages for each software are shown in Table 11 cessful ML project because ML model performance tends to decay after model deployment due to changes in the input data stream over time [64], [65].”
MethodThe ML test score: A rubric for ML production readiness and technical debt reduction
Breck, E., Cai, S., Nielsen, E., Salib, M., & Sculley, D. (2017) · 2017 IEEE International Conference on Big Data (Big Data) · doi:10.1109/bigdata.2017.8258038 · Cited by 310
Continuous software engineering and beyond: trends and challenges
Fitzgerald, B., & Stol, K.-J. (2014) · Proceedings of the 1st International Workshop on Rapid Continuous Software Engineering · doi:10.1145/2593812.2593813 · Cited by 212
Applying DevOps Practices of Continuous Automation for Machine Learning
Karamitsos, I., Thabit, S., & Apostolopoulos, C. (2020) · Information · doi:10.3390/info11070363 · Cited by 140 · cited 3×
“The goal of DevOps is to study ways to improve service quality and features in order to meet customer needs [23], [24].”
BackgroundDemystifying MLOps and Presenting a Recipe for the Selection of Open-Source Tools
Ruf, P., Madan, M., Reich, C., & Ould-Abdeslam, D. (2021) · Applied Sciences · doi:10.3390/app11198861 · Cited by 112 · cited 2×
“The model may become less accurate because the data used in training the model are no longer representative of the new data existing in production [71].”
BackgroundMethodGranlund, T., Stirbu, V., & Mikkonen, T. (2021) · SN Computer Science · doi:10.1007/s42979-021-00726-1 · Cited by 53 · cited 3×
“The use case concerns Oravizio [38], a software product that provides data-driven information on patient-level risks related to hip and knee joint replacement surgery.”
MLOps Challenges in Multi-Organization Setup: Experiences from Two Real-World Cases
Granlund, T., Kopponen, A., Stirbu, V., Myllyaho, L., & Mikkonen, T. (2021) · 2021 IEEE/ACM 1st Workshop on AI Engineering - Software Engineering for AI (WAIN) · doi:10.1109/wain52551.2021.00019 · Cited by 48
Machine Learning Explainability for External Stakeholders
Bhatt, U., Andrus, M., Weller, A., & Xiang, A. (2020) · arXiv · doi:10.48550/arxiv.2007.05408 · Cited by 41
Machine Learning Operations: A Survey on MLOps Tool Support
Hewage, N., & Meedeniya, D. (2022) · arXiv · doi:10.48550/arxiv.2202.10169 · Cited by 38 · cited 1×
“In this section, we want to summarize these challenges and describe some of the most popular ML frameworks like MLflow, Kubeflow, MLPerf, etc. [54].”
MethodFrom a Data Science Driven Process to a Continuous Delivery Process for Machine Learning Systems
Lwakatare, L. E., Crnkovic, I., Rånge, E., & Bosch, J. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-64148-1_12 · Cited by 28 · cited 1×
“The outputs of this stage are components (packages and artifacts) to be deployed in the pre-production/production environment of continuous delivery [75].”
Background
Cited by
Steidl, M., Felderer, M., & Ramler, R. (2023) · Journal of Systems and Software · doi:10.1016/j.jss.2023.111615 · Cited by 126
Towards Trustworthy Machine Learning in Production: An Overview of the Robustness in MLOps Approach
Bayram, F., & Ahmed, B. S. (2025) · ACM Computing Surveys · doi:10.1145/3708497 · Cited by 48
“The platform integrates an incremental validation methodology of data quality for evolving datasets.”
Algorithmovigilance, lessons from pharmacovigilance
Balendran, A., Benchoufi, M., Evgeniou, T., & Ravaud, P. (2024) · npj Digital Medicine · doi:10.1038/s41746-024-01237-y · Cited by 34
“These tests are often automated (e.g., tests are computed every month or accuracy is aggregated over a couple of weeks long window), aligning with the emergence of MLOps aimed at operationalizing ML monitoring 3 .”
BackgroundMethodNavigating the DevOps landscape
Zhang, X., Zhao, P., & Jaskolka, J. (2025) · Journal of Systems and Software · doi:10.1016/j.jss.2024.112331 · Cited by 21
The Past, Present, and Future of Research on the Continuous Development of AI
Steidl, M., Ramler, R., & Felderer, M. (2024) · 2024 50th Euromicro Conference on Software Engineering and Advanced Applications (SEAA) · doi:10.1109/seaa64295.2024.00054 · Cited by 3
Metadata
- Authors
- Testi, M., Ballabio, M., Frontoni, E., Iannello, G., Moccia, S., Soda, P., & Vessio, G.
- Venue
- IEEE Access
- Published
- 2022
- Volume
- 10
- Pages
- 63606-63618
- ISSN
- 2169-3536 (ISSN-L)
- OpenAlex
- W4285158860
- Cited by
- 114
- References
- 25
- FWCI
- 22.38
- Type
- journal article
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.