Demystifying MLOps and Presenting a Recipe for the Selection of Open-Source Tools
Open accessThe DOI matches its Crossref record; 61 of the 61 references it cites were found in a registry or an index.
Abstract
Nowadays, machine learning projects have become more and more relevant to various real-world use cases. The success of complex Neural Network models depends upon many factors, as the requirement for structured and machine learning-centric project development management arises. Due to the multitude of tools available for different operational phases, responsibilities and requirements become more and more unclear. In this work, Machine Learning Operations (MLOps) technologies and tools for every part of the overall project pipeline, as well as involved roles, are examined and clearly defined. With the focus on the inter-connectivity of specific tools and comparison by well-selected requirements of MLOps, model performance, input data, and system quality metrics are briefly discussed. By identifying aspects of machine learning, which can be reused from project to project, open-source tools which help in specific parts of the pipeline, and possible combinations, an overview of support in MLOps is given. Deep learning has revolutionized the field of Image processing, and building an automated machine learning workflow for object detection is of great interest for many organizations. For this, a simple MLOps workflow for object detection with images is portrayed.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures



Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Packaging models in containers is a commonly reported method of integration, in which the model is accessible through representation state transfer (REST) or remote procedure call (RPC) interfaces [12,16,28,45].”
From Operationalizing machine learning models: a systematic literature review (2022) · cites this paper as [45]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Models packaged in containers are simply run directly as standalone services [16,19,28,29,45].”
From Operationalizing machine learning models: a systematic literature review (2022) · cites this paper as [45]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Polyaxon 9 , MLflow 10 , TFX 11 , ZenML 12 , Flyte 13 , Seldon Core 14 and BentoML 15 are some of the more fully-integrated platforms that exist [10,45], along with the Amazon-exclusive Sage-Maker 16 [9,45].”
From Operationalizing machine learning models: a systematic literature review (2022) · cites this paper as [45]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“In productive settings, it is challenging to operate ML manually due to different stacks of software and hardware components and their interplay as well as the selection of both ( [64], [65].”
From Machine Learning Operations (MLOps): Overview, Definition, and Architecture (2023) · cites this paper as [64]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Models are commonly made available for serving predictions through a REST [12,16,26,29,40,45] or RPC [12,28,45] API.”
From Operationalizing machine learning models: a systematic literature review (2022) · cites this paper as [45]
6 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Image quality assessment: from error visibility to structural similarity
Wang, Z., Bovik, A. C., Sheikh, H. R., & Simoncelli, E. P. (2004) · IEEE Transactions on Image Processing · doi:10.1109/tip.2003.819861 · Cited by 57,892 · cited 1×
“Brief Metric Description SSIM [46] Predict image quality by measuring the structural similarity between two images.”
BackgroundFSIM: A Feature Similarity Index for Image Quality Assessment
Zhang, L., Zhang, L., Mou, X., & Zhang, D. (2011) · IEEE Transactions on Image Processing · doi:10.1109/tip.2011.2109730 · Cited by 5,203
Image information and visual quality
Sheikh, H. R., & Bovik, A. C. (2006) · IEEE Transactions on Image Processing · doi:10.1109/tip.2005.859378 · Cited by 4,104
Bergstra, J., Yamins, D. L. K., & Cox, D. (2013) · Digital Access to Scholarship at Harvard (DASH) (Harvard University) · Cited by 3,163
Most apparent distortion: full-reference image quality assessment and the role of strategy
Larson, E. C., & Chandler, D. M. (2010) · Journal of Electronic Imaging · doi:10.1117/1.3267105 · Cited by 1,953 · cited 1×
“MAD [55] The most apparent distortion metric utilizes contrast sensitivity, local luminance, and contrast masking for high-resolution images.”
BackgroundGradient Magnitude Similarity Deviation: A Highly Efficient Perceptual Image Quality Index
Xue, W., Zhang, L., Mou, X., & Bovik, A. C. (2014) · IEEE Transactions on Image Processing · doi:10.1109/tip.2013.2293423 · Cited by 1,697
Learning under Concept Drift: A Review
Lu, J., Liu, A., Dong, F., Gu, F., Gama, J., & Zhang, G. (2018) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2018.2876857 · Cited by 1,603 · cited 1×
“As in ML, the performance of the model deployed might degrade with time due to specific phenomena such as concept drift, and data drift [31].”
BackgroundAutomated Machine Learning: Methods, Systems, Challenges
(2019) · The Springer Series on Challenges in Machine Learning · doi:10.1007/978-3-030-05318-5 · Cited by 1,485
Information content weighting for perceptual image quality assessment
Wang, Z., & Li, Q. (2011) · IEEE Transactions on Image Processing · doi:10.1109/tip.2010.2092435 · Cited by 1,304
VSI: A Visual Saliency-Induced Index for Perceptual Image Quality Assessment
Zhang, L., Shen, Y., & Li, H. (2014) · IEEE Transactions on Image Processing · doi:10.1109/tip.2014.2346028 · Cited by 1,029 · cited 1×
“VSI [57] Visual Saliency-Induced Index uses ‘saliency map’ as a quality feature and weighting function to characterize the importance of local regions.”
Research works printed but not linked: 6
- 2. Goyal, A. Machine Learning Operations. In International Journal of Information Technology Insights & Transformations [ISSN: 2581-5172 (Online)] ; Eureka Journals: Pune, India 2020; Volume 4.
- 10. Zhao, Y. Machine Learning in Production: A Literature Review. 2021. Available online: https://staff.fnwi.uva.nl/a.s.z.belloum/ LiteratureStudies/Reports/2021-LiteratureStudy-report-Yizhen.pdf (accessed on 27 July 2021).
- 13. Sureddy, M.R.; Yallamula, P. A Framework for Monitoring Data Warehousing Applications. Int. Res. J. Eng. Technol. 2020 , 7 , 7023-7029.
- 22. Zöller, M.A.; Huber, M.F. Survey on automated machine learning. arXiv 2019 , arXiv:1904.12054. · arXiv
- 30. Treveil, M.; Omont, N.; Stenac, C.; Lefevre, K.; Phan, D.; Zentici, J.; Lavoillotte, A.; Miyazaki, M.; Heidmann, L. Introducing MLOps ; O'Reilly Media: Sebastopol, CA, USA, 2020.
- 42. Gudivada, V.; Apon, A.; Ding, J. Data quality considerations for big data and machine learning: Going beyond data cleaning and transformations. Int. J. Adv. Softw. 2017 , 10 , 1-20.
Cited by
Machine Learning Operations (MLOps): Overview, Definition, and Architecture
Kreuzberger, D., Kühl, N., & Hirschl, S. (2023) · IEEE Access · doi:10.1109/access.2023.3262138 · Cited by 766
MLOps - Definitions, Tools and Challenges
Symeonidis, G., Nerantzis, E., Kazakis, A., & Papakostas, G. A. (2022) · IEEE Annual Computing and Communication Workshop and Conference (CCWC) · doi:10.1109/ccwc54503.2022.9720902 · Cited by 227
“can be extremely effective [9], but there are still some pros and cons.”
BackgroundInfluentialMLOps: A Taxonomy and a Methodology
Testi, M., Ballabio, M., Frontoni, E., Iannello, G., Moccia, S., Soda, P., & Vessio, G. (2022) · IEEE Access · doi:10.1109/access.2022.3181730 · Cited by 114
“The model may become less accurate because the data used in training the model are no longer representative of the new data existing in production [71].”
BackgroundMethodDemocratizing artificial intelligence: How no-code AI can leverage machine learning operations
Sundberg, L., & Holmström, J. (2023) · Business Horizons · doi:10.1016/j.bushor.2023.04.003 · Cited by 118
Shankar, S., Garcia, R., Hellerstein, J. M., & Parameswaran, A. (2024) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3653697 · Cited by 154
“To address such challenges, other MLOps papers have surveyed the proliferating number of industry-originated tools in MLOps [53, 80, 83, 98].”
BackgroundDeep learning for detecting macroplastic litter in water bodies: A review
Jia, T., Kapelan, Z. S., de Vries, R., Vriend, P., Peereboom, E. C., Okkerman, I., & Taormina, R. (2023) · Water Research · doi:10.1016/j.watres.2023.119632 · Cited by 127
MLOps: A ReviewPreprint
Wazir, S., Kashyap, G. S., & Saxena, P. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2308.10908 · Cited by 133
“org/ also does not keep user organisation and does not provide full updation, unlike clustering tests [36].”
BackgroundMethodResilience and Resilient Systems of Artificial Intelligence: Taxonomy, Models and Methods
Москаленко, В. В., Kharchenko, V. S., Москаленко, А. С., & Kuzikov, B. (2023) · Algorithms · doi:10.3390/a16030165 · Cited by 89
Cognitive Soil Digital Twin for Monitoring the Soil Ecosystem: A Conceptual Framework
Tsakiridis, N., Samarinas, N., Kalopesa, E., & Zalidis, G. C. (2023) · Soil Systems · doi:10.3390/soilsystems7040088 · Cited by 86
MLOps best practices, challenges and maturity models: A systematic literature review
Zarour, M., Alzabut, H., & Al-Sarayreh, K. T. (2025) · Information and Software Technology · doi:10.1016/j.infsof.2025.107733 · Cited by 91
Artificial Intelligence in Industry 4.0: A Review of Integration Challenges for Industrial Systems
Windmann, A., Wittenberg, P., Schieseck, M., & Niggemann, O. (2024) · IEEE International Conference on Industrial Informatics (INDIN) · doi:10.1109/indin58382.2024.10774364 · Cited by 83
Machine learning operations landscape: platforms and tools
Berberi, L., Kozlov, V. Y., Nguyen, G., Díaz, J. S.-P., Calatrava, A., Moltó, G., Tran, V., & García, Á. L. (2025) · Artificial Intelligence Review · doi:10.1007/s10462-025-11164-3 · Cited by 69
“Therefore, manual processes to create and control ML systems can cause additional costs, lack of reproducibility, and future problems in maintaining them (Ruf et al. 2021).”
Cob-Parro, A. C., Lalangui, Y., & Lazcano, R. (2024) · Agronomy · doi:10.3390/agronomy14020259 · Cited by 61
Westergaard, G., Erden, U., Mateo, O. A., Lampo, S. M., Akıncı, T. Ç., & Topsakal, O. (2024) · Information · doi:10.3390/info15010039 · Cited by 53
Liang, P., Song, B., Zhan, X., Zhou, C. W., & Yuan, J. (2024) · Applied and Computational Engineering · doi:10.54254/2755-2721/67/20240690 · Cited by 53
What drives MLOps adoption? An analysis using the TOE framework
Das, S. D., & Bala, P. K. (2023) · Journal of Decision System · doi:10.1080/12460125.2023.2214306 · Cited by 57
“The MLOps tools can reduce this overhead on the ML team by automating a number of time-consuming tasks related to model monitoring, maintenance and ML system downtime (Ruf et al., 2021; Subramanya et al., 2022).”
Towards Trustworthy Machine Learning in Production: An Overview of the Robustness in MLOps Approach
Bayram, F., & Ahmed, B. S. (2025) · ACM Computing Surveys · doi:10.1145/3708497 · Cited by 48
“MLOps is an extension of DevOps, which is a well-known and mature topic in software engineering, to run ML systems in production [135].”
Intelligent Edge Computing and Machine Learning: A Survey of Optimization and Applications
Cajas, S. A., Samanta, J., Suárez‐Cetrulo, A. L., & Carbajo, R. S. (2025) · Future Internet · doi:10.3390/fi17090417 · Cited by 57
“Unlike standard software design, ML solutions require data relationships that challenge long-term maintenance in network environments [108].”
An analysis of the challenges in the adoption of MLOps
Amrit, C., & Narayanappa, A. K. (2024) · Journal of Innovation & Knowledge · doi:10.1016/j.jik.2024.100637 · Cited by 56
Mcmurray, S., & Sodhro, A. H. (2023) · Sensors · doi:10.3390/s23073470 · Cited by 45
“[11] argued that there is no single tool for fully-automated MLOps workflow implementation, and the availability of several tools showed overlapping features which increased redundancy.”
MethodInfluentialEdge intelligence secure frameworks: Current state and future challenges
Villar-Rodríguez, E., Arostegi, M., Torre-Bastida, A. I., Regueiro, C., & López-de-Armentia, J. (2023) · Computers & Security · doi:10.1016/j.cose.2023.103278 · Cited by 53
Machine Learning Operations: A Survey on MLOps Tool Support
Hewage, N., & Meedeniya, D. (2022) · arXiv · doi:10.48550/arxiv.2202.10169 · Cited by 38
A Systematic Review of Machine Learning DevOps
Mboweni, T., Masombuka, T., & Dongmo, C. (2022) · 2022 International Conference on Electrical, Computer and Energy Technologies (ICECET) · doi:10.1109/icecet55527.2022.9872968 · Cited by 34
“Batch #4: Zhou, Ding and October (2020) [35]; Ruf et al., (2021) [36]; Hewage and Meedeniya (2022) [37] Yes -”
BackgroundOperationalizing machine learning models: a systematic literature review
Kolltveit, A. B., & Li, J. (2022) · Proceedings of the 1st Workshop on Software Engineering for Responsible AI · doi:10.1145/3526073.3527584 · Cited by 34
“The model may be integrated directly into the application code [19, 29, 45], traditionally first having to be rewritten for production [21].”
BackgroundMethodInfluentialMLHOps: Machine Learning Health Operations
Khattak, F. K., Subasri, V., Krishnan, A., Pou-Prom, C., Koçak, S. A., Dolatabadi, E., Pandya, D., Seyyed-Kalantari, L., & Rudzicz, F. (2024) · IEEE Access · doi:10.1109/access.2024.3521279 · Cited by 39
“Recently, MLOps has become more well-defined and widely implemented due to the reusability and standardization benefits across various applications [253].”
Conceptualizing the Secure Machine Learning Operations (SecMLOps) Paradigm
Zhang, X., & Jaskolka, J. (2022) · IEEE International Conference on Software Quality, Reliability and Security (QRS) · doi:10.1109/qrs57517.2022.00023 · Cited by 40
“The MLOps workflow or process, maturity level models, tools, and platforms from both grey and scientific literature are presented in [19], [20], and an additional comparison and selection of tools for meeting specific requirements of MLOps is presented in [21].”
BackgroundMethodUnlabeled learning algorithms and operations: overview and future trends in defense sector
Oliveira, E. E., Rodrigues, M., Pereira, J. P., Lopes, A. M., Mestric, I. I., & Bjelogrlic, S. (2024) · Artificial Intelligence Review · doi:10.1007/s10462-023-10692-0 · Cited by 29
“MLOps is a recent phenomenon with a wide spectrum of activities, supported by different software tools, and as new tools appear with different purposes and objectives, the responsibilities and requirements become less clear (Ruf et al. 2021).”
InfluentialAn empirical study of challenges in machine learning asset management
Zhao, Z., Chen, Y., Bangash, A. A., Adams, B., & Hassan, A. E. (2024) · Empirical Software Engineering · doi:10.1007/s10664-024-10474-4 · Cited by 31
Toward Sociotechnical AI: Mapping Vulnerabilities for Machine Learning in Context
Dobbe, R., & Wolters, A. (2024) · Minds and Machines · doi:10.1007/s11023-024-09668-y · Cited by 25
Oakes, B. J., Famelis, M., & Sahraoui, H. A. (2024) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3638243 · Cited by 30
Big Data-driven MLOps workflow for annual high-resolution land cover classification models
Burgueño-Romero, A. M., Barba-González, C., & Aldana-Montes, J. F. (2024) · Future Generation Computer Systems · doi:10.1016/j.future.2024.107499 · Cited by 25
MLOps: a guide to its adoption in the context of responsible AI
Matsui, B. M. A., & Goya, D. (2022) · doi:10.1145/3526073.3527591 · Cited by 32
“When they exist, they are mostly focused on specific aspects rather than broader characteristics of MLOps itself, as in [26], which mainly addresses CI/CD concepts in MLOps for AIoT (Artificial Intelligence of Things) applications, or [29], which focuses on MLOps tools.”
BackgroundML-Enabled Systems Model Deployment and Monitoring: Status Quo and Problems
Zimelewicz, E., Kalinowski, M., Méndez, D., Giray, G., Alves, A. P. S., Lavesson, N., Azevedo, K., Villamizar, H., Escovedo, T., Lopes, H., et al. (2024) · Lecture notes in business information processing · doi:10.1007/978-3-031-56281-5_7 · Cited by 32
Díaz-de-Arcaya, J., Torre-Bastida, A. I., Miñón, R., & Almeida, A. (2022) · Future Generation Computer Systems · doi:10.1016/j.future.2022.10.008 · Cited by 25
Dynamic data management for continuous retraining
Baumann, N., Kusmenko, E., Ritz, J., Rumpe⋆, B., & Weber, M. B. (2022) · ACM/IEEE International Conference on Model Driven Engineering Languages and Systems Companion (MODELS-C) · doi:10.1145/3550356.3561568 · Cited by 23
“A selection of tools from [28] with ML focus is presented in the following.”
MethodPredicting the Type of Road Traffic Accident for Test Scenario Generation
Bäumler, M., & Prokop, G. (2024) · IEEE Access · doi:10.1109/access.2024.3367744 · Cited by 19
“Initially, we selected the CatBoost and BERT PTM based on an exploratory data analysis, as proposed in [60].”
MethodMLOps: A Multiple Case Study in Industry 4.0
Faubel, L., & Schmid, K. (2024) · IEEE International Conference on Emerging Technologies and Factory Automation (ETFA) · doi:10.1109/etfa61755.2024.10711136 · Cited by 24
“The general idea behind MLOps is to utilize automation to bring machine learning models into production in an efficient manner, as well as to ensure quality improvement of these models both before and after deployment [5]–[7].”
BackgroundWarnett, S. J., Ntentos, E., & Zdun, U. (2024) · Journal of Systems and Software · doi:10.1016/j.jss.2024.112257 · Cited by 18
“Nevertheless, MLOps is still at a somewhat early stage, and there is only sometimes agreement or consensus regarding what MLOps as a relatively new set of practices actually means [18, 24, 25, 29].”
Requirements and Reference Architecture for MLOps:Insights from IndustryPreprint
Kumara, I., Arts, R., Nucci, D. D., Kazman, R., Heuvel, W. J. V. D., & Tamburri, D. A. (2022) · doi:10.36227/techrxiv.21397413.v1 · Cited by 28
Tran, T.-A., Ruppert, T., & Abonyi, J. (2024) · Computers · doi:10.3390/computers13100252 · Cited by 20
Godwin, R. C., & Melvin, R. L. (2024) · SoftwareX · doi:10.1016/j.softx.2024.101723 · Cited by 19
MLOps in Data Science Projects: A Review
Haertel, C., Staegemann, D. G., Daase, C., Pohl, M., Nahhas, A., & Turowski, K. (2023) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata59044.2023.10386139 · Cited by 16
“In particular, emphasis should be put on mapping the project requirements to the required technologies and architectures as well suitable evaluation metrics [54, 61], constituting the fundament for ensuring better MLOps and DS effectiveness.”
BackgroundMLOps critical success factors - A systematic literature review
Mehmood, Y., Sabahat, N., & Ijaz, M. A. (2024) · VFAST Transactions on Software Engineering · doi:10.21015/vtse.v12i1.1747 · Cited by 20
MLOps: Overview of Current State and Future Directions
Bodor, A., Hnida, M., & Daoudi, N. (2023) · Lecture notes in networks and systems · doi:10.1007/978-3-031-26852-6_14 · Cited by 14
MLOps in freight rail operations
Pineda-Jaramillo, J., & Viti, F. (2023) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2023.106222 · Cited by 14
MLOps-Enabled Security Strategies for Next-Generation Operational Technologies
Ahmad, T., Adnan, M., Rafi, S., Akbar, M. A., & Anwar, A. (2024) · International Conference on Evaluation and Assessment in Software Engineering (EASE) · doi:10.1145/3661167.3661283 · Cited by 17
“It is essential to include security measures in these pipelines to quickly identify and address vulnerabilities, promoting a proactive approach to security [26].”
BackgroundEnd-to-end lifecycle machine learning framework for predictive maintenance of critical equipment
Marchand, J., Laval, J., Sekhari, A., Cheutet, V., & Danielou, J.-B. (2025) · Enterprise Information Systems · doi:10.1080/17517575.2024.2448008 · Cited by 15
“In addition to CI and CD, MLOps introduces also a new practice known as ‘continuous training’ and ‘continuous monitoring’, which aims to automatically and continuously retrain the model when data drift or when the model performance is degrading (Ruf et al. 2021).”
Model Provenance Management in MLOps Pipeline
Mei, S., Liu, C., Wang, Q., & Su, H. (2022) · doi:10.1145/3512850.3512861 · Cited by 12
“In [7], a potential implementation of MLOps pipeline was presented using open source tools and platforms.”
MethodOn-Premise Artificial Intelligence as a Service for Small and Medium Size Setups
Fortuna, C., Mušić, D., Cerar, G., Čampa, A., Kapsalis, P., & Mohorčič, M. (2023) · Engineering cyber-physical systems and critical infrastructures · doi:10.1007/978-3-031-29301-6_3 · Cited by 11
Advancing MLOps from Ad hoc to Kaizen
John, M. M., Gillblad, D., Olsson, H. H., & Bosch, J. (2023) · Euromicro Conference on Software Engineering and Advanced Applications (SEAA) · doi:10.1109/seaa60479.2023.00023 · Cited by 12
The Environmental Cost of Engineering Machine Learning-Enabled Systems: A Mapping Study
Chadli, K., Botterweck, G., & Saber, T. (2024) · Workshop on Machine Learning and Systems (EuroMLSys) · doi:10.1145/3642970.3655828 · Cited by 17
“This means models are automatically updated based on new data, ensuring they stay effective in real-world scenarios [50].”
BackgroundNeghawi, E., Wang, Z., Huang, J., & Liu, Y. (2023) · IEEE Annual International Computer Software and Applications Conference (COMPSAC) · doi:10.1109/compsac57700.2023.00185 · Cited by 10
“Furthermore, we evaluated an examination of MLOps technologies and tools for each stage of the project pipeline and the roles involved [14].”
MethodToward an Open Source MLOps Architecture
Burgueño-Romero, A. M., Benítez‐Hidalgo, A., Barba-González, C., & Aldana-Montes, J. F. (2024) · IEEE Software · doi:10.1109/ms.2024.3421675 · Cited by 9
“The modularity, flexibility, and transparency of emerging OS solutions are unparal-leled [5].”
Exploring Trade-Offs in MLOps Adoption
John, M. M., Olsson, H. H., Bosch, J., & Gillblad, D. (2023) · Asia-Pacific Software Engineering Conference (APSEC) · doi:10.1109/apsec60848.2023.00047 · Cited by 11
An overview and solution for democratizing AI workflows at the network edge
Čop, A., Bertalanič, B., & Fortuna, C. (2025) · Journal of Network and Computer Applications · doi:10.1016/j.jnca.2025.104180 · Cited by 10
“The data storage and data acquisition process, often referred to as Big Data , is an initial collection of data and storage along with data processing and depending on the definition can be a part of MLOps or a separate data preparation process [42, 51].”
InfluentialZhang, X., Zhao, P., Jaskolka, J., Li, H., & Lu, R. (2026) · Empirical Software Engineering · doi:10.1007/s10664-025-10795-y · Cited by 11
“Organizations lacking established ML pipelines or security expertise may find the eight specialized roles and comprehensive processes overwhelming (Ruf et al. 2021) to implement simultaneously.”
MLOps Pipeline Development: The OSSARA Use Case
Moreschini, S., Hästbacka, D., & Taibi, D. (2023) · ACM Conference on Research in Adaptive and Convergent Systems (RACS) · doi:10.1145/3599957.3606211 · Cited by 8
“In [19], with the aim of presenting the potential of MLOps, the authors compared a specific set of tools for MLOps.”
MethodRequirements and Reference Architecture for MLOps:Insights from IndustryPreprint
Kumara, I., Arts, R., Nucci, D. D., Heuvel, W. J. V. D., & Tamburri, D. A. (2022) · doi:10.36227/techrxiv.21397413 · Cited by 10
Towards MLOps: A DevOps Tools Recommender System for Machine Learning SystemPreprint
Shah, P. S. U., Ahmad, N., & Beg, M. O. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2402.12867 · Cited by 8
“MLOps pipeline can be constructed by the combination of open-source tools and the availability of open-source tools is abundant [2] and the number of the tools is expected to be increased further.”
BackgroundEnd-to-end data quality-driven framework for machine learning in production environment
Bayram, F., Ahmed, B. S., & Hallin, E. (2025) · Heliyon · doi:10.1016/j.heliyon.2025.e44416 · Cited by 10
“Continuous monitoring and logging track the performance and health of ML models in production by capturing metrics related to model predictions, data quality, and system performance, which help detect anomalies and diagnose issues [40].”
Leveraging AI for Accurate Time Series Forecasting
Ehsan, A. (2025) · International Journal for Global Academic & Scientific Research · doi:10.55938/ijgasr.v3i4.154 · Cited by 7
“Employing a dual model mirrors LSTMs' malleability and diversity in working with a range of datasets with different temporal structures [9] .”
DevOps Practices Into Machine Learning
Jain, S., & Kumar, P. (2024) · IEEE International Conference on Intelligent Systems, Smart and Green Technologies (ICISSGT) · doi:10.1109/icissgt58904.2024.00029 · Cited by 7
Aspects of Module Placement in Machine Learning Operations for Cyber Physical Systems
Ruf, P., Reich, C., & Abdeslam, D. O. (2022) · Mediterranean Conference on Embedded Computing (MECO) · doi:10.1109/meco55406.2022.9797080 · Cited by 7
Self-citationAn Exploratory Study of Dataset and Model Management in Open Source Machine Learning Applications
Toma, T. R., & Bezemer, C. (2024) · IEEE/ACM International Conference on AI Engineering – Software Engineering for AI (CAIN) · doi:10.1145/3644815.3644963 · Cited by 8
“Many recent studies have compared these ML AMSs from different perspectives [9–11, 15, 20, 23–26, 28].”
BackgroundFeature Management for Machine Learning Operation Pipelines in AI Native Networks
Cerar, G., Bertalanič, B., Mohorčič, M., Grobelnik, M., & Fortuna, C. (2023) · International Balkan Conference on Communications and Networking (BalkanCom) · doi:10.1109/balkancom58402.2023.10167936 · Cited by 6
“The main steps of KDP consist of 1) data analysis, 2) data preparation (pre-processing), 3) model training and evaluation and 4) model deployment [2] as also represented in Figure 1.”
MethodInfluentialMachine Learning Operations: A Mapping Study
Chakraborty, A., Das, S., & Gary, K. A. (2025) · Communications in computer and information science · doi:10.1007/978-3-031-86644-9_1 · Cited by 8
On the Democratization of Machine Learning Pipelines
Carqueja, A., Cabral, B., Fernandes, J. P., & Lourenço, N. (2022) · IEEE Symposium Series on Computational Intelligence (SSCI) · doi:10.1109/ssci51031.2022.10022107 · Cited by 9
“This poses an issue, as there can be multiple valid workflows that tackle an arbitrary ML problem, as corroborated by the different representations given by several authors ([11], [12], [14]).”
BackgroundMethodModular oversight methodology: a framework to aid ethical alignment of algorithmic creations
Kyriakou, K., & Otterbacher, J. (2024) · Design Science · doi:10.1017/dsj.2024.23 · Cited by 6
“Research has demonstrated that the accuracy of an ML model may decay over time because of the nonrepresentative training data compared to the new data in production (Ruf et al. 2021; Matsui & Goya 2022b).”
A Conceptual Architecture for AI in Supply Chain Risk Management
Paul, S. K., Riaz, S., & Das, S. (2022) · IEEE Region 10 Conference (TENCON) · doi:10.1109/tencon55691.2022.9978154 · Cited by 5
“MLOps refers to a combination of capabilities encompassing ML, Development Operations (DevOps), and data engineering to enable automation of the ML pipeline and reduce the manual tasks [14].”
End-to-End Automation of ML Model Lifecycle Management using Machine Learning Operations Platforms
Hsu, C.-C., Chen, P., & Wu, I.-Z. (2024) · IEEE International Conference on Consumer Electronics - Taiwan (ICCE-Taiwan) · doi:10.1109/icce-taiwan62264.2024.10674445 · Cited by 5
“There are several MLOps platforms available, including Kubeflow, MLflow, Metaflow, Flyte, and MLReef [1], [2], [3].”
MethodReddy, R., Munoz, Y., Lipps, C., & Schotten, H. D. (2023) · International Balkan Conference on Communications and Networking (BalkanCom) · doi:10.1109/balkancom58402.2023.10167924 · Cited by 5
Sustainable Engineering of Machine Learning-Enabled Systems: A Systematic Mapping StudyPreprint
Chadli, K., Botterweck, G., & Saber, T. (2024) · Research Square · doi:10.21203/rs.3.rs-4694122/v1 · Cited by 8
An architectural perspective on MLOps: Structures, processes, tools, and stakeholders
Najafabadi, F. A., Bogner, J., Gerostathopoulos, I., & Lago, P. (2026) · Information and Software Technology · doi:10.1016/j.infsof.2026.108029 · Cited by 4
Methodology applied to computer audit with artificial intelligence: a systematic review
Suárez, S. L. R., Huamani, B. M., Meléndez, M. A., & Ovalle, C. (2024) · IAES International Journal of Artificial Intelligence · doi:10.11591/ijai.v13.i4.pp3727-3738 · Cited by 4
“…[86] 2 Performance in projects with MLOps [17], [27], [28], [30], [33], [35], [37], [39], [41], [42], [45], [48]–[50], [52], [53], [55], [56], [58]-[61], [68], [69], [71], [73]–[75], [78]–[82], [85], [87], [88], [90], [92]–[94], [105], [106] 3 Detection of anomalies in projects with MLOps [10],…”
Testi, M., Fiorentino, M. C., Ballabio, M., Visani, G., Ciccozzi, M., Frontoni, E., Moccia, S., & Vessio, G. (2025) · Medical & Biological Engineering & Computing · doi:10.1007/s11517-025-03436-5 · Cited by 5
Towards Integrated Model-Based Machine Learning Experimentation Framework
Runpakprakun, J., Husen, J. H., Washizaki, H., Yoshioka, N., & Fukazawa, Y. (2023) · International Conference on Dependable Systems and Their Applications (DSA) · doi:10.1109/dsa59317.2023.00086 · Cited by 4
FAIRness Along the Machine Learning Lifecycle Using Dataverse in Combination with MLflow
Sherpa, L., Khaydarov, V., & Müller-Pfefferkorn, R. (2024) · Data Science Journal · doi:10.5334/dsj-2024-055 · Cited by 5
Reusing ML Models in Dynamic Data Environments: Data Similarity-Based Approach for Efficient MLOps
Peixoto, E., Torres, D., Carneiro, D. R., Silva, B., & Marques, R. (2025) · Big Data and Cognitive Computing · doi:10.3390/bdcc9020047 · Cited by 6
“These practices are essential for scaling machine learning solutions and maintaining their performance in real-world applications [7].”
Miñón, R., Díaz-de-Arcaya, J., Torre-Bastida, A. I., Zárate, G., & Moreno-Fernández-de-Leceta, A. (2022) · International Conference on Smart and Sustainable Technologies (SpliTech) · doi:10.23919/splitech55088.2022.9854211 · Cited by 7
“Fortunately, recently, the MLOps methodology [1] has emerged aiming at minimizing the existing leap between laboratory proof of concept and production-real AI products.”
Drawing and Analysis of Bounding Boxes for Object Detection with Anchor-Based Models
Madan, M., Reich, C., & Hassenpflug, F. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-31435-3_24 · Cited by 6
Self-citationEmpowering Machine Learning Development with Service-Oriented Computing Principles
Yousefi, M. H. N., Degeler, V., & Lazovik, A. (2023) · Communications in computer and information science · doi:10.1007/978-3-031-45728-9_2 · Cited by 5
A Machine Learning Operations (MLOps) Monitoring Model Using BI-LSTM and SARSA Algorithms
Elgamal, Z. S., ElFangary, L. M., & Fahmy, H. (2024) · International Journal of Advanced Computer Science and Applications · doi:10.14569/ijacsa.2024.0151060 · Cited by 3
“Continuous delivery (CD) core purpose is to deliver newly created features to the end user as rapidly as possible by building the software in a way that is constantly in a production-ready state to ensure that code updates might be released on demand fast and safely [15] [20] [21].”
BackgroundMethodInfluentialUSING MLOPS FOR DEPLOYMENT OF OPINION MINING MODEL AS A SERVICE FOR SMART CITY APPLICATIONS
Mussina, A., Yedilkhan, D., Alimzhanov, Y., Nugumanova, A., Aubakirov, S., & Mansurova, A. (2025) · Scientific Journal of Astana IT University · doi:10.37943/21cpqx5616 · Cited by 3
“In order to use reliable data, the quality of which is guaranteed, the authors pay special attention to the problems of implementing data versioning. b. Instrumental research One of the most detailed reviews of development tools for MLOps projects is presented in [26].”
Git Workflow for Active Learning: A Development Methodology Proposal for Data-Centric AI Projects
Stieler, F., & Bauer, B. (2023) · International Conference on Evaluation of Novel Approaches to Software Engineering · doi:10.5220/0011988400003464 · Cited by 3
Esteban, A., Pardo, C., Suescún, E., & Villarreal, V. (2025) · IEEE Congreso Internacional en Inteligencia Ambiental, Ingeniería de Software y Salud Electrónica y Móvil (AmITIC) · doi:10.1109/amitic68284.2025.11214628 · Cited by 2
“…ML within any of its processes); therefore, how products and/or services have been developed must evolve, since apart from code, ML projects depend on datasets, thus a definition of new roles, new activities, new artifacts that traditional and/or agile methodologies do not cover is needed [2], [3].”
A Systematic Review of MLOps Tools: Tool Adoption, Lifecycle Coverage, and Critical Insights
Micallef, Z., Rajenthiram, K., & Gerostathopoulos, I. (2025) · arXiv (Cornell University) · doi:10.5281/zenodo.15459744 · Cited by 2
An Approach to Experiment Reproducibility Through MLOps and Semantic Web Technologies
Seaman, D., Peñafiel, D., Palacio‐Baus, K., & Saquicela, V. (2023) · Latin American Computer Conference (CLEI) · doi:10.1109/clei60451.2023.10346140 · Cited by 2
“In MLOps, in addition to CI and CD, CT (Continuous Training) is included, since in most cases the Machine Learning model requires to be retrained several times [3].”
BackgroundMethodInfluentialMeta-Analysis of the Machine Learning Operations Open Source Ecosystem
Zimmerman, I., Silge, J. D., Abedin, P., & Sanchez-Arias, R. (2023) · International Conference on Machine Learning and Applications · doi:10.1109/icmla58977.2023.00136 · Cited by 2
“Another opinion is that each tool falls into one (or sometimes multiple) categories to fulfill these tasks: all-in-one platform, data pipeline, infrastructure, modeling and training, monitoring, and serving [9].”
BackgroundChallenges and Solutions in Cross-Platform ML Systems Integration
Patel, P. (2023) · Journal of Artificial Intelligence & Cloud Computing · doi:10.47363/jaicc/2023(2)233 · Cited by 2
Security Concerns: Machine Learning Approaches in the Modern Era and Web Engineering
Ashraf, H. (2024) · Advances in web technologies and engineering book series · doi:10.4018/979-8-3693-3703-5.ch012 · Cited by 2
Danciu, G. M., Nicolae, I.-E., Ilie, I., & Nechifor, C. S. (2023) · International Conference on Applied Mathematics & Computer Science (ICAMCS) · doi:10.1109/icamcs59110.2023.00013 · Cited by 2
Enhancing the MLOPS Deployment Process Using Gen AI
Pilaniwala, P., Chhabra, G., & Naik, A. (2024) · doi:10.5121/csit.2024.142002 · Cited by 1
“Ruf et al. examined and presented their findings on selecting the best open-source tools that are available for various operational and execution phases of MLOps [11].”
BackgroundKEENsight: Cloud Based Collaborative Environment for Streamlining Machine Learning Development
Sherpa, L., Attar-Khorasani, S., Sankar, R., Müller-Pfefferkorn, R., & Ghiasvand, S. (2024) · International Conference on Information, Intelligence, Systems & Applications (IISA) · doi:10.1109/iisa62523.2024.10786688 · Cited by 2
“A comparative study of different platforms can be found in [13], [14].”
Machine Learning Lifecycle Explorer: Leverage the Lifecycle Metadata (onto) Logically
Charoenwut, P., Drobnjaković, M., Oh, H., Nikolov, A., & Kulvatunyou, B. (2025) · IFIP advances in information and communication technology · doi:10.1007/978-3-032-03550-9_25 · Cited by 1
“Earlier research by Ruf, P., Madan, M., Reich, C., & Ould-Abdeslam, D. [8] found that while some platforms aimed to cover the entire MLOps workflow, they might not always align with an organization’s specific needs.”
Agiles Arbeiten im öffentlichen Sektor: Erfahrungen aus Deutschland, Österreich und der Schweiz
Hegele, Y., & Stoll, A. (2025) · Interdisziplinäre Organisations- und Verwaltungsforschung · doi:10.1007/978-3-658-48296-1 · Cited by 1
Khaldy, M. A. A., Altarawneh, A., & Shaheen, A. (2026) · International Conference on Computational Intelligence Approaches and Applications (ICCIAA) · doi:10.1109/icciaa68481.2026.11543964 · Cited by 1
“Monitoring integration is also influenced by pipeline maturity; MLOps reviews emphasize that reliable systems depend on structured pipelines, monitoring, and accountability mechanisms [18].”
A Systematic Review of MLOps Tools: Tool Adoption, Lifecycle Coverage, and Critical Insights
Micallef, Z., Rajenthiram, K., & Gerostathopoulos, I. (2026) · IEEE/ACM International Conference on AI Engineering – Software Engineering for AI (CAIN) · doi:10.1145/3793653.3793785 · Cited by 1
“Multiple studies [35] [36] [46] highlight the importance of evaluating not just each tool’s individual strengths but also how well they interoperate with other tools and the dependencies they introduce.”
Design of a Scientific Data Analysis Support Platform
Martindale, N., Hite, J. M., STEWART, S. L., & Adams, M. (2022) · Proceedings of the Python in Science Conferences · doi:10.25080/majora-212e5952-01b · Cited by 2
Design and Development of Modern day Machine Learning Applications - A Survey
Siddeshwara, R. S., Rohit, V. S., Pasha, A., & Manakar, A. S. (2022) · International Journal of Scientific Research in Science Engineering and Technology · doi:10.32628/ijsrset229632 · Cited by 1
Cloud Computing Continuum Research Topics and Challenges. A Multi-source Analysis
Alonso, J., Orue-Echevarria, L., & Areizaga, E. (2022) · Communications in computer and information science · doi:10.1007/978-3-031-23298-5_7 · Cited by 1
Metadata
- Authors
- Ruf, P., Madan, M., Reich, C., & Ould-Abdeslam, D.
- Affiliations
- Furtwangen University (DE)
- Institut de Recherche en Informatique, Mathématiques, Automatique et Signal (IRIMAS), Université de Haute-Alsace, 61 Rue Albert Camus, 68093 Mulhouse, France
- Corresponding author
- Reich, Christoph
- Venue
- Applied Sciences
- Publisher
- MDPI AG
- Published
- 2021-09-23
- Volume
- 11
- Issue
- 19
- Pages
- 8861
- ISSN
- 2076-3417 (ISSN-L)
- OpenAlex
- W3202461888
- Cited by
- 112
- References
- 50
- Citations by year
- 2026: 11 · 2025: 22 · 2024: 33 · 2023: 28 · 2022: 18
- FWCI
- 17.86
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- Applied Sciences (publishedVersion, cc-by)
- Repository of Futwangen University of Applied Sciences (Furtwangen University) (submittedVersion, cc-by)
- HAL (submittedVersion, other-oa)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
- Social Sciences › Decision Sciences › Information Systems and Management › Scientific Computing and Data Management
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Data Classification
- Keywords
- MLOps, open source tools, machine learning pipelines, tool selection, object detection, deep learning, image processing, model evaluation metrics
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.