The DOI matches its Crossref record; 60 of the 61 references it cites were found in a registry or an index, and 1 are not checked yet.
Demystifying MLOps and Presenting a Recipe for the Selection of Open-Source Tools
Open accessAbstract
Nowadays, machine learning projects have become more and more relevant to various real-world use cases. The success of complex Neural Network models depends upon many factors, as the requirement for structured and machine learning-centric project development management arises. Due to the multitude of tools available for different operational phases, responsibilities and requirements become more and more unclear. In this work, Machine Learning Operations (MLOps) technologies and tools for every part of the overall project pipeline, as well as involved roles, are examined and clearly defined. With the focus on the inter-connectivity of specific tools and comparison by well-selected requirements of MLOps, model performance, input data, and system quality metrics are briefly discussed. By identifying aspects of machine learning, which can be reused from project to project, open-source tools which help in specific parts of the pipeline, and possible combinations, an overview of support in MLOps is given. Deep learning has revolutionized the field of Image processing, and building an automated machine learning workflow for object detection is of great interest for many organizations. For this, a simple MLOps workflow for object detection with images is portrayed.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures



Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Packaging models in containers is a commonly reported method of integration, in which the model is accessible through representation state transfer (REST) or remote procedure call (RPC) interfaces [12,16,28,45].”
From Operationalizing machine learning models: a systematic literature review (2022) · cites this paper as [45]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Models packaged in containers are simply run directly as standalone services [16,19,28,29,45].”
From Operationalizing machine learning models: a systematic literature review (2022) · cites this paper as [45]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Polyaxon 9 , MLflow 10 , TFX 11 , ZenML 12 , Flyte 13 , Seldon Core 14 and BentoML 15 are some of the more fully-integrated platforms that exist [10,45], along with the Amazon-exclusive Sage-Maker 16 [9,45].”
From Operationalizing machine learning models: a systematic literature review (2022) · cites this paper as [45]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“In productive settings, it is challenging to operate ML manually due to different stacks of software and hardware components and their interplay as well as the selection of both ( [64], [65].”
From Machine Learning Operations (MLOps): Overview, Definition, and Architecture (2023) · cites this paper as [64]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Models are commonly made available for serving predictions through a REST [12,16,26,29,40,45] or RPC [12,28,45] API.”
From Operationalizing machine learning models: a systematic literature review (2022) · cites this paper as [45]
6 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Image quality assessment: from error visibility to structural similarity
Wang, Z., Bovik, A. C., Sheikh, H. R., & Simoncelli, E. P. (2004) · IEEE Transactions on Image Processing · doi:10.1109/tip.2003.819861 · Cited by 57,892
FSIM: A Feature Similarity Index for Image Quality Assessment
Zhang, L., Zhang, L., Mou, X., & Zhang, D. (2011) · IEEE Transactions on Image Processing · doi:10.1109/tip.2011.2109730 · Cited by 5,203
Image information and visual quality
Sheikh, H. R., & Bovik, A. C. (2006) · IEEE Transactions on Image Processing · doi:10.1109/tip.2005.859378 · Cited by 4,104
Most apparent distortion: full-reference image quality assessment and the role of strategy
Larson, E. C., & Chandler, D. M. (2010) · Journal of Electronic Imaging · doi:10.1117/1.3267105 · Cited by 1,953
Bergstra, J., Yamins, D. L. K., & Cox, D. (2013) · Digital Access to Scholarship at Harvard (DASH) (Harvard University) · Cited by 1,852
Gradient Magnitude Similarity Deviation: A Highly Efficient Perceptual Image Quality Index
Xue, W., Zhang, L., Mou, X., & Bovik, A. C. (2014) · IEEE Transactions on Image Processing · doi:10.1109/tip.2013.2293423 · Cited by 1,697
Learning under Concept Drift: A Review
Lu, J., Liu, A., Dong, F., Gu, F., Gama, J., & Zhang, G. (2018) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2018.2876857 · Cited by 1,603
Automated Machine Learning: Methods, Systems, Challenges
(2019) · The Springer Series on Challenges in Machine Learning · doi:10.1007/978-3-030-05318-5 · Cited by 1,485
Information content weighting for perceptual image quality assessment
Wang, Z., & Li, Q. (2011) · IEEE Transactions on Image Processing · doi:10.1109/tip.2010.2092435 · Cited by 1,304
VSI: A Visual Saliency-Induced Index for Perceptual Image Quality Assessment
Zhang, L., Shen, Y., & Li, H. (2014) · IEEE Transactions on Image Processing · doi:10.1109/tip.2014.2346028 · Cited by 1,029
A Unifying Review of Deep and Shallow Anomaly Detection
Ruff, L., Kauffmann, J. R., Vandermeulen, R. A., Montavon, G., Samek, W., Kloft, M., Dietterich, T. G., & Muller, K.-R. (2021) · Proceedings of the IEEE · doi:10.1109/jproc.2021.3052449 · Cited by 914
(2015) · neural information processing systems · Cited by 912
Automatic Metallic Surface Defect Detection and Recognition with Convolutional Neural Networks
Tao, X., Zhang, D., Ma, W., Liu, X., & Xu, D. (2018) · Applied Sciences · doi:10.3390/app8091575 · Cited by 471
Mons, B., Neylon, C., Velterop, J., Dumontier, M., da Silva Santos, L. O. B., & Wilkinson, M. D. (2017) · Information Services and Use · doi:10.3233/isu-170824 · Cited by 417
TFX: A TensorFlow-Based Production-Scale Machine Learning Platform
Baylor, D., Breck, E., Cheng, H.-T., Fiedel, N., Foo, C. Y., Haque, Z., Haykal, S., Ispir, M., Jain, V., Koc, L., et al. (2017) · Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/3097983.3098021 · Cited by 350
A NEW FULL-REFERENCE QUALITY METRICS BASED ON HVS
Egiazarian, K., Astola, J. T., Ponomarenko, N. N., Лукин, Battisti, F., & Carli, M. (2006) · Cited by 253
Automating large-scale data quality verification
Schelter, S., Lange, D., Schmidt, P., Celikel, M., Biessmann, F., & Grafberger, A. (2018) · Proceedings of the VLDB Endowment · doi:10.14778/3229863.3229867 · Cited by 240
Mattson, P., Cheng, C., Coleman, C., Diamos, G., Micikevicius, P., Patterson, D., Tang, H., Wei, G.-Y., Bailis, P., Bittorf, V., et al. (2019) · arXiv · doi:10.48550/arxiv.1910.01500 · Cited by 206 · cited 1×
“ML is capable of offering multiple correct solutions for a single problem, unlike traditional technologies that offer only one perfect solution [58].”
BackgroundWho Needs MLOps: What Data Scientists Seek to Accomplish and How Can MLOps Help?Preprint
Mäkinen, S., Skogström, H., Laaksonen, E., & Mikkonen, T. (2021) · arXiv (Cornell University) · Cited by 169
Content-partitioned structural similarity index for image quality assessment
Li, C., & Bovik, A. C. (2010) · Signal Processing: Image Communication · doi:10.1016/j.image.2010.03.004 · Cited by 168
Survey and Performance Analysis of Deep Learning Based Object Detection in Challenging Environments
Ahmed, M. S., Hashmi, K. A., Pagani, A., Liwicki, M., Stricker, D., & Afzal, M. Z. (2021) · Sensors · doi:10.3390/s21155116 · Cited by 151
Taleb, I., Serhani, M. A., & Rachida Dssouli, A. (2018) · IEEE International Congress on Big Data (BigData Congress) · doi:10.1109/bigdatacongress.2018.00029 · Cited by 144
Applying DevOps Practices of Continuous Automation for Machine Learning
Karamitsos, I., Thabit, S., & Apostolopoulos, C. (2020) · Information · doi:10.3390/info11070363 · Cited by 140
Data Integration and Machine Learning: A Natural Synergy
Dong, X. L., & Rekatsinas, T. (2018) · ACM SIGMOD International Conference on Management of Data (SIGMOD) · doi:10.1145/3183713.3197387 · Cited by 132
Sustainable MLOps: Trends and Challenges
Tamburri, D. A. (2020) · 2020 22nd International Symposium on Symbolic and Numeric Algorithms for Scientific Computing (SYNASC) · doi:10.1109/synasc51798.2020.00015 · Cited by 124
Three-component weighted structural similarity index
Li, C., & Bovik, A. C. (2009) · Proceedings of SPIE, the International Society for Optical Engineering/Proceedings of SPIE · doi:10.1117/12.811821 · Cited by 110
Data Stewardship for Open Science: Implementing FAIR Principles
Mons, B. (2018) · Chapman & Hall/CRC eBooks · doi:10.1201/9781315380711 · Cited by 107
Concept Drift Adaptation Techniques in Distributed Environment for Real-World Data Streams
Mehmood, H., Kostakos, P., Cortés, M., Anagnostopoulos, T., Pirttikangas, S., & Gilman, E. (2021) · Smart Cities · doi:10.3390/smartcities4010021 · Cited by 98
A Model for Data Governance - Organising Accountabilities for Data Quality Management
Wende, K. (2007) · Alexandria (UniSG) (University of St.Gallen) · Cited by 83
Pergl, R., Hooft, R. W. W., Suchánek, M., Knaisl, V., & Slifka, J. (2019) · Data Science Journal · doi:10.5334/dsj-2019-059 · Cited by 66
How Much Automation Does a Data Scientist Want?Preprint
Wang, D., Liao, Q. V., Zhang, Y., Khurana, U., Samulowitz, H., Park, S., Muller, M. J., & Amini, L. D. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2101.03970 · Cited by 66
On the Co-evolution of ML Pipelines and Source Code - Empirical Study of DVC Projects
Barrak, A., Eghan, E. E., & Adams, B. (2021) · IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER) · doi:10.1109/saner50967.2021.00046 · Cited by 61
Introduction: Measuring Data Quality for Ongoing Improvement
Sebastian-Coleman, L. (2013) · Elsevier eBooks · doi:10.1016/b978-0-12-397033-6.00020-1 · Cited by 57
Image/Video Deep Anomaly Detection: A SurveyPreprint
Mohammadi, B., Fathy, M., & Sabokrou, M. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2103.01739 · Cited by 57
Towards a Full-Reference Quality Assessment for Color Images Using Directional Statistics
Lee, D., & Plataniotis, K. N. (2015) · IEEE Transactions on Image Processing · doi:10.1109/tip.2015.2456419 · Cited by 54
MLOps Challenges in Multi-Organization Setup: Experiences from Two Real-World Cases
Granlund, T., Kopponen, A., Stirbu, V., Myllyaho, L., & Mikkonen, T. (2021) · 2021 IEEE/ACM 1st Workshop on AI Engineering - Software Engineering for AI (WAIN) · doi:10.1109/wain52551.2021.00019 · Cited by 48
Ramasubramanian, K., & Singh, A. (2016) · Apress eBooks · doi:10.1007/978-1-4842-2334-5 · Cited by 41
MLOps Challenges in Multi-Organization Setup: Experiences from Two\n Real-World CasesPreprint
Granlund, T., Kopponen, A., Stirbu, V., et al. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2103.08937 · Cited by 40
Peng, G., Ritchey, N. A., Casey, K. S., Kearns, E. J., Privette, J. L., Saunders, D., Jones, P., Maycock, T. K., & Ansari, S. (2016) · D-Lib Magazine · doi:10.1045/may2016-peng · Cited by 34
Using AntiPatterns to avoid MLOps MistakesPreprint
Muralidhar, N., Muthiah, S., Butler, P., Jain, M., Yu, Y., Burne, K., Li, W., Jones, D., Arunachalam, P., McCormick, H. W., et al. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2107.00079 · Cited by 34
Benchmarking Machine Learning Solutions in Production
Cardoso Silva, L., Rezende Zagatti, F., Silva Sette, B., Nildaimon dos Santos Silva, L., Lucredio, D., Furtado Silva, D., & de Medeiros Caseli, H. (2020) · 2020 19th IEEE International Conference on Machine Learning and Applications (ICMLA) · doi:10.1109/icmla51294.2020.00104 · Cited by 33
Data Stewardship on the map: A study of tasks and roles in Dutch research institutes
Verheul, I., Imming, M., Ringerma, J., Mordant, A., Ploeg, J.-L. V. D., & Pronk, M. (2019) · Socio-Environmental Systems Modeling · doi:10.5281/zenodo.2669149 · Cited by 31
Modern Web Performance Optimization: Methods, Tools, and Patterns to Speed Up Digital Platforms
Shivakumar, S. K. (2020) · Apress eBooks · doi:10.1007/978-1-4842-6528-4 · Cited by 28
Who Needs MLOps: What Data Scientists Seek to Accomplish and How Can MLOps Help?
Makinen, S., Skogstrom, H., Laaksonen, E., & Mikkonen, T. (2021) · 2021 IEEE/ACM 1st Workshop on AI Engineering - Software Engineering for AI (WAIN) · doi:10.1109/wain52551.2021.00024 · Cited by 28
From the Edge to the Cloud: Model Serving in ML.NET.
Lee, Y., Scolari, A., Chun, B.-G., Weimer, M., & Interlandi, M. (2018) · IEEE Data(base) Engineering Bulletin · Cited by 25
Data Stewardship on the map: A study of tasks and roles in Dutch research institutes
Verheul, I., Imming, M., Ringerma, J., et al. (2019) · Socio-Environmental Systems Modeling · doi:10.5281/zenodo.2669150 · Cited by 25
Fursin, G., Guillou, H., & Essayan, N. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2001.07935 · Cited by 22
The DevOps Adoption Playbook: A Guide to adopting DevOps in a multi‐speed IT Enterprise
Sharma, S. (2017) · Wiley eBooks · doi:10.1002/9781119310778 · Cited by 20
Streamlining the development of data-driven industrial applications by automated machine learning
Kißkalt, D., Mayr, A., Lutz, B., Rögele, A., & Franke, J. E. (2020) · Procedia CIRP · doi:10.1016/j.procir.2020.04.009 · Cited by 19
Reliable Fleet Analytics for Edge IoT SolutionsPreprint
Raj, E., Westerlund, M., & Espinosa-Leal, L. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2101.04414 · Cited by 19
Characterizing IoT Data and its Quality for UsePreprint
Zubair, N., Niranjan, A. K., Hebbar, K., & Simmhan, Y. L. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1906.10497 · Cited by 19
Continuous Training for Production ML in the TensorFlow Extended (TFX) Platform
Baylor, D., Haas, K., Katsiapis, K., Leong, S., Liu, R., Mewald, C., Miao, H., Polyzotis, N., Trott, M. D., & Zinkevich, M. (2019) · Cited by 17
A comparison of image quality metric performances under practical conditions
Zampolo, R. F., & Seara, R. (2005) · Proceedings - International Conference on Image Processing · doi:10.1109/icip.2005.1530611 · Cited by 16
DQLearn : A Toolkit for Structured Data Quality Learning
Shrivastava, S., Patel, D., Zhou, N., Iyengar, A., & Bhamidipaty, A. (2020) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata50022.2020.9378296 · Cited by 14
Machine Learning Model Evaluation
Ramasubramanian, K., & Singh, A. N. (2016) · Apress eBooks · doi:10.1007/978-1-4842-2334-5_7 · Cited by 13
Sebastian-Coleman, L. (2013) · Elsevier eBooks · doi:10.1016/b978-0-12-397033-6.00004-3 · Cited by 5
Intricacies of Unstructured Data
Rai, R. K. (2017) · ICST Transactions on Scalable Information Systems · doi:10.4108/eai.25-9-2017.153151 · Cited by 4
Lacagnina, C., Peng, G., Ivánová, I., Downs, R. R., Ramapriyan, H. K., Moroni, D. F., Wei, Y., Wyborn, L., Jones, D., & Ganske, A. (2021) · doi:10.1002/essoar.10508481.1 · Cited by 4
Lacagnina, C., Peng, G., Ivánová, I., Downs, R. R., Ramapriyan, H. K., Moroni, D. F., Wei, Y., Wyborn, L., Jones, D., & Ganske, A. (2021) · doi:10.1002/essoar.10508601.1 · Cited by 4
Web Performance Monitoring and Infrastructure Planning
Shivakumar, S. K. (2020) · Apress eBooks · doi:10.1007/978-1-4842-6528-4_7 · Cited by 3
Research works printed but not linked: 5
- Goyal, A. (2020). Machine Learning Operations. International Journal of Information Technology Insights & Transformations [ISSN:2581-5172 (Online)], Eureka Journals.
- Zhao, Y. (2021, July 27). Machine Learning in Production: A Literature Review. Available online: https://staff.fnwi.uva.nl/a.s.z.belloum/LiteratureStudies/Reports/2021-LiteratureStudy-report-Yizhen.pdf.
- Sureddy, 2020, A Framework for Monitoring Data Warehousing Applications, Int. Res. J. Eng. Technol., 7, 7023
- Zöller, M.A., and Huber, M.F. (2019). Survey on automated machine learning. arXiv.
- Gudivada, 2017, Data quality considerations for big data and machine learning: Going beyond data cleaning and transformations, Int. J. Adv. Softw., 10, 1
Cited by
Machine Learning Operations (MLOps): Overview, Definition, and Architecture
Kreuzberger, D., Kühl, N., & Hirschl, S. (2023) · IEEE Access · doi:10.1109/access.2023.3262138 · Cited by 766
MLOps - Definitions, Tools and Challenges
Symeonidis, G., Nerantzis, E., Kazakis, A., & Papakostas, G. A. (2022) · IEEE Annual Computing and Communication Workshop and Conference (CCWC) · doi:10.1109/ccwc54503.2022.9720902 · Cited by 180
MLOps: A Taxonomy and a Methodology
Testi, M., Ballabio, M., Frontoni, E., Iannello, G., Moccia, S., Soda, P., & Vessio, G. (2022) · IEEE Access · doi:10.1109/access.2022.3181730 · Cited by 204
Democratizing artificial intelligence: How no-code AI can leverage machine learning operations
Sundberg, L., & Holmström, J. (2023) · Business Horizons · doi:10.1016/j.bushor.2023.04.003 · Cited by 164
Shankar, S., Garcia, R., Hellerstein, J. M., & Parameswaran, A. (2024) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3653697 · Cited by 152
Metadata
- Authors
- Ruf, P., Madan, M., Reich, C., & Ould-Abdeslam, D.
- Affiliations
- Furtwangen University (DE)
- Institut de Recherche en Informatique, Mathématiques, Automatique et Signal (IRIMAS), Université de Haute-Alsace, 61 Rue Albert Camus, 68093 Mulhouse, France
- Corresponding author
- Reich, Christoph
- Venue
- Applied Sciences
- Publisher
- MDPI AG
- Published
- 2021-09-23
- Volume
- 11
- Issue
- 19
- Pages
- 8861
- ISSN
- 2076-3417 (ISSN-L)
- OpenAlex
- W3202461888
- Cited by
- 112
- References
- 50
- Citations by year
- 2026: 11 · 2025: 22 · 2024: 33 · 2023: 28 · 2022: 18
- FWCI
- 17.86
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- Applied Sciences (publishedVersion, cc-by)
- Repository of Futwangen University of Applied Sciences (Furtwangen University) (submittedVersion, cc-by)
- HAL (submittedVersion, other-oa)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
- Social Sciences › Decision Sciences › Information Systems and Management › Scientific Computing and Data Management
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Data Classification
- Keywords
- MLOps, open source tools, machine learning pipelines, tool selection, object detection, deep learning, image processing, model evaluation metrics
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.