A Multivocal Review of MLOps Practices, Challenges and Open Issues
Open accessThe DOI matches its Crossref record; 19 of the 19 references it cites were found in a registry or an index; 19 of the 334 sources its claims cite were checked for support.
Abstract
MLOps has emerged as a key solution to address many socio-technical challenges of bringing ML models to production, such as integrating ML models with non-ML software, continuous monitoring, maintenance, and retraining of deployed models. Despite the utility of MLOps, an integrated body of knowledge regarding MLOps remains elusive because of its extensive scope due to the diversity of ML productionalization challenges it addresses. Whilst the existing literature reviews provide valuable snapshots of specific practices, tools, and research prototypes related to MLOps at various times, they focus on particular facets of MLOps, thus fail to offer a comprehensive and invariant framework that can weave these perspectives into a unified understanding of MLOps. This article presents a Multivocal Literature Review that systematically analyzes a corpus of 150 peer-reviewed and 48 grey literature to synthesize a unified conceptualization of MLOps and develop a snapshot of its best practices, adoption challenges, and solutions.
Overview
CoChat Summary
TL;DR
This multivocal literature review synthesizes 150 peer-reviewed and 48 grey literature sources to build a unified conceptualization of MLOps, map its practices, and identify adoption challenges and open issues. It finds MLOps definitions remain diverse but increasingly mature, with emerging emphasis on compliance, governance, security, quality, and human-in-the-loop automation.
Key findings
- The review identifies eight MLOps definitions; the most common are “continuous ML” (34 studies), “DevOps for ML” (32 studies), and cultural/organizational practices for ML (18 studies).
- MLOps activities span project initiation, data collection/preparation/analysis, model development, deployment, monitoring, pipeline management, artifact management, quality assurance, and team management, with roles including domain experts, business analysts, data engineers, data scientists, ML engineers, MLOps engineers, DevOps engineers, software engineers, QA engineers, operational engineers, architects, security experts, compliance teams, and data governance roles.
- State-of-the-art practices are grouped into seven categories: adapting team structure/work practices, improving transparency and provenance, continuous monitoring, systemizing infrastructure/environment provisioning, systemizing model deployment, setting up automated cross-silo pipelines, and abstracting MLOps complexity.
- Adoption challenges include socio-technical issues (cultural changes, MLOps fragmentation, responsible MLOps), pipeline-related issues (complexity, optimized deployment, large-scale pipeline management, continual learning, automation decision dilemma), and platform-related issues (customization, scalability, artifact management, distributed computing environments, migration to MLOps platforms).
Methods
- A Multivocal Literature Review (MLR) following Garousi et al., combining academic and grey literature, with thematic analysis based on Braun and Clarke.
- Academic search covered IEEE Xplore, ACM Digital Library, Web of Science, and Scopus; grey literature search used Google Search and arXiv. Selection involved inclusion/exclusion criteria, quality assessment, and backward/forward snowballing.
- From 5,519 academic records reduced to 3,977 after deduplication, 279 remained after metadata-based selection, 132 after full-text/quality assessment, and 18 added via snowballing, yielding 150 peer-reviewed studies; grey literature went from 453 to 423 and finally 48 studies. The corpus covers literature up to September 2023.
Limitations
- The paper states threats to validity concern study search, study selection, data extraction, and theme analysis; it also states that excluding non-free resources is a limitation.
- The paper reports that grey literature search stopped at theoretical saturation and that gathered results roughly covered 50% of all search results, which may limit coverage.
- Evident from the text: the review’s snapshot is bounded by its September 2023 timeframe, and the authors note MLOps is rapidly evolving, so tools and practices may become outdated.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures




Verification
Claims the paper makes
18 Valid · 17 Not valid · 194 Not checked · 117 Own results and interpretations
Every source the paper cites for this claim was checked and supports it.
In the paper
“they focus on particular facets of MLOps, thus fail to offer a comprehensive and invariant framework that can weave these perspectives into a unified understanding of MLOps”
§ A Multivocal Review of MLOps Practices, Challenges and Open Issues > Abstract
What the sources say
- Supportedcites Steidl et al. [23] 23. Monika Steidl, Rudolf Ramler, and Michael Felderer. 2024. The past, present, and future of research on the continuous development of AI. In Proceedings of the 2024 50th Euromicro Conference on Software Engineering and Advanced Applications (SEAA). IEEE, 300-308. source
The Introduction (p. 1, Sec. I) states that since 2020 literature reviews 'have attempted to collect and synthesize this research from different perspectives', that they 'employ different search terms and selection criteria ... partitioning the primary sources', and that 'up to now, a meta-analysis of all existing literature reviews is not available' while 'a holistic overview needs to incorporate knowledge from the entire set of primary studies'. The Abstract and RQ2 (Sec. III-B) reinforce this: 33 reviews cover 12 different topics (pipelines 15, challenges 16, SE for AI 17, tools 11, etc.) with minimal overlap, and their collective topics have not been comprehensively analysed. The citing sentence's 'invariant' is the citing authors' own wording, but the substance — facet-focused reviews lacking a unified, comprehensive synthesis — is what our copy asserts and motivates its own holistic pipeline overview. (LLM, conf=0.85)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“Kolltveit et al. [13] reported an SLR on the operationalization of ML systems, which covers all steps following model training and evaluation.”
§ A Multivocal Review of MLOps Practices, Challenges and Open Issues > 7 Related Literature Reviews
What the sources say
- Supportedcites [13] 13. Ask Berstad Kolltveit and Jingyue Li. 2022. Operationalizing machine learning models: A systematic literature review. In Proceedings of the 1st Workshop on Software Engineering for Responsible AI. 1-8. source
The Abstract identifies the paper as a systematic literature review of ML model operationalization. It defines operationalization as encompassing all steps after model training and evaluation, including packaging, deployment integration, serving, and monitoring. (LLM, conf=0.95)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The sources with a verdict support this claim; the rest weren’t checked yet, have no full text we could read, or their verdicts aren’t rated yet.
In the paper
“Conceptualization: The conceptualization of MLOps in earlier systematic studies aligns closely with our RQ1.1 results (e.g., [14], [17]).”
§ A Multivocal Review of MLOps Practices, Challenges and Open Issues > 7.1 Evolution of MLOps
What the sources say
- Supportedcites [14] 14. Dominik Kreuzberger, Niklas Kühl, and Sebastian Hirschl. 2023. Machine learning operations (mlops): Overview, definition, and architecture. IEEE Access 11 (2023), 31866-31879. source
The Methodology section describes a structured literature review, and the Conceptualization section provides a definition of MLOps synthesized from the study’s findings. This supports citing the paper as an earlier study that conceptualized MLOps; the comparison with the citing authors’ RQ1.1 results is their own assessment. (LLM, conf=0.88)
- Not checked — no full text of the sourcecites [17] 17. Tsakani Mboweni, Themba Masombuka, and Cyrille Dongmo. 2022. A systematic review of machine learning devops. In Proceedings of the 2022 International Conference on Electrical, Computer and Energy Technologies (ICECET). IEEE, 1-6. source
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“32 refer to it as”
§ A Multivocal Review of MLOps Practices, Challenges and Open Issues > 3 RQ1: Conceptualization of MLOps
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“34 studies define MLOps as”
§ A Multivocal Review of MLOps Practices, Challenges and Open Issues > 3 RQ1: Conceptualization of MLOps
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
63 more claims
Create a free account to see all 346 claims and to verify the 8 references still open.
- Probably not a substantial claim. 235
- It may state two separate claims as one. 103
- The quote only partly states this claim. 1
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“During the metadata-based selection, we followed the inclusion and exclusion criteria listed in our online appendix [7]”
From A Multivocal Review of MLOps Practices, Challenges and Open Issues (2026) · cites this paper as [7]
The cited source says what the claim attributes to it.
In the citing paper
“We used a standardized template to extract key study details (e.g., objectives, methodologies, findings) [7].”
From A Multivocal Review of MLOps Practices, Challenges and Open Issues (2026) · cites this paper as [7]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ A Multivocal Review of MLOps Practices, Challenges and Open Issues > 8 Conclusion
“Software Engineers (SE)”
§ A Multivocal Review of MLOps Practices, Challenges and Open Issues > 8 Conclusion
“Security Experts (SE)”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ A Multivocal Review of MLOps Practices, Challenges and Open Issues > 4.1 Adapting Team Structure and Work Practices for MLOps
“Abstracting MLOps Complexity (Section 5.7)”
§ A Multivocal Review of MLOps Practices, Challenges and Open Issues > 4.7 Abstracting MLOps Complexity
“## 4.7 Abstracting MLOps Complexity”
Citations
Cites
Using thematic analysis in psychology
Braun, V., & Clarke, V. (2006) · Qualitative Research in Psychology · doi:10.1191/1478088706qp063oa · Cited by 194,414
Wohlin, C. (2014) · Proceedings of the 18th International Conference on Evaluation and Assessment in Software Engineering · doi:10.1145/2601248.2601268 · Cited by 4,234 · cited 2×
“As the final step in academic literature selection, we applied forward and backward snowballing based on the guidelines by Wohlin et al. [26].”
Garousi, V., Felderer, M., & Mäntylä, M. V. (2019) · Information and Software Technology · doi:10.1016/j.infsof.2018.09.006 · Cited by 831
Machine Learning Operations (MLOps): Overview, Definition, and Architecture
Kreuzberger, D., Kühl, N., & Hirschl, S. (2023) · IEEE Access · doi:10.1109/access.2023.3262138 · Cited by 766 · cited 3×
“This results in accidental complexities, and it is challenging to separate accidental complexity from essential complexity (PS154).”
InfluentialDissanayake, N., Jayatilaka, A., Zahedi, M., & Babar, M. A. (2021) · Information and Software Technology · doi:10.1016/j.infsof.2021.106771 · Cited by 155 · cited 1×
“Thematic analysis is a qualitative research methodology widely used in systematic literature reviews ( SLRs ) to understand patterns within the extracted data by identifying, analyzing, and reporting themes [4, 6].”
Self-citationTowards MLOps: A Framework and Maturity Model
John, M. M., Olsson, H. H., & Bosch, J. (2021) · 2021 47th Euromicro Conference on Software Engineering and Advanced Applications (SEAA) · doi:10.1109/seaa53835.2021.00050 · Cited by 130 · cited 1×
“John et al. [11] delivered an MLR on the adoption of MLOps in practice.”
Steidl, M., Felderer, M., & Ramler, R. (2023) · Journal of Systems and Software · doi:10.1016/j.jss.2023.111615 · Cited by 126
Data Preparation for Software Vulnerability Prediction: A Systematic Literature Review
Croft, R., Xie, Y., & Babar, M. A. (2022) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2022.3171202 · Cited by 96
Self-citationMachine Learning Operations (MLOps): Overview, Definition, and Architecture
Kreuzberger, D., Kühl, N., & Hirschl, S. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2205.02302 · Cited by 51
A Systematic Review of Machine Learning DevOps
Mboweni, T., Masombuka, T., & Dongmo, C. (2022) · 2022 International Conference on Electrical, Computer and Energy Technologies (ICECET) · doi:10.1109/icecet55527.2022.9872968 · Cited by 34 · cited 3×
“1 results (e.g., [14], [17]).”
Research works printed but not linked: 3
- Leonhard Faubel and Klaus Schmid. 2023. Review protocol: A systematic literature review of MLOps. (2023).
- David Sculley Gary Holt Daniel Golovin Eugene Davydov Todd Phillips Dietmar Ebner Vinay Chaudhary Michael Young Jean-Francois Crespo and Dan Dennison. 2015. Hidden technical debt in machine learning systems. In Proceedings of the 29th International Conference on Neural Information Processing Systems 2 (2015) 2503–2511.
- Hironori Washizaki. 2024. Guide to the software engineering body of knowledge (SWEBOK guide), version 4.0. IEEE Computer Society, Waseda University, Japan. http://www. swebok. org IEEE Computer Society.
Cited by
Artificial intelligence in drug discovery — what it is, where we stand and the path forward
Bender, A., Thomas, M. C., Scannell, J. W., Shaywitz, D. A., Ghiandoni, G. M., Greener, J. G., Pruteanu, L.-L., Jacobson, R. D., Handa, K., Hirano, M., et al. (2026) · Nature Reviews Drug Discovery · doi:10.1038/s41573-026-01496-2 · Cited by 41
Cheimarios, N. (2025) · Frontiers in Physics · doi:10.3389/fphy.2025.1711356 · Cited by 7
“Academic-grade workflows tend to benefit most from lightweight components: version-controlled experiment tracking, containerized execution environments, and periodic—not continuous—retraining schedules [255, 256].”
A Platform for Machine Learning Operations for Network Constrained Far-Edge Devices
McCormack, C., & Mitiche, I. (2025) · Applied System Innovation · doi:10.3390/asi8050141 · Cited by 1
“…to the Edge but rather prioritises the idea of moving models from Cloud to Edge without consideration of uptime reliability or network circumstance, nor do they include monitoring or condition management for models built for individual devices, rather focusing on models that can be re-used [10].”
Reviewing the Implementation Barriers of AI-Driven Data Governance Frameworks in Modern Enterprises
Kukoyi, A. H. (2024) · International Journal of Scientific Research and Modern Technology. · doi:10.38124/ijsrmt.v3i11.1036 · Cited by 1
Nagaraj, P. B., Pulakhandam, W., Chaluvadi, A., et al. (2025) · International Conference on Image Information Processing (ICIIP) · doi:10.1109/iciip68302.2025.11346334 · Cited by 1
“In AVs, object detection plays a vital role in the perception task, and its limited computational capacity and uncertainty in data make it difficult to delineate research to practice [1], [2], [3].”
Leveraging cloud-native infrastructure for dynamic and flexible quantum-classical MLOps
Impedovo, A., Losavio, V. N., & Loglisci, C. A. A. M. D. (2026) · Quantum Machine Intelligence · doi:10.1007/s42484-026-00360-1 · Cited by 1
van Beem, E. H. (2026) · International Review of Law Computers & Technology · doi:10.1080/13600869.2026.2668320 · Cited by 1
Бондаренко, Д. В. (2026) · Regional and Branch Economy · doi:10.47576/2949-1916.2026.4.4.021 · Cited by 1
Architecting Governable Artifact Evolution in AI-Enabled Production Systems: A Model-Driven DSLPreprint
Caballero, M., Hernández, J.-J., Gómez, J. É., Galvez, J. J. H., Andrés, O. R., & Santana, J. C. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.6140218
Software Automated Model Lifecycle Management
Mishra, G., Mamodiya, U., & Avasthi, A. (2026) · doi:10.1109/icasst68917.2026.11484351
A Practical Multi-Phase Framework for Implementing MLOps in Enterprise Environments: A Review
Alblwi, A. (2026) · International Conference on AI Innovations and Industry (ICAIII) · doi:10.1109/icaiii69475.2026.11521210
“Data scientists do experiments and create models; ML engineers take care of feature engineering and model training (mostly); yet, MLOps engineers are responsible for pipeline automation, infrastructure management, model deployment, and monitoring; they connect data science and operations [6], [18].”
InfluentialYaghoubi, E., Sianaki, O. A., Ghasemi, F., Kamali, Z., Rajabipour, A., & Sandanayake, M. S. (2026) · Lecture notes on data engineering and communications technologies · doi:10.1007/978-3-032-23341-7_31
Ayouni, S., Khan, R. A., Alwageed, H. S., Keshta, I. M., Almagrabi, A. O., & Alzahrani, M. (2026) · Egyptian Informatics Journal · doi:10.1016/j.eij.2026.100991
Asad, M., & Alhasawi, Y. (2026) · Cluster Computing · doi:10.1007/s10586-026-06201-x · Cited by 1
Ahmed, A. Y., Ali, H. J., Alkawari, J. M., Alajmi, R. N. H., Al-Ammari, J., & Albalooshi, F. A. (2026) · International Conference on Frontiers of Engineering and Emerging Technologies (FET) · doi:10.1109/fet68771.2026.11601648
“Another important trend is the call for cross-platform standardization, driven by the increasing reliance on heterogeneous MLOps tools that require consistent interfaces, shared metadata stores, and integrated monitoring capabilities [5], [13], [23].”
InfluentialA Road Map for Tamed Interactions with Large Language Models
Scotti, V., Keim, J., Hey, T., Metzger, A., Koziolek, A., & Mirandola, R. (2026) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3832776 · Cited by 2
From Data to Decisions: Precision Agriculture and Predictive Analytics for Field-Ready AI Adoption
Randiaml, D., Tharupath, K. M. C., & Wickramasinghe, W. M. R. H. (2026) · Advances in computational intelligence and robotics book series · doi:10.4018/979-8-3373-9295-0.ch008
Brisc, D., Korodi, A., & Nicolae, A. (2026) · IEEE Access · doi:10.1109/access.2026.3724866
“The operational deployment of machine learning models requires system-level infrastructure capable of supporting data ingestion, model training, inference, and lifecycle management [26].”
Artificial intelligence integration in livestock metabolomics
Hossein‐Zadeh, N. G. (2026) · Elsevier eBooks · doi:10.1016/b978-0-443-45663-3.00002-6
A Systematic Assessment of Maturity Models for AI/ML-Based Software Development
Okay, E., & Özcan‐Top, Ö. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-36587-3_18
AI-Driven Visual Analytics: Designing Intelligent and Explainable Decision Systems
Acevedo, M. A. A., Bolivar, J. J., & Yosifov, N. M. (2026) · Advances in computational intelligence and robotics book series · doi:10.4018/979-8-2600-3930-4.ch004
KFP-Inspector: Reproducible Benchmarking and Bottleneck Diagnosis for Kubeflow Pipelines
Li, X., & Deng, C. (2026) · International Conference on Cyber Security, Artificial Intelligence and Digital Economy (CSAIDE) · doi:10.1145/3819836.3819910
A Unified Metadata Driven Platform for Data and ML Pipeline Orchestration
Pamula, V. K. (2026) · doi:10.1109/tic68483.2026.11703026
“However, the automated live documentation through Swag-ger UI in FastAPI is another additional advantage that comes along with it [4].”
Incremental Deep Learning Architectures for Adaptive Malware Detection in Dynamic Data Environments
Pomsathit, A., & Keedumrongkool, T. (2026) · International Electrical Engineering Congress (iEECON) · doi:10.1109/ieecon68065.2026.11707131
“An emphasis on reproducibility, data governance and efficient resource utilization will ensure a sustainable continual learning deployment [19].”
The ethical half-life of AI: moral decay outpaces technical obsolescence
Chao, L., & Ma, X. (2026) · AI and Ethics · doi:10.1007/s43681-026-01428-1
Najafabadi, F. A., Haug, M., Rajenthiram, K., Bogner, J., & Gerostathopoulos, I. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-39236-7_6
Metadata
- Authors
- Eken, B., Pallewatta, S., Tran, N., Tosun, A., & Babar, M. A.
- Affiliations
- Sakarya University (TR)
- The University of Adelaide (AU)
- Istanbul Technical University
- Venue
- ACM Computing Surveys
- Published
- 2025-09-08
- Volume
- 58
- Issue
- 2
- Pages
- 1-35
- ISSN
- 0360-0300 (ISSN-L)
- 1557-7341
- OpenAlex
- W4414080965
- Cited by
- 33
- References
- 18
- Citations by year
- 2026: 25 · 2025: 7
- FWCI
- 32.98
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- ACM Computing Surveys (publishedVersion, cc-by)
- Istanbul Technical University Academic Open Archive (Istanbul Technical University) (acceptedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
- Physical Sciences › Computer Science › Computer Networks and Communications › Software System Performance and Reliability
- Social Sciences › Decision Sciences › Information Systems and Management › Scientific Computing and Data Management
- Keywords
- MLOps, model deployment, model monitoring, model maintenance, model retraining
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (qwen3.8-omni-flash and gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.