A Multivocal Review of MLOps Practices, Challenges and Open Issues
Open accessThe DOI matches its Crossref record; 20 of the 20 references it cites were found in a registry or an index; 19 of the 334 sources its claims cite were checked for support.
Abstract
MLOps has emerged as a key solution to address many socio-technical challenges of bringing ML models to production, such as integrating ML models with non-ML software, continuous monitoring, maintenance, and retraining of deployed models. Despite the utility of MLOps, an integrated body of knowledge regarding MLOps remains elusive because of its extensive scope due to the diversity of ML productionalization challenges it addresses. Whilst the existing literature reviews provide valuable snapshots of specific practices, tools, and research prototypes related to MLOps at various times, they focus on particular facets of MLOps, thus fail to offer a comprehensive and invariant framework that can weave these perspectives into a unified understanding of MLOps. This article presents a Multivocal Literature Review that systematically analyzes a corpus of 150 peer-reviewed and 48 grey literature to synthesize a unified conceptualization of MLOps and develop a snapshot of its best practices, adoption challenges, and solutions.
Overview
CoChat Summary
TL;DR
This multivocal literature review synthesizes 150 peer-reviewed and 48 grey literature sources to build a unified conceptualization of MLOps, map its practices, and identify adoption challenges and open issues. It finds MLOps definitions remain diverse but increasingly mature, with emerging emphasis on compliance, governance, security, quality, and human-in-the-loop automation.
Key findings
- The review identifies eight MLOps definitions; the most common are “continuous ML” (34 studies), “DevOps for ML” (32 studies), and cultural/organizational practices for ML (18 studies).
- MLOps activities span project initiation, data collection/preparation/analysis, model development, deployment, monitoring, pipeline management, artifact management, quality assurance, and team management, with roles including domain experts, business analysts, data engineers, data scientists, ML engineers, MLOps engineers, DevOps engineers, software engineers, QA engineers, operational engineers, architects, security experts, compliance teams, and data governance roles.
- State-of-the-art practices are grouped into seven categories: adapting team structure/work practices, improving transparency and provenance, continuous monitoring, systemizing infrastructure/environment provisioning, systemizing model deployment, setting up automated cross-silo pipelines, and abstracting MLOps complexity.
- Adoption challenges include socio-technical issues (cultural changes, MLOps fragmentation, responsible MLOps), pipeline-related issues (complexity, optimized deployment, large-scale pipeline management, continual learning, automation decision dilemma), and platform-related issues (customization, scalability, artifact management, distributed computing environments, migration to MLOps platforms).
Methods
- A Multivocal Literature Review (MLR) following Garousi et al., combining academic and grey literature, with thematic analysis based on Braun and Clarke.
- Academic search covered IEEE Xplore, ACM Digital Library, Web of Science, and Scopus; grey literature search used Google Search and arXiv. Selection involved inclusion/exclusion criteria, quality assessment, and backward/forward snowballing.
- From 5,519 academic records reduced to 3,977 after deduplication, 279 remained after metadata-based selection, 132 after full-text/quality assessment, and 18 added via snowballing, yielding 150 peer-reviewed studies; grey literature went from 453 to 423 and finally 48 studies. The corpus covers literature up to September 2023.
Limitations
- The paper states threats to validity concern study search, study selection, data extraction, and theme analysis; it also states that excluding non-free resources is a limitation.
- The paper reports that grey literature search stopped at theoretical saturation and that gathered results roughly covered 50% of all search results, which may limit coverage.
- Evident from the text: the review’s snapshot is bounded by its September 2023 timeframe, and the authors note MLOps is rapidly evolving, so tools and practices may become outdated.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures




Verification
Claims the paper makes
18 Valid · 17 Not valid · 194 Not checked · 117 Own results and interpretations
Every source the paper cites for this claim was checked and supports it.
In the paper
“they focus on particular facets of MLOps, thus fail to offer a comprehensive and invariant framework that can weave these perspectives into a unified understanding of MLOps”
§ A Multivocal Review of MLOps Practices, Challenges and Open Issues > Abstract
What the sources say
- Supportedcites Steidl et al. [23] 23. Monika Steidl, Rudolf Ramler, and Michael Felderer. 2024. The past, present, and future of research on the continuous development of AI. In Proceedings of the 2024 50th Euromicro Conference on Software Engineering and Advanced Applications (SEAA). IEEE, 300-308. source
The Introduction (p. 1, Sec. I) states that since 2020 literature reviews 'have attempted to collect and synthesize this research from different perspectives', that they 'employ different search terms and selection criteria ... partitioning the primary sources', and that 'up to now, a meta-analysis of all existing literature reviews is not available' while 'a holistic overview needs to incorporate knowledge from the entire set of primary studies'. The Abstract and RQ2 (Sec. III-B) reinforce this: 33 reviews cover 12 different topics (pipelines 15, challenges 16, SE for AI 17, tools 11, etc.) with minimal overlap, and their collective topics have not been comprehensively analysed. The citing sentence's 'invariant' is the citing authors' own wording, but the substance — facet-focused reviews lacking a unified, comprehensive synthesis — is what our copy asserts and motivates its own holistic pipeline overview. (LLM, conf=0.85)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“Kolltveit et al. [13] reported an SLR on the operationalization of ML systems, which covers all steps following model training and evaluation.”
§ A Multivocal Review of MLOps Practices, Challenges and Open Issues > 7 Related Literature Reviews
What the sources say
- Supportedcites [13] 13. Ask Berstad Kolltveit and Jingyue Li. 2022. Operationalizing machine learning models: A systematic literature review. In Proceedings of the 1st Workshop on Software Engineering for Responsible AI. 1-8. source
The Abstract identifies the paper as a systematic literature review of ML model operationalization. It defines operationalization as encompassing all steps after model training and evaluation, including packaging, deployment integration, serving, and monitoring. (LLM, conf=0.95)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The sources with a verdict support this claim; the rest weren’t checked yet, have no full text we could read, or their verdicts aren’t rated yet.
In the paper
“Conceptualization: The conceptualization of MLOps in earlier systematic studies aligns closely with our RQ1.1 results (e.g., [14], [17]).”
§ A Multivocal Review of MLOps Practices, Challenges and Open Issues > 7.1 Evolution of MLOps
What the sources say
- Supportedcites [14] 14. Dominik Kreuzberger, Niklas Kühl, and Sebastian Hirschl. 2023. Machine learning operations (mlops): Overview, definition, and architecture. IEEE Access 11 (2023), 31866-31879. source
The Methodology section describes a structured literature review, and the Conceptualization section provides a definition of MLOps synthesized from the study’s findings. This supports citing the paper as an earlier study that conceptualized MLOps; the comparison with the citing authors’ RQ1.1 results is their own assessment. (LLM, conf=0.88)
- Not checked — no full text of the sourcecites [17] 17. Tsakani Mboweni, Themba Masombuka, and Cyrille Dongmo. 2022. A systematic review of machine learning devops. In Proceedings of the 2022 International Conference on Electrical, Computer and Energy Technologies (ICECET). IEEE, 1-6. source
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“32 refer to it as”
§ A Multivocal Review of MLOps Practices, Challenges and Open Issues > 3 RQ1: Conceptualization of MLOps
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“34 studies define MLOps as”
§ A Multivocal Review of MLOps Practices, Challenges and Open Issues > 3 RQ1: Conceptualization of MLOps
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
63 more claims
Create a free account to see all 346 claims and to verify the 8 references still open.
- Probably not a substantial claim. 235
- It may state two separate claims as one. 103
- The quote only partly states this claim. 1
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“During the metadata-based selection, we followed the inclusion and exclusion criteria listed in our online appendix [7]”
From A Multivocal Review of MLOps Practices, Challenges and Open Issues (2026) · cites this paper as [7]
The cited source says what the claim attributes to it.
In the citing paper
“We used a standardized template to extract key study details (e.g., objectives, methodologies, findings) [7].”
From A Multivocal Review of MLOps Practices, Challenges and Open Issues (2026) · cites this paper as [7]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ A Multivocal Review of MLOps Practices, Challenges and Open Issues > 8 Conclusion
“Software Engineers (SE)”
§ A Multivocal Review of MLOps Practices, Challenges and Open Issues > 8 Conclusion
“Security Experts (SE)”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ A Multivocal Review of MLOps Practices, Challenges and Open Issues > 4.1 Adapting Team Structure and Work Practices for MLOps
“Abstracting MLOps Complexity (Section 5.7)”
§ A Multivocal Review of MLOps Practices, Challenges and Open Issues > 4.7 Abstracting MLOps Complexity
“## 4.7 Abstracting MLOps Complexity”
Citations
Cites
Using thematic analysis in psychology
Braun, V., & Clarke, V. (2006) · Qualitative Research in Psychology · doi:10.1191/1478088706qp063oa · Cited by 194,414
Wohlin, C. (2014) · Proceedings of the 18th International Conference on Evaluation and Assessment in Software Engineering · doi:10.1145/2601248.2601268 · Cited by 4,234 · cited 2×
“As the final step in academic literature selection, we applied forward and backward snowballing based on the guidelines by Wohlin et al. [26].”
Garousi, V., Felderer, M., & Mäntylä, M. V. (2019) · Information and Software Technology · doi:10.1016/j.infsof.2018.09.006 · Cited by 831
Machine Learning Operations (MLOps): Overview, Definition, and Architecture
Kreuzberger, D., Kühl, N., & Hirschl, S. (2023) · IEEE Access · doi:10.1109/access.2023.3262138 · Cited by 766 · cited 3×
“This results in accidental complexities, and it is challenging to separate accidental complexity from essential complexity (PS154).”
InfluentialTFX: A TensorFlow-Based Production-Scale Machine Learning Platform
Baylor, D., Breck, E., Cheng, H.-T., Fiedel, N., Foo, C. Y., Haque, Z., Haykal, S., Ispir, M., Jain, V., Koc, L., et al. (2017) · Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/3097983.3098021 · Cited by 350
Dissanayake, N., Jayatilaka, A., Zahedi, M., & Babar, M. A. (2021) · Information and Software Technology · doi:10.1016/j.infsof.2021.106771 · Cited by 190 · cited 1×
“Thematic analysis is a qualitative research methodology widely used in systematic literature reviews ( SLRs ) to understand patterns within the extracted data by identifying, analyzing, and reporting themes [4, 6].”
Self-citationTowards MLOps: A Framework and Maturity Model
John, M. M., Olsson, H. H., & Bosch, J. (2021) · 2021 47th Euromicro Conference on Software Engineering and Advanced Applications (SEAA) · doi:10.1109/seaa53835.2021.00050 · Cited by 130 · cited 1×
“John et al. [11] delivered an MLR on the adoption of MLOps in practice.”
Steidl, M., Felderer, M., & Ramler, R. (2023) · Journal of Systems and Software · doi:10.1016/j.jss.2023.111615 · Cited by 126
Data Preparation for Software Vulnerability Prediction: A Systematic Literature Review
Croft, R., Xie, Y., & Babar, M. A. (2022) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2022.3171202 · Cited by 96
Self-citationMachine Learning Operations (MLOps): Overview, Definition, and Architecture
Kreuzberger, D., Kühl, N., & Hirschl, S. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2205.02302 · Cited by 51
Research works printed but not linked: 4
- 1. Kent Beck, Mike Beedle, Arie Van Bennekum, Alistair Cockburn, Ward Cunningham, Martin Fowler, James Grenning, Jim Highsmith, Andrew Hunt, Ron Jeffries, et al. 2001. The agile manifesto. Software Development 9, 8 (2001), 28-35. http://agilemanifesto.org
- 8. Leonhard Faubel and Klaus Schmid. 2023. Review protocol: A systematic literature review of MLOps. (2023).
- 21. David Sculley, Gary Holt, Daniel Golovin, Eugene Davydov, Todd Phillips, Dietmar Ebner, Vinay Chaudhary, Michael Young, Jean-Francois Crespo, and Dan Dennison. 2015. Hidden technical debt in machine learning systems. In Proceedings of the 29th International Conference on Neural Information Processing Systems 2 (2015), 2503-2511.
- 25. Hironori Washizaki. 2024. Guide to the software engineering body of knowledge (SWEBOK guide), version 4.0. IEEE Computer Society, Waseda University, Japan. http://www. swebok. org IEEE Computer Society .
Cited by
Artificial intelligence in drug discovery — what it is, where we stand and the path forward
Bender, A., Thomas, M. C., Scannell, J. W., Shaywitz, D. A., Ghiandoni, G. M., Greener, J. G., Pruteanu, L.-L., Jacobson, R. D., Handa, K., Hirano, M., et al. (2026) · Nature Reviews Drug Discovery · doi:10.1038/s41573-026-01496-2 · Cited by 44
Cheimarios, N. (2025) · Frontiers in Physics · doi:10.3389/fphy.2025.1711356 · Cited by 7
“Academic-grade workflows tend to benefit most from lightweight components: version-controlled experiment tracking, containerized execution environments, and periodic—not continuous—retraining schedules [255, 256].”
A Platform for Machine Learning Operations for Network Constrained Far-Edge Devices
McCormack, C., & Mitiche, I. (2025) · Applied System Innovation · doi:10.3390/asi8050141 · Cited by 1
“…to the Edge but rather prioritises the idea of moving models from Cloud to Edge without consideration of uptime reliability or network circumstance, nor do they include monitoring or condition management for models built for individual devices, rather focusing on models that can be re-used [10].”
Reviewing the Implementation Barriers of AI-Driven Data Governance Frameworks in Modern Enterprises
Kukoyi, A. H. (2024) · International Journal of Scientific Research and Modern Technology. · doi:10.38124/ijsrmt.v3i11.1036 · Cited by 1
Nagaraj, P. B., Pulakhandam, W., Chaluvadi, A., et al. (2025) · International Conference on Image Information Processing (ICIIP) · doi:10.1109/iciip68302.2025.11346334 · Cited by 1
“In AVs, object detection plays a vital role in the perception task, and its limited computational capacity and uncertainty in data make it difficult to delineate research to practice [1], [2], [3].”
Metadata
- Authors
- Eken, B., Pallewatta, S., Tran, N., Tosun, A., & Babar, M. A.
- Affiliations
- Sakarya University (TR)
- The University of Adelaide (AU)
- Istanbul Technical University
- Venue
- ACM Computing Surveys
- Published
- 2025-09-08
- Volume
- 58
- Issue
- 2
- Pages
- 1-35
- ISSN
- 0360-0300 (ISSN-L)
- 1557-7341
- OpenAlex
- W4414080965
- Cited by
- 33
- References
- 18
- Citations by year
- 2026: 25 · 2025: 7
- FWCI
- 32.98
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- ACM Computing Surveys (publishedVersion, cc-by)
- Istanbul Technical University Academic Open Archive (Istanbul Technical University) (acceptedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
- Physical Sciences › Computer Science › Computer Networks and Communications › Software System Performance and Reliability
- Social Sciences › Decision Sciences › Information Systems and Management › Scientific Computing and Data Management
- Keywords
- MLOps, model deployment, model monitoring, model maintenance, model retraining
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (qwen3.8-omni-flash and gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.