CoChat

A Multivocal Review of MLOps Practices, Challenges and Open Issues

Open access
Eken, Beyza, Pallewatta, Samodha, Tran, Nguyen, Tosun, Ayse, Babar, Muhammad Ali · 2026 · ACM Computing Surveys

The DOI matches its Crossref record; 19 of the 19 references it cites were found in a registry or an index; 19 of the 334 sources its claims cite were checked for support.

Abstract

MLOps has emerged as a key solution to address many socio-technical challenges of bringing ML models to production, such as integrating ML models with non-ML software, continuous monitoring, maintenance, and retraining of deployed models. Despite the utility of MLOps, an integrated body of knowledge regarding MLOps remains elusive because of its extensive scope due to the diversity of ML productionalization challenges it addresses. Whilst the existing literature reviews provide valuable snapshots of specific practices, tools, and research prototypes related to MLOps at various times, they focus on particular facets of MLOps, thus fail to offer a comprehensive and invariant framework that can weave these perspectives into a unified understanding of MLOps. This article presents a Multivocal Literature Review that systematically analyzes a corpus of 150 peer-reviewed and 48 grey literature to synthesize a unified conceptualization of MLOps and develop a snapshot of its best practices, adoption challenges, and solutions.

Overview

CoChat Summary

TL;DR

This multivocal literature review synthesizes 150 peer-reviewed and 48 grey literature sources to build a unified conceptualization of MLOps, map its practices, and identify adoption challenges and open issues. It finds MLOps definitions remain diverse but increasingly mature, with emerging emphasis on compliance, governance, security, quality, and human-in-the-loop automation.

Key findings

  • The review identifies eight MLOps definitions; the most common are “continuous ML” (34 studies), “DevOps for ML” (32 studies), and cultural/organizational practices for ML (18 studies).
  • MLOps activities span project initiation, data collection/preparation/analysis, model development, deployment, monitoring, pipeline management, artifact management, quality assurance, and team management, with roles including domain experts, business analysts, data engineers, data scientists, ML engineers, MLOps engineers, DevOps engineers, software engineers, QA engineers, operational engineers, architects, security experts, compliance teams, and data governance roles.
  • State-of-the-art practices are grouped into seven categories: adapting team structure/work practices, improving transparency and provenance, continuous monitoring, systemizing infrastructure/environment provisioning, systemizing model deployment, setting up automated cross-silo pipelines, and abstracting MLOps complexity.
  • Adoption challenges include socio-technical issues (cultural changes, MLOps fragmentation, responsible MLOps), pipeline-related issues (complexity, optimized deployment, large-scale pipeline management, continual learning, automation decision dilemma), and platform-related issues (customization, scalability, artifact management, distributed computing environments, migration to MLOps platforms).

Methods

  • A Multivocal Literature Review (MLR) following Garousi et al., combining academic and grey literature, with thematic analysis based on Braun and Clarke.
  • Academic search covered IEEE Xplore, ACM Digital Library, Web of Science, and Scopus; grey literature search used Google Search and arXiv. Selection involved inclusion/exclusion criteria, quality assessment, and backward/forward snowballing.
  • From 5,519 academic records reduced to 3,977 after deduplication, 279 remained after metadata-based selection, 132 after full-text/quality assessment, and 18 added via snowballing, yielding 150 peer-reviewed studies; grey literature went from 453 to 423 and finally 48 studies. The corpus covers literature up to September 2023.

Limitations

  • The paper states threats to validity concern study search, study selection, data extraction, and theme analysis; it also states that excluding non-free resources is a limitation.
  • The paper reports that grey literature search stopped at theoretical saturation and that gathered results roughly covered 50% of all search results, which may limit coverage.
  • Evident from the text: the review’s snapshot is bounded by its September 2023 timeframe, and the authors note MLOps is rapidly evolving, so tools and practices may become outdated.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1145/3747346

Cites

19 of 19 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Figures

Fig. 1 Research methodology.
Fig. 1 Research methodology.CC-BY · From the PDF
Fig. 2 MLOps definitions and the ML productionalization problems they address.
Fig. 2 MLOps definitions and the ML productionalization problems they address.CC-BY · From the PDF
Fig. 3 A pipeline for ML development and operations, and associated roles: Domain Experts (DME); Business Analysts (BA); Data Engineers (DE); Data Scientists (DS); ML Engineers (MLE); MLOps Engineers (MLOE); DevOps Engineers (DOE); Software Engineers (SE); Quality Assurance Engineers (QAE); Operational Engineers (OPE); Technical and Solution Architects (ARC); Security Experts (SE); Business Owners/Managers (BM); Product Owners/Managers/Experts (PM); Data Governance Officer (DGO), Data Quality Board (DQB), Data Steward (DST)
Fig. 3 A pipeline for ML development and operations, and associated roles: Domain Experts (DME); Business Analysts (BA); Data Engineers (DE); Data Scientists (DS); ML Engineers (MLE); MLOps Engineers (MLOE); DevOps Engineers (DOE); Software Engineers (SE); Quality Assurance Engineers (QAE); Operational Engineers (OPE); Technical and Solution Architects (ARC); Security Experts (SE); Business Owners/Managers (BM); Product Owners/Managers/Experts (PM); Data Governance Officer (DGO), Data Quality Board (DQB), Data Steward (DST)CC-BY · From the PDF
Fig. 4 MLOps practices and techniques, and the corresponding ML productionalization challenges.
Fig. 4 MLOps practices and techniques, and the corresponding ML productionalization challenges.CC-BY · From the PDF

Verification

Claims the paper makes

18 Valid · 17 Not valid · 194 Not checked · 117 Own results and interpretations

63 more claims

Create a free account to see all 346 claims and to verify the 8 references still open.

See all claims — free account
  • Probably not a substantial claim. 235
  • It may state two separate claims as one. 103
  • The quote only partly states this claim. 1

Create a free account to see these claims.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

2 of 2

Citations

Cites

Show all 19 references

Research works printed but not linked: 3

  1. Leonhard Faubel and Klaus Schmid. 2023. Review protocol: A systematic literature review of MLOps. (2023).
  2. David Sculley Gary Holt Daniel Golovin Eugene Davydov Todd Phillips Dietmar Ebner Vinay Chaudhary Michael Young Jean-Francois Crespo and Dan Dennison. 2015. Hidden technical debt in machine learning systems. In Proceedings of the 29th International Conference on Neural Information Processing Systems 2 (2015) 2503–2511.
  3. Hironori Washizaki. 2024. Guide to the software engineering body of knowledge (SWEBOK guide), version 4.0. IEEE Computer Society, Waseda University, Japan. http://www. swebok. org IEEE Computer Society.

Cited by

Metadata

Authors
Eken, B., Pallewatta, S., Tran, N., Tosun, A., & Babar, M. A.
Affiliations
  • Sakarya University (TR)
  • The University of Adelaide (AU)
  • Istanbul Technical University
Venue
ACM Computing Surveys
Published
2025-09-08
Volume
58
Issue
2
Pages
1-35
ISSN
  • 0360-0300 (ISSN-L)
  • 1557-7341
OpenAlex
W4414080965
Cited by
33
References
18
Citations by year
2026: 25 · 2025: 7
FWCI
32.98
Type
journal article
Language
en
Open access
Licence
cc-by
Locations
  • ACM Computing Surveys (publishedVersion, cc-by)
  • Istanbul Technical University Academic Open Archive (Istanbul Technical University) (acceptedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
  • Physical Sciences › Computer Science › Computer Networks and Communications › Software System Performance and Reliability
  • Social Sciences › Decision Sciences › Information Systems and Management › Scientific Computing and Data Management
Keywords
MLOps, model deployment, model monitoring, model maintenance, model retraining

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (qwen3.8-omni-flash and gpt-6-sol).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.