CoChat
Sign Up
Cited by 766Software Engineering Techniques and PracticesSoftware System Performance and ReliabilityScientific Computing and Data Management

Machine Learning Operations (MLOps): Overview, Definition, and Architecture

Open access
Kreuzberger, Dominik, Kühl, Niklas, Hirschl, Sebastian·2023·IEEE Access
DOI 10.1109/access.2023.3262138
Open this paper in CoChat — free accountOpen PDF

Abstract

The final goal of all industrial machine learning (ML) projects is to develop ML products and rapidly bring them into production. However, it is highly challenging to automate and operationalize ML products and thus many ML endeavors fail to deliver on their expectations. The paradigm of Machine Learning Operations (MLOps) addresses this issue. MLOps includes several aspects, such as best practices, sets of concepts, and development culture. However, MLOps is still a vague term and its consequences for researchers and professionals are ambiguous. To address this gap, we conduct mixed-method research, including a literature review, a tool review, and expert interviews. As a result of these investigations, we contribute to the body of knowledge by providing an aggregated overview of the necessary principles, components, and roles, as well as the associated architecture and workflows. Furthermore, we provide a comprehensive definition of MLOps and highlight open challenges in the field.

Integrity signals

Registry

Crossref · 10.1109/access.2023.3262138

Cites

63 of 63 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

8 of 37 cited sources checked · 1 supported · 0 flagged · accuracy not yet measured

  • Findingcitation

    In productive settings, it is challenging to operate ML manually due to different stacks of software and hardware components and their interplay as well as the selection of both.

    “In productive settings, it is challenging to operate ML manually due to different stacks of software and hardware components and their interplay as well as the selection of both ( [64], [65].”
    Quote in the paperSubstantial claim1 of 2 sources checked
    • cites [64] source

      Section 3.7 (p. 8) describes manual ML workflows as complex and error-prone, and Section 5.3 (p. 31) explains that tool choices must account for compatibility between development and production stacks. The paper does not establish that selecting and coordinating both software and hardware components is the reason manual operation is challenging. (LLM, conf=0.83)

      Experimental — accuracy not yet measured
    • cites [65] Hewage, 2022, Machine learning operations: A survey on MLOps tool support, arXiv:2202.10169 source
  • Findingcitation

    Decreased prediction accuracy due to concept drift can be detected by a certain concept drift detection algorithm.

    “Decreased prediction accuracy due to concept drift can be detected by a certain concept drift detection algorithm [59].”
    Quote in the paperNot a substantial claim1 of 1 sources checked
  • Interpretationcitation

    Versioning of data, model, and code is required to ensure robustness and reproducibility.

    “as well as versioning of data, model, and code to ensure robustness and reproducibility [7], [32], [39].”
    Quote in the paperNot a substantial claim2 of 3 sources checked
    • cites [7] source
    • cites [32] source

      The Conclusions say that maintaining coherence and quality as models multiply requires systematic versioning of models and datasets. Background and Motivation describes reproducible increments involving code, data, and models. Our copy does not explicitly require versioning code or say that versioning ensures robustness and reproducibility. (LLM, conf=0.85)

      Experimental — accuracy not yet measured
    • cites [39] source

      In §6 (p. 13), the paper says a lack of versioning made CK workflows difficult to keep stable and bug-free, and describes a platform intended to version and test components and workflows. That supports the general link between versioning and reliability, but it does not state that versioning each of data, models, and code is required to ensure robustness and reproducibility. (LLM, conf=0.82)

      Experimental — accuracy not yet measured
  • Findingcitation

    There is a lack of highly skilled experts for MLOps roles, especially architects, data engineers, ML engineers, and DevOps engineers.

    “As our identified sources point out, there is a lack of highly skilled experts for these rolesespecially with regard to architects, data engineers, ML engineers, and DevOps engineers [26], [32], [38], [α, ε].”
    Quote in the paperNot a substantial claim1 of 3 sources checked
    • cites [26] source
    • cites [32] source

      The Survey section lists “not enough engineers/DevOps personnel” as an obstacle respondents were asked to rate, and the Figure 6 caption identifies it as a perceived challenge. The cited paper does not establish a shortage of highly skilled experts or single out architects, data engineers, and ML engineers as roles with shortages. (LLM, conf=0.82)

      Experimental — accuracy not yet measured
    • cites [38] source
  • Findingreasoning

    Nine principles are required to realize MLOps.

    “Based on the outlined methodology, we identified nine principles required to realize MLOps.”
    Quote in the paperSubstantial claim
  • Classified as probably not a substantial claim. 30
  • May combine two claims. 8

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationreasoning

    The large number of artifacts requires strong governance.

    “that require a strong governance [27], [32], [45]”
    Quote in the paperNot a substantial claim2 of 3 sources checked
    • cites [27] source

      Section 3.4 (page not provided) says production-grade AI model serving needs adequate governance, including versioning, access control, and auditability. It does not say that a large number of artifacts is what creates that need. (LLM, conf=0.82)

      Experimental — accuracy not yet measured
    • cites [32] source

      The Conclusions (page unavailable) say that as the number of models grows, maintaining coherence and quality becomes more complex and requires systematic versioning of models and data sets. This supports a need to manage growing ML assets, but the claim broadens that specific requirement to strong governance of artifacts generally. (LLM, conf=0.83)

      Experimental — accuracy not yet measured
    • cites [45] source

  • Classified as probably not a substantial claim. May combine two claims.

    Interpretationcitation

    Monitoring of the ML infrastructure, CI/CD, and orchestration are required.

    “Additionally, monitoring of the ML infrastructure, CI/CD, and orchestration are required[7],[23],[24],[28],[32],[33], [50], [α, ζ, η, θ].”
    Quote in the paperSubstantial claim1 of 3 sources checked
    • cites [7] source
    • cites [23] source
    • cites [24] source

      Section 4.2 (pp. 10, 13) calls infrastructure-resource monitoring essential and a robust CI/CD system necessary for rapid, reliable pipeline updates. Section 5 (p. 13) calls for an orchestration tool when deploying a scalable model, but does not say orchestration is required for every ML system; the citing sentence omits that condition. (LLM, conf=0.84)

      Experimental — accuracy not yet measured

  • Classified as probably not a substantial claim. May combine two claims.

    Findingown data

    By conducting a mixed-method study analyzing existing literature and tools, as well as interviewing eight experts from the field, the authors uncover four main aspects of MLOps: its principles, components, roles, and architecture.

    “By conducting a mixed-method study analyzing existing literature and tools, as well as interviewing eight experts from the field, we uncover four main aspects of MLOps: its principles, components, roles, and architecture.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingreasoning

    Retraining can be triggered automatically when a statistical threshold is reached, when new feature data is available, or scheduled periodically.

    “Retraining is not only triggered automatically when a statistical threshold is reached; it can also be triggered when new feature data is available, or it can be scheduled periodically.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingcitation

    Teams often work in silos rather than in cooperative setups, hindering MLOps.

    “This is often hindered because teams work in silos rather than in cooperative setups [α].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [α]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingreasoning

    Only a few peer-reviewed studies are available at the time of this research.

    “Thus, only a few peer-reviewed studies are available at the time of this research.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingcitation

    Failures can be a combination of ML infrastructure and software within the MLOps stack.

    “Failures can be a combination of ML infrastructure and software within the MLOps stack [7].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [7] source

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Findingnone given

    Different knowledge levels and specialized terminologies make communication difficult.

    “Additionally, different knowledge levels and specialized terminologies make communication difficult.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingreasoning

    It is challenging to resolve a potential support request (e.g., finding root cause) because many parties and components are involved.

    “Lastly, it is challenging to resolve a potential support request (e.g., by finding the root cause), as many parties and components are involved.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Findingnone given

    A constant incoming stream of new data forces retraining capabilities.

    “Also, a constant incoming stream of new data forces retraining capabilities.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Findingnone given

    In the real world, data scientists still manage ML workflows manually to a great extent.

    “In the real world, we observe data scientists still managing ML workflows manually to a great extent.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingreasoning

    MLOps requires specific roles: business stakeholder, solution architect, data scientist, data engineer, software engineer, DevOps engineer, and ML engineer/MLOps engineer.

    “After describing the principles and their resulting instantiation of components, we identify necessary roles in order to realize MLOps in the following.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Premisecitation

    A large number of ML projects fail, with many ML proofs of concept never progressing as far as production.

    “A large number of ML projects fail-with many ML proofs of concept never progressing as far as production [5].”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [5] van der Meulen, 2018, Gartner Says Nearly Half of CIOs Are Planning to Deploy Artificial Intelligence

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    DevOps ensures better software quality.

    “As empirical results demonstrate, DevOps ensures better software quality [15].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [15] source

  • May combine two claims.

    Premisecitation

    The ML community has focused extensively on building ML models, but not on building production-ready ML products or providing coordination of complex ML system components and infrastructure.

    “the ML community has focused extensively on the building of ML models, but not on (a) building production-ready ML products and (b) providing the necessary coordination of the resulting, often complex ML system components and infrastructure, including the roles required to automate and operate an ML system in a real-world setting [6].”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [6] source

  • Probably not a substantial claim: may not pass S4 (based). May combine two claims.

    Interpretationnone given

    MLOps is an interdisciplinary group process, and the interplay of different roles is crucial to design, manage, automate, and operate an ML system in production.

    “MLOps is an interdisciplinary group process, and the interplay of different roles is crucial to design, manage, automate, and operate an ML system in production.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Interpretationreasoning

    MLOps is positioned at the intersection of machine learning, software engineering, DevOps, and data engineering.

    “It becomes obvious that the term MLOps is positioned at the intersection of machine learning, software engineering, DevOps, and data engineering (see Figure 5).”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Interpretationnone given

    The paradigm of MLOps addresses the identified challenges.

    “The paradigm of Machine Learning Operations (MLOps) addresses these challenges.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: Only a small number of ML proofs of concept progress into deployment and production.
    • Rests on an assumption: The academic space has focused intensively on ML model building and benchmarking, but too little on operating complex ML systems in real-world scenarios.

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Premisenone given

    The actual deployment of the model depends on the use case and falls typically into one of these categories: real-time, batch, or serverless inference.

    “The actual deployment of the model depends on the use case and falls typically into one of these categories: real-time, batch, or serverless inference.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Study design and conductown data

    Eight expert interviews were conducted.

    “In total, we conduct eight interviews with experts (pseudonymized with αθ), whose details are depicted in Table 2.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Premisenone given

    The use of DevOps for ML, MLOps, and continuous practices in combination with ML is a relatively new field in academic literature.

    “It should be mentioned that the use of DevOps for ML, MLOps, and continuous practices in combination with ML is a relatively new field in academic literature.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Premisenone given

    Enterprise software/cloud services often allow connection to open-source tools via APIs and vice versa.

    “Enterprise softwares / cloud services often allow the connection to open-source tools via their APIs and vice versa.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable).

    Interpretationown data

    The results support a common understanding of the term MLOps and its associated concepts.

    “The results support a common understanding of the term MLOps and its associated concepts”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Interpretationreasoning

    The derived architecture is technology-agnostic.

    “The artifact was designed to be technology-agnostic.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductcitation

    Interviews were conducted until theoretical saturation was reached.

    “Interviews are conducted until no new categories and concepts emerge in the analysis of the data.According to Glaser and Strauss [21], this stage is called ''theoretical saturation.''”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Glaser and Strauss [21] source

  • Classified as probably not a substantial claim.

    Interpretationreasoning

    The identified principles are instantiated as technical components.

    “After identifying the principles that need to be incorporated into MLOps, we now elaborate on the precise components and implement them in the ML systems design.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Interpretationnone given

    There are constraints when it comes to API interface connections and combinations.

    “However, there are certainly also some constraints when it comes to API interfaces connections and combinations.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

    Interpretationcitation

    To successfully develop and run ML products, there needs to be a culture shift away from model-driven machine learning toward a product-oriented discipline.

    “As our insights from literature and interviews show, to successfully develop and run ML products, there needs to be a culture shift away from model-driven machine learning toward a productoriented discipline [γ].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [γ]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based).

    Interpretationnone given

    It is hard to say which technologies are good to combine and which aren't.

    “In general, it is hard to say which technologies are good to combine and which aren't.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Interpretationreasoning

    The MLOps process and components can be built from best-of-breed open-source tools, enterprise solutions, or a mix.

    “This means the MLOps process and components can either be built out of ''best-of-breed'' opensource tools, but also with enterprise solutions.Also, a mix and match combination of enterprise and open-source tools to realize MLOps is possible.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable).

    Interpretationreasoning

    MLOps needs to be a group process.

    “thus MLOps needs to be a group process [α].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [α]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationcitation

    Retraining is a repetitive task that requires a high level of automation.

    “This is a repetitive task which, again, requires a high level of automation [66], [θ].”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites [66] source
    • cites [θ]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductcitation

    Concepts were identified when they appeared multiple times in different interviews.

    “Concepts were identified when they appeared multiple times in different interviews [21].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [21] source
33 of 33

  • The paper relies on this without support.

    The final goal of all industrial ML projects is to develop ML products and rapidly bring them into production.

    “The final goal of all industrial machine learning (ML) projects is to develop ML products and rapidly bring them into production.”

  • The paper relies on this without support.

    MLOps is still a vague term and its consequences for researchers and professionals are ambiguous.

    “However, MLOps is still a vague term and its consequences for researchers and professionals are ambiguous.”

  • The paper relies on this without support.

    Most use cases have a read-heavy workload, combined with a batch or streaming-based ingestion pattern on very large data sets.

    “Most use cases have a read-heavy workload, combined with a batch or streaming-based ingestion pattern on (very) large data sets.”

  • The paper relies on this without support.

    Deep learning workloads (training and inference) are matrix-multiplication-heavy and therefore computation bound.

    “Typically, deep learning workloads (training and inference) are matrix-multiplication-heavy and therefore computation bound.”

  • The paper relies on this without support.

    GPUs are optimized towards matrix-multiplication-heavy workloads.

    “GPUs are optimized towards this type of workload and should be the primary focus for compute node specification.”

  • The paper relies on this without support.

    Only a small number of ML proofs of concept progress into deployment and production.

    “However, only a small number of these proofs of concept progress into deployment and production.”

  • The paper relies on this without support.

    The academic space has focused intensively on ML model building and benchmarking, but too little on operating complex ML systems in real-world scenarios.

    “Furthermore, the academic space has focused intensively on machine learning model building and benchmarking, but too little on operating complex machine learning systems in real-world scenarios.”
7 of 7

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Kreuzberger et al. [14] reported a mixed-method research, where they conducted an interview with practitioners in addition to an SLR.

    “Kreuzberger et al. [14] reported a mixed-method research, where they conducted an interview with practitioners in addition to an SLR.”

    From A Multivocal Review of MLOps Practices, Challenges and Open Issues (2026) · cites this paper as [14]

  • Kreuzberger et al. [14] discussed several open challenges related to adopting MLOps, but it is not the main objective of the study.

    “While they also discussed several open challenges related to adopting MLOps, it is not the main objective of the study.”

    From A Multivocal Review of MLOps Practices, Challenges and Open Issues (2026) · cites this paper as [14]

  • The conceptualization of MLOps in earlier systematic studies aligns closely with our RQ1.1 results (e.g., [14], [17]).

    “Conceptualization: The conceptualization of MLOps in earlier systematic studies aligns closely with our RQ1.1 results (e.g., [14], [17]).”

    From A Multivocal Review of MLOps Practices, Challenges and Open Issues (2026) · cites this paper as [14]

  • Our findings explicitly differentiate the responsibilities of the MLOps engineer role from those of ML engineers, positioning it within clearly defined workflows and clarifying its interactions with other emerging roles.

    “Although an earlier study identified the MLOps engineer role [14], our findings explicitly differentiate its responsibilities from those of ML engineers, positioning it within clearly defined workflows and clarifying its interactions with other emerging roles (Section 3.3).”

    From A Multivocal Review of MLOps Practices, Challenges and Open Issues (2026) · cites this paper as [14]

Cites

Flagged first, then by citations

2 of the 65 references couldn’t be matched to a known paper.

Research works printed but not linked: 7

  1. van der Meulen, 2018, Gartner Says Nearly Half of CIOs Are Planning to Deploy Artificial Intelligence
  2. Royce, Managing the development of large software systems, 1
  3. Beck, 2001, The agile manifesto
  4. Debois, 2009, Patrick Debois Devopsdays Ghent
  5. Mezak, 2018, The Origins of DevOps: What’s in a Name? DevOps. com
  6. Rütz, 2019, DEVOPS: A systematic literature review
  7. Baier, Challenges in the Deployment and, 1

Cited by

Sorted by citations

Related

Authors
Kreuzberger, D., Kühl, N., & Hirschl, S.
Affiliations
  • Karlsruhe Institute of Technology (DE)
  • IBM (Germany) (DE)
ORCID
Kühl, Niklas
Venue
IEEE Access
Publisher
Institute of Electrical and Electronics Engineers (IEEE)
Published
2023
Volume
11
Pages
31866-31879
ISSN
2169-3536 (ISSN-L)
DOI
10.1109/access.2023.3262138
OpenAlex
W4360995249
Cited by
766
References
65
Citations by year
2026: 245 · 2025: 239 · 2024: 168 · 2023: 106 · 2022: 8
FWCI
177.46
Type
journal article
Language
en
Open access
Licence
Locations
  • IEEE Access (publishedVersion, other-oa)
  • EPub Bayreuth (University of Bayreuth) (acceptedVersion, other-oa)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
Funders
  • Deutsche Forschungsgemeinschaft (491183248)
  • Open Access Publishing Fund of the University of Bayreuth
Topics
  • Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
  • Physical Sciences › Computer Science › Computer Networks and Communications › Software System Performance and Reliability
  • Social Sciences › Decision Sciences › Information Systems and Management › Scientific Computing and Data Management
Keywords
MLOps

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-11.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-11.

Cited sources are checked by gpt-6-sol.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review