The DOI matches its Crossref record; 63 of the 65 references it cites were found in a registry or an index; 8 of the 37 sources its claims cite were checked for support.
Machine Learning Operations (MLOps): Overview, Definition, and Architecture
Open accessAbstract
The final goal of all industrial machine learning (ML) projects is to develop ML products and rapidly bring them into production. However, it is highly challenging to automate and operationalize ML products and thus many ML endeavors fail to deliver on their expectations. The paradigm of Machine Learning Operations (MLOps) addresses this issue. MLOps includes several aspects, such as best practices, sets of concepts, and development culture. However, MLOps is still a vague term and its consequences for researchers and professionals are ambiguous. To address this gap, we conduct mixed-method research, including a literature review, a tool review, and expert interviews. As a result of these investigations, we contribute to the body of knowledge by providing an aggregated overview of the necessary principles, components, and roles, as well as the associated architecture and workflows. Furthermore, we provide a comprehensive definition of MLOps and highlight open challenges in the field.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
13 No basis given · 10 Own interpretation · 8 Sources not found · 7 No full text of the sources · 5 Partly supported · 3 Not checked yet · 2 Own method · 2 Own result · 1 Supported · accuracy not yet measured
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“In productive settings, it is challenging to operate ML manually due to different stacks of software and hardware components and their interplay as well as the selection of both ( [64], [65].”
§ Machine Learning Operations (MLOps): Overview, Definition, and Architecture > C. OPERATIONAL CHALLENGES
What the sources say
- Partly supportedcites [64] sourceExperimental — accuracy not yet measured
Section 3.7 (p. 8) describes manual ML workflows as complex and error-prone, and Section 5.3 (p. 31) explains that tool choices must account for compatibility between development and production stacks. The paper does not establish that selecting and coordinating both software and hardware components is the reason manual operation is challenging. (LLM, conf=0.83)
- Not checked — no full text of the sourcecites [65] Hewage, 2022, Machine learning operations: A survey on MLOps tool support, arXiv:2202.10169 source
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“Decreased prediction accuracy due to concept drift can be detected by a certain concept drift detection algorithm [59].”
§ Machine Learning Operations (MLOps): Overview, Definition, and Architecture > V. ARCHITECTURE AND WORKFLOW
What the sources say
- Supportedcites [59] source
The Change detection section describes an algorithm that tracks correct predictions over time and triggers an alarm when false predictions exceed a threshold. This supports detecting a decline in prediction accuracy associated with concept drift, provided true labels are available promptly. (LLM, conf=0.91)
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“as well as versioning of data, model, and code to ensure robustness and reproducibility [7], [32], [39].”
§ Machine Learning Operations (MLOps): Overview, Definition, and Architecture > C. OPERATIONAL CHALLENGES
What the sources say
- Not checked — no full text of the sourcecites [7] source
- Partly supportedcites [32] sourceExperimental — accuracy not yet measured
The Conclusions say that maintaining coherence and quality as models multiply requires systematic versioning of models and datasets. Background and Motivation describes reproducible increments involving code, data, and models. Our copy does not explicitly require versioning code or say that versioning ensures robustness and reproducibility. (LLM, conf=0.85)
- Partly supportedcites [39] sourceExperimental — accuracy not yet measured
In §6 (p. 13), the paper says a lack of versioning made CK workflows difficult to keep stable and bug-free, and describes a platform intended to version and test components and workflows. That supports the general link between versioning and reliability, but it does not state that versioning each of data, models, and code is required to ensure robustness and reproducibility. (LLM, conf=0.82)
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“As our identified sources point out, there is a lack of highly skilled experts for these rolesespecially with regard to architects, data engineers, ML engineers, and DevOps engineers [26], [32], [38], [α, ε].”
§ Machine Learning Operations (MLOps): Overview, Definition, and Architecture > A. ORGANIZATIONAL CHALLENGES
What the sources say
- Not checked — no full text of the sourcecites [26] source
- Partly supportedcites [32] sourceExperimental — accuracy not yet measured
The Survey section lists “not enough engineers/DevOps personnel” as an obstacle respondents were asked to rate, and the Figure 6 caption identifies it as a perceived challenge. The cited paper does not establish a shortage of highly skilled experts or single out architects, data engineers, and ML engineers as roles with shortages. (LLM, conf=0.82)
- Not checked — no full text of the sourcecites [38] source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Based on the outlined methodology, we identified nine principles required to realize MLOps.”
§ Machine Learning Operations (MLOps): Overview, Definition, and Architecture > A. PRINCIPLES
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“The mindset and culture of data science practice is a typical challenge in organizational settings [63].”
§ Machine Learning Operations (MLOps): Overview, Definition, and Architecture > A. ORGANIZATIONAL CHALLENGES
What the sources say
- Not matchedcites [63] Baier, Challenges in the Deployment and
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Not matched
None of the sources cited for this claim have been checked yet.
In the paper
“This is related to the necessary education of future professionalsas MLOps is typically not part of data science education [33] [γ].”
§ Machine Learning Operations (MLOps): Overview, Definition, and Architecture > A. ORGANIZATIONAL CHALLENGES
What the sources say
- Not checked — no full text of the sourcecites [33] source
- Not matchedcites [γ]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
None of the sources cited for this claim have been checked yet.
In the paper
“and requires a high level of flexibility in terms of scalability of the infrastructure [7], [33], [δ].”
§ Machine Learning Operations (MLOps): Overview, Definition, and Architecture > B. ML SYSTEM CHALLENGES
What the sources say
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“On the basis of the identified principles, components, and roles, we derive a generalized MLOps end-to-end architecture to give ML researchers and practitioners proper guidance.”
§ Machine Learning Operations (MLOps): Overview, Definition, and Architecture > V. ARCHITECTURE AND WORKFLOW
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“A major challenge with regard to MLOps systems is designing for fluctuating demand, especially in relation to the process of ML training [33].”
§ Machine Learning Operations (MLOps): Overview, Definition, and Architecture > B. ML SYSTEM CHALLENGES
What the sources say
- Not checked — no full text of the sourcecites [33] source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Probably not a substantial claim. 30
- It may state two separate claims as one. 8
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“that require a strong governance [27], [32], [45]”
§ Machine Learning Operations (MLOps): Overview, Definition, and Architecture > C. OPERATIONAL CHALLENGES
What the sources say
- Partly supportedcites [27] sourceExperimental — accuracy not yet measured
Section 3.4 (page not provided) says production-grade AI model serving needs adequate governance, including versioning, access control, and auditability. It does not say that a large number of artifacts is what creates that need. (LLM, conf=0.82)
- Partly supportedcites [32] sourceExperimental — accuracy not yet measured
The Conclusions (page unavailable) say that as the number of models grows, maintaining coherence and quality becomes more complex and requires systematic versioning of models and data sets. This supports a need to manage growing ML assets, but the claim broadens that specific requirement to strong governance of artifacts generally. (LLM, conf=0.83)
- Not checked — no full text of the sourcecites [45] source
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Partly supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Additionally, monitoring of the ML infrastructure, CI/CD, and orchestration are required[7],[23],[24],[28],[32],[33], [50], [α, ζ, η, θ].”
§ Machine Learning Operations (MLOps): Overview, Definition, and Architecture > VIII. CONCLUSION
What the sources say
- Not checked — no full text of the sourcecites [7] source
- Not checked — no full text of the sourcecites [23] source
- Partly supportedcites [24] sourceExperimental — accuracy not yet measured
Section 4.2 (pp. 10, 13) calls infrastructure-resource monitoring essential and a robust CI/CD system necessary for rapid, reliable pipeline updates. Section 5 (p. 13) calls for an orchestration tool when deploying a scalable model, but does not say orchestration is required for every ML system; the citing sentence omits that condition. (LLM, conf=0.84)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“By conducting a mixed-method study analyzing existing literature and tools, as well as interviewing eight experts from the field, we uncover four main aspects of MLOps: its principles, components, roles, and architecture.”
§ Machine Learning Operations (MLOps): Overview, Definition, and Architecture > VIII. CONCLUSION
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Retraining is not only triggered automatically when a statistical threshold is reached; it can also be triggered when new feature data is available, or it can be scheduled periodically.”
§ Machine Learning Operations (MLOps): Overview, Definition, and Architecture > V. ARCHITECTURE AND WORKFLOW
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
In the paper
“This is often hindered because teams work in silos rather than in cooperative setups [α].”
§ Machine Learning Operations (MLOps): Overview, Definition, and Architecture > A. ORGANIZATIONAL CHALLENGES
What the sources say
- Not matchedcites [α]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Not matched
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Thus, only a few peer-reviewed studies are available at the time of this research.”
§ Machine Learning Operations (MLOps): Overview, Definition, and Architecture > A. LITERATURE REVIEW
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“Failures can be a combination of ML infrastructure and software within the MLOps stack [7].”
§ Machine Learning Operations (MLOps): Overview, Definition, and Architecture > C. OPERATIONAL CHALLENGES
What the sources say
- Not checked — no full text of the sourcecites [7] source
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Additionally, different knowledge levels and specialized terminologies make communication difficult.”
§ Machine Learning Operations (MLOps): Overview, Definition, and Architecture > A. ORGANIZATIONAL CHALLENGES
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Lastly, it is challenging to resolve a potential support request (e.g., by finding the root cause), as many parties and components are involved.”
§ Machine Learning Operations (MLOps): Overview, Definition, and Architecture > C. OPERATIONAL CHALLENGES
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Also, a constant incoming stream of new data forces retraining capabilities.”
§ Machine Learning Operations (MLOps): Overview, Definition, and Architecture > C. OPERATIONAL CHALLENGES
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“In the real world, we observe data scientists still managing ML workflows manually to a great extent.”
§ Machine Learning Operations (MLOps): Overview, Definition, and Architecture > VIII. CONCLUSION
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“After describing the principles and their resulting instantiation of components, we identify necessary roles in order to realize MLOps in the following.”
§ Machine Learning Operations (MLOps): Overview, Definition, and Architecture > C. ROLES
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“A large number of ML projects fail-with many ML proofs of concept never progressing as far as production [5].”
§ Machine Learning Operations (MLOps): Overview, Definition, and Architecture > I. INTRODUCTION
What the sources say
- Not matchedcites [5] van der Meulen, 2018, Gartner Says Nearly Half of CIOs Are Planning to Deploy Artificial Intelligence
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Not matched
In the paper
“As empirical results demonstrate, DevOps ensures better software quality [15].”
§ Machine Learning Operations (MLOps): Overview, Definition, and Architecture > II. FOUNDATIONS OF DEVOPS
What the sources say
- Not checked — no full text of the sourcecites [15] source
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“the ML community has focused extensively on the building of ML models, but not on (a) building production-ready ML products and (b) providing the necessary coordination of the resulting, often complex ML system components and infrastructure, including the roles required to automate and operate an ML system in a real-world setting [6].”
§ Machine Learning Operations (MLOps): Overview, Definition, and Architecture > I. INTRODUCTION
What the sources say
- Not checked — no full text of the sourcecites [6] source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“MLOps is an interdisciplinary group process, and the interplay of different roles is crucial to design, manage, automate, and operate an ML system in production.”
§ Machine Learning Operations (MLOps): Overview, Definition, and Architecture > C. ROLES
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“It becomes obvious that the term MLOps is positioned at the intersection of machine learning, software engineering, DevOps, and data engineering (see Figure 5).”
§ Machine Learning Operations (MLOps): Overview, Definition, and Architecture > VI. CONCEPTUALIZATON
boostedcorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The paradigm of Machine Learning Operations (MLOps) addresses these challenges.”
§ Machine Learning Operations (MLOps): Overview, Definition, and Architecture > VIII. CONCLUSION
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
- Rests on an assumption: Only a small number of ML proofs of concept progress into deployment and production.
- Rests on an assumption: The academic space has focused intensively on ML model building and benchmarking, but too little on operating complex ML systems in real-world scenarios.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“The actual deployment of the model depends on the use case and falls typically into one of these categories: real-time, batch, or serverless inference.”
§ Machine Learning Operations (MLOps): Overview, Definition, and Architecture > VIII. CONCLUSION
plainconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“In total, we conduct eight interviews with experts (pseudonymized with αθ), whose details are depicted in Table 2.”
§ Machine Learning Operations (MLOps): Overview, Definition, and Architecture > C. INTERVIEW STUDY
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“It should be mentioned that the use of DevOps for ML, MLOps, and continuous practices in combination with ML is a relatively new field in academic literature.”
§ Machine Learning Operations (MLOps): Overview, Definition, and Architecture > A. LITERATURE REVIEW
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Enterprise softwares / cloud services often allow the connection to open-source tools via their APIs and vice versa.”
§ Machine Learning Operations (MLOps): Overview, Definition, and Architecture > V. ARCHITECTURE AND WORKFLOW
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The results support a common understanding of the term MLOps and its associated concepts”
§ Machine Learning Operations (MLOps): Overview, Definition, and Architecture > VIII. CONCLUSION
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The artifact was designed to be technology-agnostic.”
§ Machine Learning Operations (MLOps): Overview, Definition, and Architecture > V. ARCHITECTURE AND WORKFLOW
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
In the paper
“Interviews are conducted until no new categories and concepts emerge in the analysis of the data.According to Glaser and Strauss [21], this stage is called ''theoretical saturation.''”
§ Machine Learning Operations (MLOps): Overview, Definition, and Architecture > C. INTERVIEW STUDY
What the sources say
- Not checked — no full text of the sourcecites Glaser and Strauss [21] source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“After identifying the principles that need to be incorporated into MLOps, we now elaborate on the precise components and implement them in the ML systems design.”
§ Machine Learning Operations (MLOps): Overview, Definition, and Architecture > B. TECHNICAL COMPONENTS
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“However, there are certainly also some constraints when it comes to API interfaces connections and combinations.”
§ Machine Learning Operations (MLOps): Overview, Definition, and Architecture > V. ARCHITECTURE AND WORKFLOW
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
In the paper
“As our insights from literature and interviews show, to successfully develop and run ML products, there needs to be a culture shift away from model-driven machine learning toward a productoriented discipline [γ].”
§ Machine Learning Operations (MLOps): Overview, Definition, and Architecture > A. ORGANIZATIONAL CHALLENGES
What the sources say
- Not matchedcites [γ]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it.
- Not matched
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In general, it is hard to say which technologies are good to combine and which aren't.”
§ Machine Learning Operations (MLOps): Overview, Definition, and Architecture > V. ARCHITECTURE AND WORKFLOW
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This means the MLOps process and components can either be built out of ''best-of-breed'' opensource tools, but also with enterprise solutions.Also, a mix and match combination of enterprise and open-source tools to realize MLOps is possible.”
§ Machine Learning Operations (MLOps): Overview, Definition, and Architecture > V. ARCHITECTURE AND WORKFLOW
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
In the paper
“thus MLOps needs to be a group process [α].”
§ Machine Learning Operations (MLOps): Overview, Definition, and Architecture > A. ORGANIZATIONAL CHALLENGES
What the sources say
- Not matchedcites [α]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence.
- Not matched
None of the sources cited for this claim have been checked yet.
In the paper
“This is a repetitive task which, again, requires a high level of automation [66], [θ].”
§ Machine Learning Operations (MLOps): Overview, Definition, and Architecture > C. OPERATIONAL CHALLENGES
What the sources say
- Not checked — no full text of the sourcecites [66] source
- Not matchedcites [θ]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“Concepts were identified when they appeared multiple times in different interviews [21].”
§ Machine Learning Operations (MLOps): Overview, Definition, and Architecture > C. INTERVIEW STUDY
What the sources say
- Not checked — no full text of the sourcecites [21] source
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The paper relies on this without support.
The final goal of all industrial ML projects is to develop ML products and rapidly bring them into production.
“The final goal of all industrial machine learning (ML) projects is to develop ML products and rapidly bring them into production.”
The paper relies on this without support.
MLOps is still a vague term and its consequences for researchers and professionals are ambiguous.
“However, MLOps is still a vague term and its consequences for researchers and professionals are ambiguous.”
The paper relies on this without support.
Most use cases have a read-heavy workload, combined with a batch or streaming-based ingestion pattern on very large data sets.
“Most use cases have a read-heavy workload, combined with a batch or streaming-based ingestion pattern on (very) large data sets.”
The paper relies on this without support.
Deep learning workloads (training and inference) are matrix-multiplication-heavy and therefore computation bound.
“Typically, deep learning workloads (training and inference) are matrix-multiplication-heavy and therefore computation bound.”
The paper relies on this without support.
GPUs are optimized towards matrix-multiplication-heavy workloads.
“GPUs are optimized towards this type of workload and should be the primary focus for compute node specification.”
The paper relies on this without support.
Only a small number of ML proofs of concept progress into deployment and production.
“However, only a small number of these proofs of concept progress into deployment and production.”
The paper relies on this without support.
The academic space has focused intensively on ML model building and benchmarking, but too little on operating complex ML systems in real-world scenarios.
“Furthermore, the academic space has focused intensively on machine learning model building and benchmarking, but too little on operating complex machine learning systems in real-world scenarios.”
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“Kreuzberger et al. [14] reported a mixed-method research, where they conducted an interview with practitioners in addition to an SLR.”
From A Multivocal Review of MLOps Practices, Challenges and Open Issues (2026) · cites this paper as [14]
The cited source says what the claim attributes to it.
In the citing paper
“While they also discussed several open challenges related to adopting MLOps, it is not the main objective of the study.”
From A Multivocal Review of MLOps Practices, Challenges and Open Issues (2026) · cites this paper as [14]
The cited source says what the claim attributes to it.
In the citing paper
“Conceptualization: The conceptualization of MLOps in earlier systematic studies aligns closely with our RQ1.1 results (e.g., [14], [17]).”
From A Multivocal Review of MLOps Practices, Challenges and Open Issues (2026) · cites this paper as [14]
The cited source says what the claim attributes to it.
In the citing paper
“Although an earlier study identified the MLOps engineer role [14], our findings explicitly differentiate its responsibilities from those of ML engineers, positioning it within clearly defined workflows and clarifying its interactions with other emerging roles (Section 3.3).”
From A Multivocal Review of MLOps Practices, Challenges and Open Issues (2026) · cites this paper as [14]
Citations
Cites
The Discovery of Grounded Theory: Strategies for Qualitative Research
Glaser, B. G., & Strauss, A. L. (2017) · Aldine (Routledge reprint) · doi:10.4324/9780203793206 · Cited by 25,789
Grounded theory research: Procedures, canons, and evaluative criteria
Corbin, J. M., & Strauss, A. (1990) · Qualitative Sociology · doi:10.1007/bf00988593 · Cited by 11,133
Analyzing the Past to Prepare for the Future: Writing a Literature Review
Webster, J., & Watson, R. T. (2002) · MIS Quarterly · doi:10.2307/4132319 · Cited by 7,449 · cited 2×
“As a first step, we conduct a structured literature review [20,43] to obtain an overview of relevant research.”
MethodThe Hadoop Distributed File System
Shvachko, K. V., Kuang, H., Radia, S., & Chansler, R. J. (2010) · Digest of papers - IEEE Symposium on Mass Storage Systems · doi:10.1109/msst.2010.5496972 · Cited by 4,911 · cited 1×
“file systems [41], [42], or distributed databases [43], [44] combined with parallel and distributed data processing algorithms (e.”
Han, S., Mao, H., & Dally, W. J. (2015) · arXiv · doi:10.48550/arxiv.1510.00149 · Cited by 4,662
Systematic Literature Reviews in Software Engineering—A Systematic Literature Review
Kitchenham, B., Pearl Brereton, O., Budgen, D., Turner, M., Bailey, J., & Linkman, S. (2009) · Information and Software Technology · doi:10.1016/j.infsof.2008.09.009 · Cited by 4,537 · cited 2×
“As a first step, we conduct a structured literature review [20,43] to obtain an overview of relevant research.”
MethodGhemawat, S., Gobioff, H., & Leung, S.-T. A. (2003) · ACM SIGOPS Operating Systems Review · doi:10.1145/1165389.945450 · Cited by 2,665
Lakshman, A., & Malik, P. (2010) · ACM SIGOPS Operating Systems Review · doi:10.1145/1773912.1773922 · Cited by 2,664 · cited 1×
“file systems [41], [42], or distributed databases [43], [44] combined with parallel and distributed data processing algorithms (e.”
The qualitative interview in IS research: Examining the craft
Myers, M. D., & Newman, M. (2007) · Information and Organization · doi:10.1016/j.infoandorg.2006.11.001 · Cited by 1,941 · cited 3×
“In total, we conduct eight interviews with experts (α - θ), whose details are depicted in Table 2 With regard to the interview design, we prepare a semi-structured guide with several questions, documented as an interview script [33].”
MethodQuantized Neural Networks: Training Neural Networks with Low Precision Weights and ActivationsPreprint
Hubara, I., Courbariaux, M., Soudry, D., El‐Yaniv, R., & Bengio, Y. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1609.07061 · Cited by 1,663 · cited 1×
“Nets with low-precision floating-point operations [47] in combination with pruning and Hofmann Coding should be”
Method
Research works printed but not linked: 7
- van der Meulen, 2018, Gartner Says Nearly Half of CIOs Are Planning to Deploy Artificial Intelligence
- Royce, Managing the development of large software systems, 1
- Beck, 2001, The agile manifesto
- Debois, 2009, Patrick Debois Devopsdays Ghent
- Mezak, 2018, The Origins of DevOps: What’s in a Name? DevOps. com
- Rütz, 2019, DEVOPS: A systematic literature review
- Baier, Challenges in the Deployment and, 1
Cited by
Sorted by citations
Challenges and opportunities for artificial intelligence in auditing: Evidence from the field
Kokina, J., Blanchette, S., Davenport, T. H., & Pachamanova, D. (2025) · International Journal of Accounting Information Systems · doi:10.1016/j.accinf.2025.100734 · Cited by 375
Advancements in nanomaterials for nanosensors: a comprehensive review
Darwish, M. A., Abd‐Elaziem, W., Elsheikh, A. H., & Zayed, A. A. A. (2024) · Nanoscale Advances · doi:10.1039/d4na00214h · Cited by 380
Avoiding common machine learning pitfalls
Lones, M. A. (2024) · Patterns · doi:10.1016/j.patter.2024.101046 · Cited by 298
“So, it is important to approach model building in an organized manner, making sure you use data correctly and putting adequate consideration into the choice of models.”
Steidl, M., Felderer, M., & Ramler, R. (2023) · Journal of Systems and Software · doi:10.1016/j.jss.2023.111615 · Cited by 126
“Kreuzberger et al. [22] cover principles within technical components (e.g., reproducibility achieved via feature store) and required professions to build the pipeline.”
BackgroundInfluentialHadi, M. U., Al-Tashi, Q., Qureshi, R., Shah, A., Muneer, A., Irfan, M., Zafar, A., Shaikh, M. B., Akhtar, N., Hassan, S. Z., et al. (2023) · doi:10.36227/techrxiv.23589741.v4 · Cited by 193
Silva, A. B., Littlejohn, K. T., Liu, J. R., Moses, D. A., & Chang, E. F. (2024) · Nature Reviews Neuroscience · doi:10.1038/s41583-024-00819-9 · Cited by 103
“A growing literature aims to define general machine-learning principles to encourage the development of scalable, reliable and efficient systems 160 .”
El Arab, R. A., Abu-Mahfouz, M. S., Abuadas, F. H., Alzghoul, H., Almari, M., Ghannam, A., & Seweid, M. M. (2025) · Healthcare · doi:10.3390/healthcare13070701 · Cited by 108
Split Federated Learning for 6G Enabled-Networks: Requirements, Challenges, and Future Directions
Hafi, H., Brik, B., Frangoudis, P. A., Ksentini, A., & Bagaa, M. (2024) · IEEE Access · doi:10.1109/access.2024.3351600 · Cited by 90
“Therefore, it is required to not only perform continuous monitoring of both data and model profiles, but also automate the whole development process of SFL learning models, including data collection/extraction, model training, validation, and deployment [227], [228].”
Security of federated learning with IoT systems: Issues, limitations, challenges, and solutions
Yaacoub, J.-P. A., Noura, H., & Salman, O. (2023) · Internet of Things and Cyber-Physical Systems · doi:10.1016/j.iotcps.2023.04.001 · Cited by 106
A Joint Study of the Challenges, Opportunities, and Roadmap of MLOps and AIOps: A Systematic Survey
Díaz-de-Arcaya, J., Torre-Bastida, A. I., Zárate, G., Miñón, R., & Almeida, A. (2023) · ACM Computing Surveys · doi:10.1145/3625289 · Cited by 123
“First, MLOps is a well-established term in industry, although it was irst mentioned in scientiic literature in 2020, a contemporary deinition is provided by the authors of [69], in which the authors argue that MLOps leverage machine learning, DevOps, and data engineering, with the goal is to productionizing machine le…”
BackgroundMethodStudent Perspectives on the Role of Artificial Intelligence in Education: A Survey-Based Analysis
Idroes, G. M., Noviandy, T. R., Maulana, A., Irvanizam, I., Jalil, Z., Lensoni, L., Lala, A., Abas, A. H., Tallei, T. E., & Idroes, R. (2023) · Journal of Educational Management and Learning · doi:10.60084/jeml.v1i1.58 · Cited by 103
“In recent years, AI has witnessed important breakthroughs, particularly in machine learning [3, 4] and natural language processing [5, 6].”
BackgroundTian, Y., Zhang, Q., Jin, T., Zhang, Y., Lin, J., & Bai, X. (2024) · Ecological Indicators · doi:10.1016/j.ecolind.2024.112474 · Cited by 71
Transitioning from MLOps to LLMOps: Navigating the Unique Challenges of Large Language Models
Pahune, S. A., & Akhtar, Z. (2025) · Information · doi:10.3390/info16020087 · Cited by 80
Cognitive Soil Digital Twin for Monitoring the Soil Ecosystem: A Conceptual Framework
Tsakiridis, N., Samarinas, N., Kalopesa, E., & Zalidis, G. C. (2023) · Soil Systems · doi:10.3390/soilsystems7040088 · Cited by 66
Hadi, M. U., Al-Tashi, Q., Qureshi, R., Shah, A., Muneer, A., Irfan, M., Zafar, A., Shaikh, M. B., Akhtar, N., Hassan, S. Z., et al. (2023) · doi:10.36227/techrxiv.23589741.v3 · Cited by 71
Machine Learning Operations in Health Care: A Scoping Review
Rajagopal, A., Ayanian, S., Ryu, A. J., Qian, R., Legler, S. R., Peeler, E. A., Issa, M., Coons, T. J., & Kawamoto, K. (2024) · Mayo Clinic Proceedings: Digital Health · doi:10.1016/j.mcpdig.2024.06.009 · Cited by 52
Survey and insights on digital twins design and smart grid’s applications
Djebali, S., Guérard, G., & Taleb, I. (2023) · Future Generation Computer Systems · doi:10.1016/j.future.2023.11.033 · Cited by 55
Cob-Parro, A. C., Lalangui, Y., & Lazcano, R. (2024) · Agronomy · doi:10.3390/agronomy14020259 · Cited by 41
Towards Trustworthy Machine Learning in Production: An Overview of the Robustness in MLOps Approach
Bayram, F., & Ahmed, B. S. (2025) · ACM Computing Surveys · doi:10.1145/3708497 · Cited by 48
“Research has also explored the effectiveness of GANs in the energy sector for power consumption prediction and in defect detection [183].”
Almeida, B., David, J., Campos, F. S., & Cabral, P. (2024) · Applied Geography · doi:10.1016/j.apgeog.2024.103249 · Cited by 49
“The development of automated functions, akin to those found in ML Operations (MLOps) (Kreuzberger et al., 2023), is essential to streamline the process of taking ML models to production and subsequently maintaining and monitoring them in the context of satellite-based ESi modelling (Urbanowicz et…”
Unlocking the Potential of AI/ML in DevSecOps: Effective Strategies and Optimal Practices
Camacho, N. G. (2024) · Journal of Artificial Intelligence General science (JAIGS) ISSN:3006-4023 · doi:10.60087/jaigs.v2i1.p89 · Cited by 42
Machine Learning Applications in Manufacturing—Challenges, Trends, and Future Directions
Manta-Costa, A., Araújo, S. O., Peres, R. S., & Barata, J. (2024) · IEEE Open Journal of the Industrial Electronics Society · doi:10.1109/ojies.2024.3431240 · Cited by 49
“For instance, MLOps is an engineering practice that addresses issues in designing and maintaining ML models in production scenarios [11].”
BackgroundMLOps best practices, challenges and maturity models: A systematic literature review
Zarour, M., Alzabut, H., & Al-Sarayreh, K. T. (2025) · Information and Software Technology · doi:10.1016/j.infsof.2025.107733 · Cited by 44
Long, Y., Kroeger, S., Zaeh, M. F., & Brintrup, A. M. (2025) · International Journal of Production Research · doi:10.1080/00207543.2024.2447927 · Cited by 52
Boussaid, T., Rousset, F., Scuturici, V.-M., et al. (2024) · Applied Energy · doi:10.1016/j.apenergy.2024.123634 · Cited by 39
Data products, data mesh, and data fabric
Blohm, I., Wortmann, F., Legner, C., & Köbler, F. (2024) · Business & Information Systems Engineering · doi:10.1007/s12599-024-00876-5 · Cited by 41
From human-centered to symbiotic artificial intelligence: a focus on medical applications
Desolda, G., Esposito, A., Lanzilotti, R., Piccinno, A., & Costabile, M. F. (2024) · Multimedia Tools and Applications · doi:10.1007/s11042-024-20414-5 · Cited by 62
Machine learning models in phononic metamaterials
Liu, C., Yu, G., & Liu, Z. (2023) · Current Opinion in Solid State and Materials Science · doi:10.1016/j.cossms.2023.101133 · Cited by 37
Kirti, M., Maurya, A. K., & Yadav, R. S. (2024) · Concurrency and Computation Practice and Experience · doi:10.1002/cpe.8081 · Cited by 35
MLOps Challenges in Industry 4.0
Faubel, L., Schmid, K., & Eichelberger, H. (2023) · SN Computer Science · doi:10.1007/s42979-023-02282-2 · Cited by 36
“Currently, Machine Learning Operations (MLOps) as a collection of methods, techniques, and tools for integrating ML into software development practice is widely discussed as an enabler for large-scale ML applications [2, 3].”
BackgroundKummer, A., Ruppert, T., Medvegy, T., & Abonyi, J. (2022) · Results in Engineering · doi:10.1016/j.rineng.2022.100778 · Cited by 37
Economics and Equity of Large Language Models: Health Care Perspective
Nagarajan, R., Kondo, M., Salas, F., Sezgın, E., Yao, Y., Klotzman, V., Godambe, S. A., Khan, N., Limon, A., Stephenson, G. S., et al. (2024) · Journal of Medical Internet Research · doi:10.2196/64226 · Cited by 39
Enabling adaptive analytics at the edge with the Bi-Rex Big Data platform
Venanzi, R., Dahdal, S., Solimando, M., et al. (2023) · Computers in Industry · doi:10.1016/j.compind.2023.103876 · Cited by 33
Privacy-Preserving Machine Learning for Healthcare: Open Challenges and Future Perspectives
Guerra-Manzanares, A., López, L. J. L., Maniatakos, M., & Shamout, F. E. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-39539-0_3 · Cited by 46
“In addition to lack of resources, the deployment of these systems in medical settings can also be highly challenging (Kreuzberger et al., 2022; Wiesenfeld et al., 2022).”
BackgroundFilippou, K., Aifantis, G., Papakostas, G. A., & Tsekouras, G. E. (2023) · Information · doi:10.3390/info14040232 · Cited by 35
“…to assist the risk management associated with deploying ML models in production by providing traceability, monitoring, and testing capabilities [28,30,31]. set of tools and mechanisms able to enhance the collaboration between and IT professionals in applying and maintaining ML models [28,29].”
BackgroundInfluentialA Validity Perspective on Evaluating the Justified Use of Data-driven Decision-making Algorithms
Coston, A., Kawakami, A., Zhu, H., Holstein, K., & Heidari, H. (2023) · IEEE Conference on Secure and Trustworthy Machine Learning (SaTML) · doi:10.1109/satml54575.2023.00050 · Cited by 57
“While these efforts are still in their infancy, there is a growing body of work pointing to an urgent need for better tooling [69, 70].”
BackgroundDigital Twin for Continual Learning in Location Based Services
Lombardo, G., Picone, M., Mamei, M., Mordonini, M., & Poggi, A. (2023) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2023.107203 · Cited by 33
Recognising errors in AI implementation in radiology: A narrative review
Stogiannos, N., Cuocolo, R., D’Antonoli, T. A., dos Santos, D. P., Harvey, H., Huisman, M., Koçak, B., Kotter, E., Lekadir, K., Shelmerdine, S. C., et al. (2025) · European Journal of Radiology · doi:10.1016/j.ejrad.2025.112311 · Cited by 31
What drives MLOps adoption? An analysis using the TOE framework
Das, S. D., & Bala, P. K. (2023) · Journal of Decision System · doi:10.1080/12460125.2023.2214306 · Cited by 48
“Kreuzberger et al. (2022) provided an overview of the principles and technical components required to construct an MLOps system for improved machine learning (ML) results and data quality, quicker model development and deployment, and continuous monitoring for building a trustworthy AI system.”
InfluentialMachine learning operations landscape: platforms and tools
Berberi, L., Kozlov, V. Y., Nguyen, G., Díaz, J. S.-P., Calatrava, A., Moltó, G., Tran, V., & García, Á. L. (2025) · Artificial Intelligence Review · doi:10.1007/s10462-025-11164-3 · Cited by 47
“Insightful works, including (Subramanya et al. 2022 ; Mboweni et al. 2022 ; Kreuzberger et al. 2023 ; Diaz-de Arcaya et al. 2023 our work challenges this exclusive assignment, recognizing the potential limitations.”
InfluentialLiang, P., Song, B., Zhan, X., et al. (2024) · Applied and Computational Engineering · doi:10.54254/2755-2721/67/20240690 · Cited by 29
“…operations methods often fail to meet the specific needs of machine learning models in production, resulting in challenges such as the complexity of model deployment, difficulties in performance monitoring, and the absence of continuous integration and continuous deployment [1](CI/CD) processes.”
BackgroundAI Lifecycle Zero-Touch Orchestration within the Edge-to-Cloud Continuum for Industry 5.0
Alberti, E., Álvarez-Napagao, S., Anaya, V., Barroso, M., Barrué, C., Beecks, C., Bergamasco, L., Chala, S. A., Giménez-Ábalos, V., Grass, A., et al. (2024) · Syst. · doi:10.3390/systems12020048 · Cited by 26
Intelligent Edge Computing and Machine Learning: A Survey of Optimization and Applications
Cajas, S. A., Samanta, J., Suárez‐Cetrulo, A. L., & Carbajo, R. S. (2025) · Future Internet · doi:10.3390/fi17090417 · Cited by 47
“MLReef [113] utilizes git-based repositories for collaborative intelligent ML development.”
The human-AI dyad: Navigating the new frontier of entrepreneurial discourse
Jeremiah, F. (2024) · Futures · doi:10.1016/j.futures.2024.103529 · Cited by 38
Nan, Y., Ye, J., Wang, S., Yang, S., Lu, Y., Zhang, H., & Yang, X. (2024) · Atmospheric Research · doi:10.1016/j.atmosres.2024.107637 · Cited by 34
Perspectives on the strategic importance of digitalization for Modernizing African Agriculture
Uyeh, D. D., Gebremedhin, K. G., & Asem-Hiablie, S. (2023) · Computers and Electronics in Agriculture · doi:10.1016/j.compag.2023.107972 · Cited by 34
FedEYE: A scalable and flexible end-to-end federated learning platform for ophthalmology
Yan, B., Cao, D., Jiang, X., Chen, Y., Dai, W., Dong, F., Huang, W., Zhang, T., Gao, C., Chen, Q., et al. (2024) · Patterns · doi:10.1016/j.patter.2024.100928 · Cited by 32
Boisrond, K., Tardif, P.-M., & Jaafar, F. (2024) · IEEE Access · doi:10.1109/access.2024.3434618 · Cited by 30
“This is achieved through the monitoring, validation, and governance of machine learning models as presented by Kreuzberger et al. [40] Collaboration among data scientists is enabled, improving the speed, quality, and governance of model development through the monitoring and validation of machine…”
BackgroundMethodInfluentialA Systematic Review of Machine Learning DevOps
Mboweni, T., Masombuka, T., & Dongmo, C. (2022) · 2022 International Conference on Electrical, Computer and Energy Technologies (ICECET) · doi:10.1109/icecet55527.2022.9872968 · Cited by 34
“Perhaps one of the best ways to debunk some of these myths is to illustrate the main roles and their intersections contributing to the ML-DevOps paradigm as demonstrated by kreuzberger (2022) [11].”
BackgroundInfluentialAn analysis of the challenges in the adoption of MLOps
Amrit, C., & Narayanappa, A. K. (2024) · Journal of Innovation & Knowledge · doi:10.1016/j.jik.2024.100637 · Cited by 39
Ekanayake, B., Wong, J. K. W., Fini, A. A. F., Smith, P., & Thengane, V. (2024) · Project Leadership and Society · doi:10.1016/j.plas.2024.100149 · Cited by 46
“The deployment process of the model on the Streamlit platform offers an approach to a reliable and stable environment for executing and interacting with the deep learning or machine learning models as highlighted by Kreuzberger et al. (2023).”
MethodMishra, A., Jatti, V. S., Sefene, E. M., et al. (2023) · Materials Technology · doi:10.1080/10667857.2023.2295089 · Cited by 26
From Concept to Implementation: The Data-Centric Development Process for AI in Industry
Luley, P.-P., Deriu, J., Yan, P., Schatte, G. A., & Stadelmann, T. (2023) · IEEE Swiss Conference on Data Science (SDS) · doi:10.1109/sds57534.2023.00017 · Cited by 28
“For orchestration there exist many solutions such as Airflow [50], [51] or more commercial services like Microsoft Azure, Amazon Sagemaker, and Google Cloud Platform, that offer similar functionality pre-built at the expense of less control.”
MethodKarimi, H., Adibhesami, M. A., Hoseinzadeh, S., Salehi, A. A., Groppi, D., & Garcia, D. A. (2024) · Buildings · doi:10.3390/buildings14051342 · Cited by 27
On the Understandability of MLOps System Architectures
Warnett, S. J., & Zdun, U. (2024) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2024.3367488 · Cited by 25
“This problem worsens for ML systems deployed with MLOps as, in addition to the complex model and sys-tem deployments, component architectures, and CI/CD pipeline behaviours, the ML pipeline behaviours and ML-specific components need to be understood [2], [23], [29].”
BackgroundInfluentialA Multivocal Review of MLOps Practices, Challenges and Open Issues
Eken, B., Pallewatta, S., Tran, N., Tosun, A., & Babar, M. A. (2026) · ACM Computing Surveys · doi:10.1145/3747346 · Cited by 33
“This results in accidental complexities, and it is challenging to separate accidental complexity from essential complexity (PS154).”
Influentialde Almeida, J. G., Messiou, C., Withey, S. J., Matos, C., Koh, D.-M., & Papanikolaou, N. (2025) · European Radiology · doi:10.1007/s00330-025-11654-6 · Cited by 23
Data-related concepts for artificial intelligence education in K-12
Olari, V., & Romeike, R. (2024) · Computers and Education Open · doi:10.1016/j.caeo.2024.100196 · Cited by 33
“Second, advances in AI technologies and data science are constantly introducing new concepts (e.g., MLOps [46]), which increases the complexity of identifying stable, long-lasting concepts.”
BackgroundAkbasli, I. T., Bırbılen, A. Z., & Tekşam, Ö. (2025) · JAMIA Open · doi:10.1093/jamiaopen/ooae138 · Cited by 29
“39,43,45 In time-series models, AI learns patterns from longitudinal data for accurate forecasts.”
Kampezidou, S. I., Ray, A. T., Bhat, A. P., Fischer, O. J. P., & Mavris, D. N. (2024) · Eng—Advances in Engineering · doi:10.3390/eng5010021 · Cited by 25
Monitoring of Wildlife Using Unmanned Aerial Vehicle (UAV) With Machine Learning
Ahmed, Z. E., Hashim, A. H. A., Saeed, R. A., & Saeed, M. M. A. (2024) · Advances in computational intelligence and robotics book series · doi:10.4018/979-8-3693-0578-2.ch005 · Cited by 23
Sheikh, M. S., Enam, R. N., & Qureshi, R. I. (2023) · Frontiers in Computer Science · doi:10.3389/fcomp.2023.1293209 · Cited by 24
Safdar, M., Paul, P. P., Lamouche, G., Wood, G., Zimmermann, M., Hannesen, F., Bescond, C., Wanjara, P., & Zhao, Y. F. (2023) · Computers in Industry · doi:10.1016/j.compind.2023.104037 · Cited by 25
Mailach, A., & Siegmund, N. (2023) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse48619.2023.00067 · Cited by 23
Valdebenito, R., & Forcael, E. (2025) · Applied Sciences · doi:10.3390/app152312470 · Cited by 24
Raza, S. M., Minerva, R., Crespi, N., Alvi, M., Herath, M., & Dutta, H. (2025) · Computer Communications · doi:10.1016/j.comcom.2025.108144 · Cited by 27
ML meets aerospace: challenges of certifying airborne AI
Luettig, B., Akhiat, Y., & Daw, Z. (2024) · Frontiers in Aerospace Engineering · doi:10.3389/fpace.2024.1475139 · Cited by 21
Continuous fake media detection: Adapting deepfake detectors to new generative techniques
Tassone, F., Maiano, L., & Amerini, I. (2024) · Computer Vision and Image Understanding · doi:10.1016/j.cviu.2024.104143 · Cited by 20
Large Language Model Operations (LLMOps): Definition, Challenges, and Lifecycle Management
Díaz-de-Arcaya, J., López-de-Armentia, J., Miñón, R., Ojanguren, I. L., & Torre-Bastida, A. I. (2024) · International Conference on Smart and Sustainable Technologies (SpliTech) · doi:10.23919/splitech61897.2024.10612341 · Cited by 22
“Similarly, the authors in [2] mention that MLOps is still a vague term and its consequences for researchers and professionals are still ambiguous, which emphasizes the uncertainty for LLM management.”
BackgroundRequirements and software engineering for automotive perception systems: an interview study
Habibullah, K. M., Heyn, H.-M., Gay, G., Horkoff, J., Knauss, E., Borg, M., Knauss, A., Sivencrona, H., & Li, P. J. (2024) · Requirements Engineering · doi:10.1007/s00766-023-00410-1 · Cited by 21
“In terms of machine learning development, methods like CRISP-ML [85] and a more recent focus on MLOps [86] attempt to guide development.”
AI in Educational TechnologyPreprint
Zhao, T. (2023) · Preprints.org · doi:10.20944/preprints202311.0106.v1 · Cited by 29
Preventative Sensor-Based Remote Monitoring of the Diabetic Foot in Clinical Practice
Minty, E. P., Bray, E., Bachus, C. B., Everett, B., Smith, K. M., Matijevich, E., Hajizadeh, M., Armstrong, D. G., & Liden, B. A. (2023) · Sensors · doi:10.3390/s23156712 · Cited by 24
Navigating the DevOps landscape
Zhang, X., Zhao, P., & Jaskolka, J. (2025) · Journal of Systems and Software · doi:10.1016/j.jss.2024.112331 · Cited by 21
BTIS-Net: Efficient 3D U-Net for Brain Tumor Image Segmentation
Liu, L., & Xia, K. (2024) · IEEE Access · doi:10.1109/access.2024.3460797 · Cited by 19
Human reliability modeling in occupational environments toward a safe and productive operator 4.0
Kheiri, S. K., Vahedi, Z., Sun, H., Megahed, F. M., & Cavuoto, L. A. (2023) · International Journal of Industrial Ergonomics · doi:10.1016/j.ergon.2023.103479 · Cited by 19
Adapting to Changes: A Novel Framework for Continual Machine Learning in Industrial Applications
Antony, J., Jalušić, D., Bergweiler, S., Hajnal, Á., Žlabravec, V., Emődi, M., Strbad, D., Legler, T., & Marosi, A. C. (2024) · Journal of Grid Computing · doi:10.1007/s10723-024-09785-z · Cited by 18
“Machine Learning Operations (MLOps) [72] refers to the efficient lifecycle management of machine learning models, including the technologies and methodologies required.”
XAIport: A Service Framework for the Early Adoption of XAI in AI Model Development
Wang, Z., Liu, Y., Thiruselvi, A. A., & Hamou‐Lhadj, A. (2024) · IEEE/ACM International Conference on Software Engineering: New Ideas and Emerging Results (ICSE-NIER) · doi:10.1145/3639476.3639759 · Cited by 19
Addressing Challenges for the Effective Adoption of Artificial Intelligence in the Energy Sector
Park, C. (2025) · Sustainability · doi:10.3390/su17135764 · Cited by 17
Bhatt, D., Swain, M., & Yadav, D. (2025) · Journal of Contaminant Hydrology · doi:10.1016/j.jconhyd.2025.104541 · Cited by 19
MLOps Components, Tools, Process, and Metrics: A Systematic Literature Review
Woźniak, A. P., Milczarek, M., & Woźniak, J. (2025) · IEEE Access · doi:10.1109/access.2025.3534990 · Cited by 17
A continuous training approach for risk informed supplier selection and order allocation
Gabellini, M., Mak, S., Schoepf, S., Brintrup, A. M., & Regattieri, A. (2024) · Production & Manufacturing Research · doi:10.1080/21693277.2024.2447035 · Cited by 17
“The continuous acquisition and integration of new knowledge – referred to as continuous training in the AI community – has emerged as a promising avenue but remains underexplored in the context of supply chain management (Silvestre et al., 2023; Kreuzberger et al., 2023).”
A Data-Centric AI Paradigm for Socio-Industrial and Global Challenges
Majeed, A., & Hwang, S. O. (2024) · Electronics · doi:10.3390/electronics13112156 · Cited by 16
A Machine Learning Operations Platform for Streamlined Model Serving in Industry 5.0
Colombi, L., Gilli, A., Dahdal, S., Boleac, I., Tortonesi, M., Stefanelli, C., & Vignoli, M. (2024) · IEEE/IFIP Network Operations and Management Symposium (NOMS) · doi:10.1109/noms59830.2024.10575103 · Cited by 28
“Key activities in MLOps include data collection, analysis, preparation, model building, training, evaluation, selection, packaging, deployment, and monitoring [9], [11].”
BackgroundInfluentialBayram, F., Ahmed, B. S., & Hallin, E. (2024) · Journal of Systems and Software · doi:10.1016/j.jss.2024.112184 · Cited by 16
A responsible artificial intelligence framework for forensic science
Stacey, J., Fleming, R. I., Sheppard, D., Sheppard, J., Dobbie, G., & Karunakaran, D. (2025) · Forensic Science International · doi:10.1016/j.forsciint.2025.112548 · Cited by 21
Big Data-driven MLOps workflow for annual high-resolution land cover classification models
Burgueño-Romero, A. M., Barba-González, C., & Aldana-Montes, J. F. (2024) · Future Generation Computer Systems · doi:10.1016/j.future.2024.107499 · Cited by 16
RIOT-ML: toolkit for over-the-air secure updates and performance evaluation of TinyML models
Huang, Z., Zandberg, K., Schleiser, K., & Baccelli, E. (2024) · Annals of Telecommunications · doi:10.1007/s12243-024-01041-5 · Cited by 14
Wu, D., Smith, D., VanBerlo, B., Roshankar, A., Lee, H., Li, B., Ali, F., Rahman, M. A., Basmaji, J., Tschirhart, J. N., et al. (2024) · Diagnostics · doi:10.3390/diagnostics14111081 · Cited by 16
“Poorly performing subgroups were also identified via subgroup analyses and serve as targets for future data collection.”
MethodThreat Modeling AI/ML With the Attack Tree
Hoseini, S. V., Suutala, J., Partala, J., & Halunen, K. (2024) · IEEE Access · doi:10.1109/access.2024.3497011 · Cited by 22
“It can be based on a model architecture that is centralized or federated and online or offline (Batch Learning) due to the availability, distributive, and dynamicity of training data [53].”
Tailoring MLOps Techniques for Industry 5.0 Needs
Hegedűs, C., & Varga, P. (2023) · International Conference on Network and Service Management (CNSM) · doi:10.23919/cnsm59352.2023.10327814 · Cited by 17
“By automating model training, testing, and deployment, MLOps should (in theory at least) enhance model deployment speed, reliability, and scalability, making it easier for organizations to deploy and manage ML applications in production environments [3].”
BackgroundMethodEnd-User Development for Artificial Intelligence: A Systematic Literature Review
Esposito, A., Calvano, M., Curci, A., et al. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-34433-6_2 · Cited by 16
“However, the creation, testing, and deployment of AI models (and AI systems in general) is a collaborative activity that involves multiple actors with different expertise [39].”
BackgroundKristiani, E., Verma, V. K., & Chao-Tung, Y. (2026) · AI · doi:10.3390/ai7010015 · Cited by 18
Laouidi, K., Habchi, S., Fanezoune, C. K., Sallek, B., Kussul, N., & Bari, H. E. (2025) · Journal of Analytical and Applied Pyrolysis · doi:10.1016/j.jaap.2025.107098 · Cited by 14
Leuthe, D., Meyer-Hollatz, T., Plank, T., & Senkmüller, A. (2024) · Information Systems Frontiers · doi:10.1007/s10796-024-10526-6 · Cited by 25
“First, during the planning phase, besides understanding the organizational problems and opportunities, there needs to be a stronger focus on the identification of ML model requirements (Amershi et al., 2019; Kreuzberger et al., 2023).”
InfluentialMLOps: A Multiple Case Study in Industry 4.0
Faubel, L., & Schmid, K. (2024) · IEEE International Conference on Emerging Technologies and Factory Automation (ETFA) · doi:10.1109/etfa61755.2024.10711136 · Cited by 14
“While some of these papers focus on challenges related to ML systems and organizational aspects [7], [11], others identify organizational challenges, particularly in integrating collaborative MLOps practices [12] or deal with collaboration challenges, highlighting socio-technical aspects [13].”
BackgroundA Survey on Deep Learning for Design and Generation of Virtual Architecture
Wang, A., Dong, J., Lee, L., Shen, J., & Hui, P. (2024) · ACM Computing Surveys · doi:10.1145/3688569 · Cited by 24
“Various other surveys further define this scope, categorized as follows: Category 1 (machine learning [95, 143] or deep learning [1, 4, 6, 66, 87, 146]): 3D shape generation [21, 63, 137, 164], scene synthesis [181, 194], applications [34, 49], 3D reconstruction from 2D [187], and generative models…”
Verma, A. A., Trbovich, P., Mamdani, M., & Shojania, K. G. (2023) · BMJ Quality & Safety · doi:10.1136/bmjqs-2022-015713 · Cited by 17
“15 Customised implementations may also adopt ML operation approaches 39–41 to monitor data shifts 67 and model performance over time.”
BackgroundConceptualizing the Secure Machine Learning Operations (SecMLOps) Paradigm
Zhang, X., & Jaskolka, J. (2022) · IEEE International Conference on Software Quality, Reliability and Security (QRS) · doi:10.1109/qrs57517.2022.00023 · Cited by 30
“As mentioned in Section II, our description of SecMLOps in the subsequent sections is largely builds upon the work by Kreuzberger et al. [15].”
BackgroundMethodResultInfluentialSecuring RAG: A Risk Assessment and Mitigation Framework
Ammann, L., Ott, S., Landolt, C. R., & Lehmann, M. P. (2025) · IEEE Swiss Conference on Data Science (SDS) · doi:10.1109/sds66131.2025.00024 · Cited by 24
“Furthermore, we introduce a framework that embeds RAG-specific security considerations within existing security standards and provide implementation guidelines that emphasize the importance of a holistic approach to security.”
Trustworthy AI and Federated Learning for Intrusion Detection in 6G-Connected Smart Buildings
Garroppo, R. G., Giardina, P. G., Landi, G., & Ruta, M. (2025) · Future Internet · doi:10.3390/fi17050191 · Cited by 16
“Furthermore, the adoption of machine learning operations (MLOps) with a zero-touch architecture further automates the management of ML models, facilitating their seamless integration and continuous evolution within smart building security systems [16].”
Metadata
- Authors
- Kreuzberger, D., Kühl, N., & Hirschl, S.
- Affiliations
- Karlsruhe Institute of Technology (DE)
- IBM (Germany) (DE)
- ORCID
- Kühl, Niklas
- Venue
- IEEE Access
- Published
- 2023
- Volume
- 11
- Pages
- 31866-31879
- ISSN
- 2169-3536 (ISSN-L)
- OpenAlex
- W4360995249
- Cited by
- 766
- References
- 65
- Citations by year
- 2026: 245 · 2025: 239 · 2024: 168 · 2023: 106
- FWCI
- 177.46
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- IEEE Access (publishedVersion, other-oa)
- EPub Bayreuth (University of Bayreuth) (acceptedVersion, other-oa)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Funders
- Deutsche Forschungsgemeinschaft (491183248)
- Open Access Publishing Fund of the University of Bayreuth
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
- Physical Sciences › Computer Science › Computer Networks and Communications › Software System Performance and Reliability
- Social Sciences › Decision Sciences › Information Systems and Management › Scientific Computing and Data Management
- Keywords
- MLOps
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-10-11, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-11 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
