The DOI matches its Crossref record; 63 of the 65 references it cites were found in a registry or an index; 8 of the 37 sources its claims cite were checked for support.
Machine Learning Operations (MLOps): Overview, Definition, and Architecture
Open accessAbstract
The final goal of all industrial machine learning (ML) projects is to develop ML products and rapidly bring them into production. However, it is highly challenging to automate and operationalize ML products and thus many ML endeavors fail to deliver on their expectations. The paradigm of Machine Learning Operations (MLOps) addresses this issue. MLOps includes several aspects, such as best practices, sets of concepts, and development culture. However, MLOps is still a vague term and its consequences for researchers and professionals are ambiguous. To address this gap, we conduct mixed-method research, including a literature review, a tool review, and expert interviews. As a result of these investigations, we contribute to the body of knowledge by providing an aggregated overview of the necessary principles, components, and roles, as well as the associated architecture and workflows. Furthermore, we provide a comprehensive definition of MLOps and highlight open challenges in the field.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
13 No basis given · 10 Own interpretation · 8 Sources not found · 7 No full text of the sources · 5 Partly supported · 3 Not checked yet · 2 Own method · 2 Own result · 1 Supported · accuracy not yet measured
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“In productive settings, it is challenging to operate ML manually due to different stacks of software and hardware components and their interplay as well as the selection of both ( [64], [65].”
§ Machine Learning Operations (MLOps): Overview, Definition, and Architecture > C. OPERATIONAL CHALLENGES
What the sources say
- Partly supportedcites [64] sourceExperimental — accuracy not yet measured
Section 3.7 (p. 8) describes manual ML workflows as complex and error-prone, and Section 5.3 (p. 31) explains that tool choices must account for compatibility between development and production stacks. The paper does not establish that selecting and coordinating both software and hardware components is the reason manual operation is challenging. (LLM, conf=0.83)
- Not checked — no full text of the sourcecites [65] Hewage, 2022, Machine learning operations: A survey on MLOps tool support, arXiv:2202.10169 source
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“Decreased prediction accuracy due to concept drift can be detected by a certain concept drift detection algorithm [59].”
§ Machine Learning Operations (MLOps): Overview, Definition, and Architecture > V. ARCHITECTURE AND WORKFLOW
What the sources say
- Supportedcites [59] source
The Change detection section describes an algorithm that tracks correct predictions over time and triggers an alarm when false predictions exceed a threshold. This supports detecting a decline in prediction accuracy associated with concept drift, provided true labels are available promptly. (LLM, conf=0.91)
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“as well as versioning of data, model, and code to ensure robustness and reproducibility [7], [32], [39].”
§ Machine Learning Operations (MLOps): Overview, Definition, and Architecture > C. OPERATIONAL CHALLENGES
What the sources say
- Not checked — no full text of the sourcecites [7] source
- Partly supportedcites [32] sourceExperimental — accuracy not yet measured
The Conclusions say that maintaining coherence and quality as models multiply requires systematic versioning of models and datasets. Background and Motivation describes reproducible increments involving code, data, and models. Our copy does not explicitly require versioning code or say that versioning ensures robustness and reproducibility. (LLM, conf=0.85)
- Partly supportedcites [39] sourceExperimental — accuracy not yet measured
In §6 (p. 13), the paper says a lack of versioning made CK workflows difficult to keep stable and bug-free, and describes a platform intended to version and test components and workflows. That supports the general link between versioning and reliability, but it does not state that versioning each of data, models, and code is required to ensure robustness and reproducibility. (LLM, conf=0.82)
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“As our identified sources point out, there is a lack of highly skilled experts for these rolesespecially with regard to architects, data engineers, ML engineers, and DevOps engineers [26], [32], [38], [α, ε].”
§ Machine Learning Operations (MLOps): Overview, Definition, and Architecture > A. ORGANIZATIONAL CHALLENGES
What the sources say
- Not checked — no full text of the sourcecites [26] source
- Partly supportedcites [32] sourceExperimental — accuracy not yet measured
The Survey section lists “not enough engineers/DevOps personnel” as an obstacle respondents were asked to rate, and the Figure 6 caption identifies it as a perceived challenge. The cited paper does not establish a shortage of highly skilled experts or single out architects, data engineers, and ML engineers as roles with shortages. (LLM, conf=0.82)
- Not checked — no full text of the sourcecites [38] source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Based on the outlined methodology, we identified nine principles required to realize MLOps.”
§ Machine Learning Operations (MLOps): Overview, Definition, and Architecture > A. PRINCIPLES
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“The mindset and culture of data science practice is a typical challenge in organizational settings [63].”
§ Machine Learning Operations (MLOps): Overview, Definition, and Architecture > A. ORGANIZATIONAL CHALLENGES
What the sources say
- Not matchedcites [63] Baier, Challenges in the Deployment and
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Not matched
None of the sources cited for this claim have been checked yet.
In the paper
“This is related to the necessary education of future professionalsas MLOps is typically not part of data science education [33] [γ].”
§ Machine Learning Operations (MLOps): Overview, Definition, and Architecture > A. ORGANIZATIONAL CHALLENGES
What the sources say
- Not checked — no full text of the sourcecites [33] source
- Not matchedcites [γ]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
None of the sources cited for this claim have been checked yet.
In the paper
“and requires a high level of flexibility in terms of scalability of the infrastructure [7], [33], [δ].”
§ Machine Learning Operations (MLOps): Overview, Definition, and Architecture > B. ML SYSTEM CHALLENGES
What the sources say
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“On the basis of the identified principles, components, and roles, we derive a generalized MLOps end-to-end architecture to give ML researchers and practitioners proper guidance.”
§ Machine Learning Operations (MLOps): Overview, Definition, and Architecture > V. ARCHITECTURE AND WORKFLOW
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“A major challenge with regard to MLOps systems is designing for fluctuating demand, especially in relation to the process of ML training [33].”
§ Machine Learning Operations (MLOps): Overview, Definition, and Architecture > B. ML SYSTEM CHALLENGES
What the sources say
- Not checked — no full text of the sourcecites [33] source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Probably not a substantial claim. 30
- It may state two separate claims as one. 8
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“that require a strong governance [27], [32], [45]”
§ Machine Learning Operations (MLOps): Overview, Definition, and Architecture > C. OPERATIONAL CHALLENGES
What the sources say
- Partly supportedcites [27] sourceExperimental — accuracy not yet measured
Section 3.4 (page not provided) says production-grade AI model serving needs adequate governance, including versioning, access control, and auditability. It does not say that a large number of artifacts is what creates that need. (LLM, conf=0.82)
- Partly supportedcites [32] sourceExperimental — accuracy not yet measured
The Conclusions (page unavailable) say that as the number of models grows, maintaining coherence and quality becomes more complex and requires systematic versioning of models and data sets. This supports a need to manage growing ML assets, but the claim broadens that specific requirement to strong governance of artifacts generally. (LLM, conf=0.83)
- Not checked — no full text of the sourcecites [45] source
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Partly supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Additionally, monitoring of the ML infrastructure, CI/CD, and orchestration are required[7],[23],[24],[28],[32],[33], [50], [α, ζ, η, θ].”
§ Machine Learning Operations (MLOps): Overview, Definition, and Architecture > VIII. CONCLUSION
What the sources say
- Not checked — no full text of the sourcecites [7] source
- Not checked — no full text of the sourcecites [23] source
- Partly supportedcites [24] sourceExperimental — accuracy not yet measured
Section 4.2 (pp. 10, 13) calls infrastructure-resource monitoring essential and a robust CI/CD system necessary for rapid, reliable pipeline updates. Section 5 (p. 13) calls for an orchestration tool when deploying a scalable model, but does not say orchestration is required for every ML system; the citing sentence omits that condition. (LLM, conf=0.84)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“By conducting a mixed-method study analyzing existing literature and tools, as well as interviewing eight experts from the field, we uncover four main aspects of MLOps: its principles, components, roles, and architecture.”
§ Machine Learning Operations (MLOps): Overview, Definition, and Architecture > VIII. CONCLUSION
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Retraining is not only triggered automatically when a statistical threshold is reached; it can also be triggered when new feature data is available, or it can be scheduled periodically.”
§ Machine Learning Operations (MLOps): Overview, Definition, and Architecture > V. ARCHITECTURE AND WORKFLOW
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
In the paper
“This is often hindered because teams work in silos rather than in cooperative setups [α].”
§ Machine Learning Operations (MLOps): Overview, Definition, and Architecture > A. ORGANIZATIONAL CHALLENGES
What the sources say
- Not matchedcites [α]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Not matched
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Thus, only a few peer-reviewed studies are available at the time of this research.”
§ Machine Learning Operations (MLOps): Overview, Definition, and Architecture > A. LITERATURE REVIEW
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“Failures can be a combination of ML infrastructure and software within the MLOps stack [7].”
§ Machine Learning Operations (MLOps): Overview, Definition, and Architecture > C. OPERATIONAL CHALLENGES
What the sources say
- Not checked — no full text of the sourcecites [7] source
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Additionally, different knowledge levels and specialized terminologies make communication difficult.”
§ Machine Learning Operations (MLOps): Overview, Definition, and Architecture > A. ORGANIZATIONAL CHALLENGES
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Lastly, it is challenging to resolve a potential support request (e.g., by finding the root cause), as many parties and components are involved.”
§ Machine Learning Operations (MLOps): Overview, Definition, and Architecture > C. OPERATIONAL CHALLENGES
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Also, a constant incoming stream of new data forces retraining capabilities.”
§ Machine Learning Operations (MLOps): Overview, Definition, and Architecture > C. OPERATIONAL CHALLENGES
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“In the real world, we observe data scientists still managing ML workflows manually to a great extent.”
§ Machine Learning Operations (MLOps): Overview, Definition, and Architecture > VIII. CONCLUSION
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“After describing the principles and their resulting instantiation of components, we identify necessary roles in order to realize MLOps in the following.”
§ Machine Learning Operations (MLOps): Overview, Definition, and Architecture > C. ROLES
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“A large number of ML projects fail-with many ML proofs of concept never progressing as far as production [5].”
§ Machine Learning Operations (MLOps): Overview, Definition, and Architecture > I. INTRODUCTION
What the sources say
- Not matchedcites [5] van der Meulen, 2018, Gartner Says Nearly Half of CIOs Are Planning to Deploy Artificial Intelligence
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Not matched
In the paper
“As empirical results demonstrate, DevOps ensures better software quality [15].”
§ Machine Learning Operations (MLOps): Overview, Definition, and Architecture > II. FOUNDATIONS OF DEVOPS
What the sources say
- Not checked — no full text of the sourcecites [15] source
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“the ML community has focused extensively on the building of ML models, but not on (a) building production-ready ML products and (b) providing the necessary coordination of the resulting, often complex ML system components and infrastructure, including the roles required to automate and operate an ML system in a real-world setting [6].”
§ Machine Learning Operations (MLOps): Overview, Definition, and Architecture > I. INTRODUCTION
What the sources say
- Not checked — no full text of the sourcecites [6] source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“MLOps is an interdisciplinary group process, and the interplay of different roles is crucial to design, manage, automate, and operate an ML system in production.”
§ Machine Learning Operations (MLOps): Overview, Definition, and Architecture > C. ROLES
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“It becomes obvious that the term MLOps is positioned at the intersection of machine learning, software engineering, DevOps, and data engineering (see Figure 5).”
§ Machine Learning Operations (MLOps): Overview, Definition, and Architecture > VI. CONCEPTUALIZATON
boostedcorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The paradigm of Machine Learning Operations (MLOps) addresses these challenges.”
§ Machine Learning Operations (MLOps): Overview, Definition, and Architecture > VIII. CONCLUSION
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
- Rests on an assumption: Only a small number of ML proofs of concept progress into deployment and production.
- Rests on an assumption: The academic space has focused intensively on ML model building and benchmarking, but too little on operating complex ML systems in real-world scenarios.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“The actual deployment of the model depends on the use case and falls typically into one of these categories: real-time, batch, or serverless inference.”
§ Machine Learning Operations (MLOps): Overview, Definition, and Architecture > VIII. CONCLUSION
plainconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“In total, we conduct eight interviews with experts (pseudonymized with αθ), whose details are depicted in Table 2.”
§ Machine Learning Operations (MLOps): Overview, Definition, and Architecture > C. INTERVIEW STUDY
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“It should be mentioned that the use of DevOps for ML, MLOps, and continuous practices in combination with ML is a relatively new field in academic literature.”
§ Machine Learning Operations (MLOps): Overview, Definition, and Architecture > A. LITERATURE REVIEW
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Enterprise softwares / cloud services often allow the connection to open-source tools via their APIs and vice versa.”
§ Machine Learning Operations (MLOps): Overview, Definition, and Architecture > V. ARCHITECTURE AND WORKFLOW
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The results support a common understanding of the term MLOps and its associated concepts”
§ Machine Learning Operations (MLOps): Overview, Definition, and Architecture > VIII. CONCLUSION
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The artifact was designed to be technology-agnostic.”
§ Machine Learning Operations (MLOps): Overview, Definition, and Architecture > V. ARCHITECTURE AND WORKFLOW
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
In the paper
“Interviews are conducted until no new categories and concepts emerge in the analysis of the data.According to Glaser and Strauss [21], this stage is called ''theoretical saturation.''”
§ Machine Learning Operations (MLOps): Overview, Definition, and Architecture > C. INTERVIEW STUDY
What the sources say
- Not checked — no full text of the sourcecites Glaser and Strauss [21] source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“After identifying the principles that need to be incorporated into MLOps, we now elaborate on the precise components and implement them in the ML systems design.”
§ Machine Learning Operations (MLOps): Overview, Definition, and Architecture > B. TECHNICAL COMPONENTS
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“However, there are certainly also some constraints when it comes to API interfaces connections and combinations.”
§ Machine Learning Operations (MLOps): Overview, Definition, and Architecture > V. ARCHITECTURE AND WORKFLOW
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
In the paper
“As our insights from literature and interviews show, to successfully develop and run ML products, there needs to be a culture shift away from model-driven machine learning toward a productoriented discipline [γ].”
§ Machine Learning Operations (MLOps): Overview, Definition, and Architecture > A. ORGANIZATIONAL CHALLENGES
What the sources say
- Not matchedcites [γ]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it.
- Not matched
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In general, it is hard to say which technologies are good to combine and which aren't.”
§ Machine Learning Operations (MLOps): Overview, Definition, and Architecture > V. ARCHITECTURE AND WORKFLOW
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This means the MLOps process and components can either be built out of ''best-of-breed'' opensource tools, but also with enterprise solutions.Also, a mix and match combination of enterprise and open-source tools to realize MLOps is possible.”
§ Machine Learning Operations (MLOps): Overview, Definition, and Architecture > V. ARCHITECTURE AND WORKFLOW
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
In the paper
“thus MLOps needs to be a group process [α].”
§ Machine Learning Operations (MLOps): Overview, Definition, and Architecture > A. ORGANIZATIONAL CHALLENGES
What the sources say
- Not matchedcites [α]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence.
- Not matched
None of the sources cited for this claim have been checked yet.
In the paper
“This is a repetitive task which, again, requires a high level of automation [66], [θ].”
§ Machine Learning Operations (MLOps): Overview, Definition, and Architecture > C. OPERATIONAL CHALLENGES
What the sources say
- Not checked — no full text of the sourcecites [66] source
- Not matchedcites [θ]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“Concepts were identified when they appeared multiple times in different interviews [21].”
§ Machine Learning Operations (MLOps): Overview, Definition, and Architecture > C. INTERVIEW STUDY
What the sources say
- Not checked — no full text of the sourcecites [21] source
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The paper relies on this without support.
The final goal of all industrial ML projects is to develop ML products and rapidly bring them into production.
“The final goal of all industrial machine learning (ML) projects is to develop ML products and rapidly bring them into production.”
The paper relies on this without support.
MLOps is still a vague term and its consequences for researchers and professionals are ambiguous.
“However, MLOps is still a vague term and its consequences for researchers and professionals are ambiguous.”
The paper relies on this without support.
Most use cases have a read-heavy workload, combined with a batch or streaming-based ingestion pattern on very large data sets.
“Most use cases have a read-heavy workload, combined with a batch or streaming-based ingestion pattern on (very) large data sets.”
The paper relies on this without support.
Deep learning workloads (training and inference) are matrix-multiplication-heavy and therefore computation bound.
“Typically, deep learning workloads (training and inference) are matrix-multiplication-heavy and therefore computation bound.”
The paper relies on this without support.
GPUs are optimized towards matrix-multiplication-heavy workloads.
“GPUs are optimized towards this type of workload and should be the primary focus for compute node specification.”
The paper relies on this without support.
Only a small number of ML proofs of concept progress into deployment and production.
“However, only a small number of these proofs of concept progress into deployment and production.”
The paper relies on this without support.
The academic space has focused intensively on ML model building and benchmarking, but too little on operating complex ML systems in real-world scenarios.
“Furthermore, the academic space has focused intensively on machine learning model building and benchmarking, but too little on operating complex machine learning systems in real-world scenarios.”
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“Kreuzberger et al. [14] reported a mixed-method research, where they conducted an interview with practitioners in addition to an SLR.”
From A Multivocal Review of MLOps Practices, Challenges and Open Issues (2026) · cites this paper as [14]
The cited source says what the claim attributes to it.
In the citing paper
“While they also discussed several open challenges related to adopting MLOps, it is not the main objective of the study.”
From A Multivocal Review of MLOps Practices, Challenges and Open Issues (2026) · cites this paper as [14]
The cited source says what the claim attributes to it.
In the citing paper
“Conceptualization: The conceptualization of MLOps in earlier systematic studies aligns closely with our RQ1.1 results (e.g., [14], [17]).”
From A Multivocal Review of MLOps Practices, Challenges and Open Issues (2026) · cites this paper as [14]
The cited source says what the claim attributes to it.
In the citing paper
“Although an earlier study identified the MLOps engineer role [14], our findings explicitly differentiate its responsibilities from those of ML engineers, positioning it within clearly defined workflows and clarifying its interactions with other emerging roles (Section 3.3).”
From A Multivocal Review of MLOps Practices, Challenges and Open Issues (2026) · cites this paper as [14]
Citations
Cites
The Discovery of Grounded Theory: Strategies for Qualitative Research
Glaser, B. G., & Strauss, A. L. (2017) · Aldine (Routledge reprint) · doi:10.4324/9780203793206 · Cited by 25,789
Grounded theory research: Procedures, canons, and evaluative criteria
Corbin, J. M., & Strauss, A. (1990) · Qualitative Sociology · doi:10.1007/bf00988593 · Cited by 11,133
Analyzing the Past to Prepare for the Future: Writing a Literature Review
Webster, J., & Watson, R. T. (2002) · MIS Quarterly · doi:10.2307/4132319 · Cited by 7,449 · cited 2×
“As a first step, we conduct a structured literature review [20,43] to obtain an overview of relevant research.”
MethodThe Hadoop Distributed File System
Shvachko, K. V., Kuang, H., Radia, S., & Chansler, R. J. (2010) · Digest of papers - IEEE Symposium on Mass Storage Systems · doi:10.1109/msst.2010.5496972 · Cited by 4,911 · cited 1×
“file systems [41], [42], or distributed databases [43], [44] combined with parallel and distributed data processing algorithms (e.”
Han, S., Mao, H., & Dally, W. J. (2015) · arXiv · doi:10.48550/arxiv.1510.00149 · Cited by 4,662
Systematic Literature Reviews in Software Engineering—A Systematic Literature Review
Kitchenham, B., Pearl Brereton, O., Budgen, D., Turner, M., Bailey, J., & Linkman, S. (2009) · Information and Software Technology · doi:10.1016/j.infsof.2008.09.009 · Cited by 4,537 · cited 2×
“As a first step, we conduct a structured literature review [20,43] to obtain an overview of relevant research.”
MethodGhemawat, S., Gobioff, H., & Leung, S.-T. A. (2003) · ACM SIGOPS Operating Systems Review · doi:10.1145/1165389.945450 · Cited by 2,665
Lakshman, A., & Malik, P. (2010) · ACM SIGOPS Operating Systems Review · doi:10.1145/1773912.1773922 · Cited by 2,664 · cited 1×
“file systems [41], [42], or distributed databases [43], [44] combined with parallel and distributed data processing algorithms (e.”
The qualitative interview in IS research: Examining the craft
Myers, M. D., & Newman, M. (2007) · Information and Organization · doi:10.1016/j.infoandorg.2006.11.001 · Cited by 1,941 · cited 3×
“In total, we conduct eight interviews with experts (α - θ), whose details are depicted in Table 2 With regard to the interview design, we prepare a semi-structured guide with several questions, documented as an interview script [33].”
MethodQuantized Neural Networks: Training Neural Networks with Low Precision Weights and ActivationsPreprint
Hubara, I., Courbariaux, M., Soudry, D., El‐Yaniv, R., & Bengio, Y. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1609.07061 · Cited by 1,663 · cited 1×
“Nets with low-precision floating-point operations [47] in combination with pruning and Hofmann Coding should be”
MethodGhemawat, S., Gobioff, H., & Leung, S.-T. A. (2003) · ACM SIGOPS Symposium on Operating Systems Principles (SOSP) · doi:10.1145/945445.945450 · Cited by 1,444
Online Learning and Online Convex Optimization
Shalev‐Shwartz, S. (2012) · Foundations and Trends® in Machine Learning · doi:10.1561/2200000018 · Cited by 1,364
Online Learning and Online Convex Optimization
Shalev‐Shwartz, S. (2011) · now publishers, Inc. eBooks · doi:10.1561/9781601985477 · Cited by 1,171
Spanner: Google's globally-distributed database
Corbett, J. C., Dean, J. B., Epstein, M., Fikes, A., Frost, C., Furman, J. J., Ghemawat, S., Gubarev, A., Heiser, C., Hochschild, P., et al. (2012) · doi:10.5555/2387880.2387905 · Cited by 827
Corbett, J. C., Dean, J., Epstein, M., Fikes, A., Frost, C., Furman, J. J., Ghemawat, S., Gubarev, A., Heiser, C., Hochschild, P., et al. (2013) · ACM Transactions on Computer Systems · doi:10.1145/2491245 · Cited by 732 · cited 1×
“file systems [41], [42], or distributed databases [43], [44] combined with parallel and distributed data processing algorithms (e.”
Designing interviews to generate rich data for information systems research
Schultze, U., & Avital, M. (2011) · Information and Organization · doi:10.1016/j.infoandorg.2010.11.001 · Cited by 522 · cited 2×
“Finally, we conduct semi-structured interviews [33,39] with experts from different domains.”
MethodA Survey of DevOps Concepts and Challenges
Leite, L., Rocha, C., Kon, F., Milojicic, D., & Meirelles, P. (2020) · ACM Computing Surveys · doi:10.1145/3359981 · Cited by 484 · cited 1×
“Due to the commercialization of DevOps, many DevOps tools are emerging, which can be differentiated into six groups [23,28]: collaboration and knowledge sharing (e.g., Slack, Trello, GitLab wiki), source code management (e.g., GitHub, GitLab), build process (e.g., Maven), continuous integration…”
BackgroundKocielnik, R., Amershi, S., & Bennett, P. N. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300641 · Cited by 277 · cited 1×
“However, the success of many productive ML applications in real-world settings falls short of expectations [21].”
BackgroundSCARFF : A scalable framework for streaming credit card fraud detection with spark
Carcillo, F., Pozzolo, A. D., Borgne, Y. L., Caelen, O., Mazzer, Y., & Bontempi, G. (2017) · Information Fusion · doi:10.1016/j.inffus.2017.09.005 · Cited by 272 · cited 1×
“into the Spark-based feature engineering pipeline [54].”
Machine learning for continuous innovation in battery technologies
Aykol, M., Herring, P. K., & Anapolsky, A. (2020) · Nature Reviews Materials · doi:10.1038/s41578-020-0216-y · Cited by 240 · cited 1×
“Machine Learning (ML) has become an important technique to leverage the potential of data and allows businesses to be more innovative [1], efficient [13], and sustainable [22].”
BackgroundWho Needs MLOps: What Data Scientists Seek to Accomplish and How Can MLOps Help?Preprint
Mäkinen, S., Skogström, H., Laaksonen, E., & Mikkonen, T. (2021) · arXiv (Cornell University) · Cited by 169
Applying DevOps Practices of Continuous Automation for Machine Learning
Karamitsos, I., Thabit, S., & Apostolopoulos, C. (2020) · Information · doi:10.3390/info11070363 · Cited by 140 · cited 9×
“The solution architect designs the architecture and defines the technologies to be used, following a thorough evaluation [17,27] [α, ζ].”
BackgroundMethodInfluentialSustainable MLOps: Trends and Challenges
Tamburri, D. A. (2020) · 2020 22nd International Symposium on Symbolic and Numeric Algorithms for Scientific Computing (SYNASC) · doi:10.1109/synasc51798.2020.00015 · Cited by 124 · cited 5×
“[25,35,40,41]”
BackgroundMethodInfluentialDemystifying MLOps and Presenting a Recipe for the Selection of Open-Source Tools
Ruf, P., Madan, M., Reich, C., & Ould-Abdeslam, D. (2021) · Applied Sciences · doi:10.3390/app11198861 · Cited by 112
A Cloud-Based Framework for Machine Learning Workloads and Applications
Lopez Garcia, A., De Lucas, J. M., Antonacci, M., Zu Castell, W., David, M., Hardt, M., Lloret Iglesias, L., Molto, G., Plociennik, M., Tran, V., et al. (2020) · IEEE Access · doi:10.1109/access.2020.2964386 · Cited by 108
Towards MLOps: A Case Study of ML Pipeline Platform
Zhou, Y., Yu, Y., & Ding, B. (2020) · 2020 International Conference on Artificial Intelligence and Computer Engineering (ICAICE) · doi:10.1109/icaice51518.2020.00102 · Cited by 103 · cited 9×
“Reproducibility is the ability to reproduce an ML experiment and obtain the exact same results [14,32,40,46] [α, β, δ, ε, η].”
BackgroundMethodInfluentialA Data Quality-Driven View of MLOpsPreprint
Renggli, C., Rimanić, L., Gürel, N. M., Karlaš, B., Wu, W., & Zhang, C. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2102.07750 · Cited by 86 · cited 1×
“Additionally, monitoring of the ML infrastructure, CI/CD, and orchestration are required [7,10,17,26,29,36,46] [α, ζ, η, θ].”
BackgroundAn Empirical Taxonomy of DevOps in Practice
Macarthy, R. W., & Bass, J. M. (2020) · Euromicro Conference on Software Engineering and Advanced Applications (SEAA) · doi:10.1109/seaa51224.2020.00046 · Cited by 74 · cited 1×
“Due to the commercialization of DevOps, many DevOps tools are emerging, which can be differentiated into six groups [23,28]: collaboration and knowledge sharing (e.g., Slack, Trello, GitLab wiki), source code management (e.g., GitHub, GitLab), build process (e.g., Maven), continuous integration…”
BackgroundImprove software quality through practicing DevOps
Perera, P., Silva, R., & Perera, I. (2017) · 2017 Seventeenth International Conference on Advances in ICT for Emerging Regions (ICTer) · doi:10.1109/icter.2017.8257807 · Cited by 69 · cited 1×
“As empirical results demonstrate, DevOps ensures better software quality [34].”
ResultMachine Learning Technologies for Sustainability in Smart Cities in the Post-COVID Era
Heras, A. D. L., Sendra, A. L., & Zamora‐Polo, F. (2020) · Sustainability · doi:10.3390/su12229320 · Cited by 68 · cited 1×
“Machine Learning (ML) has become an important technique to leverage the potential of data and allows businesses to be more innovative [1], efficient [13], and sustainable [22].”
BackgroundThe machine learning life cycle and the cloud: implications for drug discovery
Spjuth, O., Frid, J., & Hellander, A. (2021) · Expert Opinion on Drug Discovery · doi:10.1080/17460441.2021.1932812 · Cited by 67 · cited 7×
“[25,35,40,41]”
BackgroundMethodInfluentialObeid, J. S., Davis, M., Turner, M., Meystre, S. M., Heider, P. M., O'Bryan, E. C., & Lenert, L. A. (2020) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocaa105 · Cited by 60 · cited 1×
“, Obeid [57] for assessing the risk of COVID-19 based on health records)”
BackgroundDevOps for AI – Challenges in Development of AI-enabled Applications
Lwakatare, L. E., Crnković, I., & Bosch, J. (2020) · International Conference on Software, Telecommunications and Computer Networks (SoftCOM) · doi:10.23919/softcom50211.2020.9238323 · Cited by 55
Predicting Hotel Bookings Cancellation with a Machine Learning Classification Model
António, N., de Almeida, A., & Nunes, L. (2017) · International Conference on Machine Learning and Applications · doi:10.1109/icmla.2017.00-11 · Cited by 54
MLOps Challenges in Multi-Organization Setup: Experiences from Two Real-World Cases
Granlund, T., Kopponen, A., Stirbu, V., Myllyaho, L., & Mikkonen, T. (2021) · 2021 IEEE/ACM 1st Workshop on AI Engineering - Software Engineering for AI (WAIN) · doi:10.1109/wain52551.2021.00019 · Cited by 48
MLOps Challenges in Multi-Organization Setup: Experiences from Two\n Real-World CasesPreprint
Granlund, T., Kopponen, A., Stirbu, V., et al. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2103.08937 · Cited by 40 · cited 2×
“It provides fast feedback to developers regarding the success or failure of certain steps, thus increasing the overall productivity [15,17,26,27,35,42,46] [α, β, θ].”
BackgroundMachine Learning Operations: A Survey on MLOps Tool Support
Hewage, N., & Meedeniya, D. (2022) · arXiv · doi:10.48550/arxiv.2202.10169 · Cited by 38
Application of Machine Learning in Industry 4.0
Gourisaria, M. K., Agrawal, R., Harshvardhan, G. M., Pandey, M., & Rautaray, S. S. (2021) · Studies in big data · doi:10.1007/978-981-33-6518-6_4 · Cited by 36 · cited 1×
“Machine Learning (ML) has become an important technique to leverage the potential of data and allows businesses to be more innovative [1], efficient [13], and sustainable [22].”
BackgroundHow to Cope with Change? - Preserving Validity of Predictive Services over Time
Baier, L., Kühl, N., & Satzger, G. (2019) · Proceedings of the Annual Hawaii International Conference on System Sciences · doi:10.24251/hicss.2019.133 · Cited by 33 · cited 1×
“For instance, once the model-monitoring component (C9) detects a drift in the data [3], the information is forwarded to the scheduler, which then triggers the automated ML workflow pipeline for retraining (continuous training).”
MethodSelf-citationBenchmarking Machine Learning Solutions in Production
Cardoso Silva, L., Rezende Zagatti, F., Silva Sette, B., Nildaimon dos Santos Silva, L., Lucredio, D., Furtado Silva, D., & de Medeiros Caseli, H. (2020) · 2020 19th IEEE International Conference on Machine Learning and Applications (ICMLA) · doi:10.1109/icmla51294.2020.00104 · Cited by 33 · cited 8×
“The data scientist translates the business problem into an ML problem and takes care of the model engineering, including the selection of the best-performing algorithm and hyperparameters [7,14,26,29] [α, β, γ, δ, ε, ζ, η, θ].”
BackgroundMethodInfluentialFursin, G. (2021) · Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences · doi:10.1098/rsta.2020.0211 · Cited by 33 · cited 3×
“As a foundational infrastructure layer, a scalable and distributed model serving infrastructure is recommended [7,11,25,40,45,46]”
BackgroundBuilding A Platform for Machine Learning Operations from Open Source Frameworks
Liu, Y., Ling, Z., Huo, B., Wang, B., Chen, T., & Mouine, E. (2020) · IFAC-PapersOnLine · doi:10.1016/j.ifacol.2021.04.161 · Cited by 32 · cited 2×
“The business stakeholder defines the business goal to be achieved with ML and takes care of the communication side of the business, e.g., presenting the return on investment (ROI) generated with an ML product [17,24,26] [α, β, δ, θ].”
BackgroundMethodBuilding Continuous Integration Services for Machine Learning
Karlaš, B., Interlandi, M., Renggli, C., Wu, W., Zhang, C., Mukunthu Iyappan Babu, D., Edwards, J., Lauren, C., Xu, A., & Weimer, M. (2020) · Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining · doi:10.1145/3394486.3403290 · Cited by 30 · cited 1×
“This is a repetitive task which, again, requires a high level of automation [18] [θ].”
BackgroundDynamic Autoselection and Autotuning of Machine Learning Models for Cloud Network Analytics
Karn, R. R., Kudva, P. N., & Elfadel, I. M. (2018) · IEEE Transactions on Parallel and Distributed Systems · doi:10.1109/tpds.2018.2876844 · Cited by 29
From a Data Science Driven Process to a Continuous Delivery Process for Machine Learning Systems
Lwakatare, L. E., Crnkovic, I., Rånge, E., & Bosch, J. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-64148-1_12 · Cited by 28 · cited 10×
“…training job iteration (e.g., training date and time, duration, etc.), including the model specific metadata—e.g., used parameters and the resulting performance metrics, model lineage: data and code used—to ensure the full traceability of experiment runs [26,27,29,32,35] P8 Continuous monitoring.”
BackgroundMethodInfluentialWho Needs MLOps: What Data Scientists Seek to Accomplish and How Can MLOps Help?
Makinen, S., Skogstrom, H., Laaksonen, E., & Mikkonen, T. (2021) · 2021 IEEE/ACM 1st Workshop on AI Engineering - Software Engineering for AI (WAIN) · doi:10.1109/wain52551.2021.00024 · Cited by 28
Continuous Deployment of Machine Learning Pipelines
Derakhshan, B., Mahdiraji, A. R., Rabl, T., et al. (2019) · Movebank · doi:10.5441/002/edbt.2019.35 · Cited by 27
Efficient Large Scale NLP Feature Engineering with Apache Spark
Esmaeilzadeh, A., Heidari, M., Abdolazimi, R., Hajibabaee, P., & Malekzadeh, M. (2022) · IEEE Annual Computing and Communication Workshop and Conference (CCWC) · doi:10.1109/ccwc54503.2022.9720765 · Cited by 25
Model-Driven ML-Ops for Intelligent Enterprise Applications: Vision, Approaches and Challenges
van den Heuvel, W., & Tamburri, D. A. (2020) · Lecture notes in business information processing · doi:10.1007/978-3-030-52306-0_11 · Cited by 24
Automating Tiny ML Intelligent Sensors DevOPS Using Microsoft Azure
Vuppalapati, C., Ilapakurti, A., Chillara, K., Kedari, S., & Mamidi, V. (2020) · 2020 IEEE International Conference on Big Data (Big Data) · doi:10.1109/bigdata50022.2020.9377755 · Cited by 24 · cited 3×
“It provides fast feedback to developers regarding the success or failure of certain steps, thus increasing the overall productivity [15,17,26,27,35,42,46] [α, β, θ].”
BackgroundMachine Learning Pipelines: From Research to Production
Posoldova, A. (2020) · IEEE Potentials · doi:10.1109/mpot.2020.3016280 · Cited by 20 · cited 7×
“From a research perspective, this does not come as a surprise as the ML community has focused extensively on the building of ML models, but not on (a) building production-ready ML products and (b) providing the necessary coordination of the resulting, often complex ML system components and infrastructure, including th…”
BackgroundMethodInfluentialDetecting Concept Drift With Neural Network Model UncertaintyPreprint
Baier, L., Schlör, T., Schöffer, J., et al. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2107.01873 · Cited by 14 · cited 1×
“ponent (C9) detects a drift in the data [60], the information is forwarded to the scheduler, which then triggers the automated ML workflow pipeline for retraining (continuous training).”
BackgroundSelf-citationBanerjee, A., Chen, C., Hung, C., et al. (2020) · Cited by 13
An Automatic Artificial Intelligence Training Platform Based on Kubernetes
Wu, C., Haihong, E., & Song, M. (2020) · Proceedings of the 2020 2nd International Conference on Big Data Engineering and Technology · doi:10.1145/3378904.3378921 · Cited by 12 · cited 2×
“As our identified sources point out, there is a lack of highly skilled experts for these roles—especially with regard to architects, data engineers, ML engineers, and DevOps engineers [29,41,44] [α, ε].”
BackgroundFake News Detection Using Machine Learning Models
Aljabri, M., Alomari, D. M., & Aboulnour, M. (2022) · 2022 14th International Conference on Computational Intelligence and Communication Networks (CICN) · doi:10.1109/cicn56167.2022.10008340 · Cited by 12 · cited 1×
“, Aljabri [58] for NLP-based fake news detection).”
BackgroundWhen DevOps Meets Meta-Learning: A Portfolio to Rule them all
Benni, B., Blay–Fornarino, M., Mosser, S., et al. (2019) · doi:10.1109/models-c.2019.00092 · Cited by 10 · cited 1×
“Another feedback loop is required from the monitoring component (e.g., observing the model serving performance) to the scheduler to enable the retraining [4,6,7,17,27,46]”
Backgroundml-experiment: A Python framework for reproducible data science
Molner Domenech, A., & Guillén, A. (2020) · Journal of Physics: Conference Series · doi:10.1088/1742-6596/1603/1/012025 · Cited by 10 · cited 6×
“Reproducibility is the ability to reproduce an ML experiment and obtain the exact same results [14,32,40,46] [α, β, δ, ε, η].”
MethodInfluentialBuild an Intelligent Online Marketing System: An Overview
Cui, T., Wang, Y., & Namih, B. (2019) · IEEE Internet Computing · doi:10.1109/mic.2019.2924637 · Cited by 9 · cited 1×
“in the field of online advertising [62].”
Naive Bayes versus BERT: Jupyter notebook assignments for an introductory NLP course
Foster, J., & Wagner, J. (2021) · Workshop on Teaching NLP (TeachingNLP) · doi:10.18653/v1/2021.teachingnlp-1.20 · Cited by 5 · cited 1×
“field of natural language processing (NLP) [56].”
Jaganathan, D., & Ayyanathan, N. (2022) · Algorithms for intelligent systems · doi:10.1007/978-981-16-6332-1_38 · Cited by 4 · cited 1×
“ever, to some extent, a traditional ETL tool can be used to build a feature engineering pipeline [55].”
MethodMLOps in the Financial Industry: Philosophy, Practices, and Tools
Xu, J. (2022) · WORLD SCIENTIFIC eBooks · doi:10.1142/9789811250903_0014 · Cited by 4
DevOps Portal Design for SmartX AI Cluster Employing Cloud-Native Machine Learning Workflows
Yoon, G., Han, J., Lee, S., et al. (2020) · Lecture notes on data engineering and communications technologies · doi:10.1007/978-3-030-39746-3_54 · Cited by 3 · cited 1×
“As a foundational infrastructure layer, a scalable and distributed model serving infrastructure is recommended [7,11,25,40,45,46]”
BackgroundMLOps: Machine Learning Operations
Gupta, P., & Bagchi, A. (2024) · Synthesis lectures on engineering, science, and technology · doi:10.1007/978-3-031-43725-0_10 · Cited by 3
Research works printed but not linked: 7
- van der Meulen, 2018, Gartner Says Nearly Half of CIOs Are Planning to Deploy Artificial Intelligence
- Royce, Managing the development of large software systems, 1
- Beck, 2001, The agile manifesto
- Debois, 2009, Patrick Debois Devopsdays Ghent
- Mezak, 2018, The Origins of DevOps: What’s in a Name? DevOps. com
- Rütz, 2019, DEVOPS: A systematic literature review
- Baier, Challenges in the Deployment and, 1
Cited by
Sorted by citations
Challenges and opportunities for artificial intelligence in auditing: Evidence from the field
Kokina, J., Blanchette, S., Davenport, T. H., & Pachamanova, D. (2025) · International Journal of Accounting Information Systems · doi:10.1016/j.accinf.2025.100734 · Cited by 375
Advancements in nanomaterials for nanosensors: a comprehensive review
Darwish, M. A., Abd‐Elaziem, W., Elsheikh, A. H., & Zayed, A. A. A. (2024) · Nanoscale Advances · doi:10.1039/d4na00214h · Cited by 380
Avoiding common machine learning pitfalls
Lones, M. A. (2024) · Patterns · doi:10.1016/j.patter.2024.101046 · Cited by 298
“So, it is important to approach model building in an organized manner, making sure you use data correctly and putting adequate consideration into the choice of models.”
Steidl, M., Felderer, M., & Ramler, R. (2023) · Journal of Systems and Software · doi:10.1016/j.jss.2023.111615 · Cited by 126
“Kreuzberger et al. [22] cover principles within technical components (e.g., reproducibility achieved via feature store) and required professions to build the pipeline.”
BackgroundInfluentialHadi, M. U., Al-Tashi, Q., Qureshi, R., Shah, A., Muneer, A., Irfan, M., Zafar, A., Shaikh, M. B., Akhtar, N., Hassan, S. Z., et al. (2023) · doi:10.36227/techrxiv.23589741.v4 · Cited by 193
Metadata
- Authors
- Kreuzberger, D., Kühl, N., & Hirschl, S.
- Affiliations
- Karlsruhe Institute of Technology (DE)
- IBM (Germany) (DE)
- ORCID
- Kühl, Niklas
- Venue
- IEEE Access
- Published
- 2023
- Volume
- 11
- Pages
- 31866-31879
- ISSN
- 2169-3536 (ISSN-L)
- OpenAlex
- W4360995249
- Cited by
- 766
- References
- 65
- Citations by year
- 2026: 245 · 2025: 239 · 2024: 168 · 2023: 106
- FWCI
- 177.46
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- IEEE Access (publishedVersion, other-oa)
- EPub Bayreuth (University of Bayreuth) (acceptedVersion, other-oa)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Funders
- Deutsche Forschungsgemeinschaft (491183248)
- Open Access Publishing Fund of the University of Bayreuth
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
- Physical Sciences › Computer Science › Computer Networks and Communications › Software System Performance and Reliability
- Social Sciences › Decision Sciences › Information Systems and Management › Scientific Computing and Data Management
- Keywords
- MLOps
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-10-11, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-11 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
