CoChat
Sign Up
Cited by 126Software Engineering Techniques and PracticesSoftware Engineering ResearchSoftware System Performance and Reliability

The pipeline for the continuous development of artificial intelligence models—Current state of research and practice

Open access
Steidl, Monika, Felderer, Michael, Ramler, Rudolf·2023·Journal of Systems and Software
DOI 10.1016/j.jss.2023.111615
Open this paper in CoChat — free account

Abstract

Companies struggle to continuously develop and deploy Artificial Intelligence (AI) models to complex production systems due to AI characteristics while assuring quality. To ease the development process, continuous pipelines for AI have become an active research area where consolidated and in-depth analysis regarding the terminology, triggers, tasks, and challenges is required. This paper includes a Multivocal Literature Review (MLR) where we consolidated 151 relevant formal and informal sources. In addition, nine-semi structured interviews with participants from academia and industry verified and extended the obtained information. Based on these sources, this paper provides and compares terminologies for Development and Operations (DevOps) and Continuous Integration (CI)/Continuous Delivery (CD) for AI, Machine Learning Operations (MLOps), (end-to-end) lifecycle management, and Continuous Delivery for Machine Learning (CD4ML). Furthermore, the paper provides an aggregated list of potential triggers for reiterating the pipeline, such as alert systems or schedules. In addition, this work uses a taxonomy creation strategy to present a consolidated pipeline comprising tasks regarding the continuous development of AI. This pipeline consists of four stages: Data Handling, Model Learning, Software Development and System Operations. Moreover, we map challenges regarding pipeline implementation, adaption, and usage for the continuous development of AI to these four stages.

AI snapshot

Written from the paper’s full text.

TL;DR

This paper consolidates research and practice on pipelines for the continuous development of AI through a Multivocal Literature Review of 151 sources and nine semi-structured interviews. It compares terminology, identifies triggers, proposes a four-stage taxonomy of pipeline tasks, and maps 25 challenges to those stages.

Key findings

  • The MLR identified 151 relevant sources: 79 formal papers (approximately 53%) and 72 informal sources; nine semi-structured interviews verified and extended the results.
  • The paper compares five terms: DevOps for AI, CI/CD for AI, MLOps, (end-to-end) lifecycle management, and CD4ML, noting that interviewees indicated differences between the terms are unknown in practice.
  • It aggregates triggers for starting or restarting the pipeline: feedback and alert systems, orchestration service and schedule, repository updates, and other triggers such as manual triggers and infrastructure/hardware changes.
  • It presents a taxonomy with four stages—Data Handling, Model Learning, Software Development, and System Operations—and maps 25 challenges to these stages; the most often mentioned Data Handling issue is data collection and integration with over 16 mentions, and environment/infrastructure handling was identified in 12 sources.

Methods

  • A Multivocal Literature Review based on Garousi et al. was conducted, including formal and informal sources from 2010–2021, using Google Scholar, Google, and YouTube, with backward and forward snowballing; the final set comprised 151 sources.
  • A taxonomy creation strategy based on Usman et al. used a facet approach with the four stages Data Handling, Model Learning, Software Development, and System Operations.
  • Nine semi-structured interviews were conducted between 30 July and 10 September 2021, averaging 49 minutes, with participants selected via stratified sampling from academia, industry, and industry start-ups; analysis followed Mayring’s summarizing qualitative content analysis with deductive category definition.

Limitations

  • The paper states conclusion validity is restricted because a qualitative approach was chosen; internal validity may be affected because only one researcher conducted the MLR, mitigated by a test-retest on 30 papers yielding 93.3% inclusion consistency and an average information-extraction difference of 23%.
  • The paper states construct and external validity were addressed through quality checklists, documented selection, a 5-point contribution scale, combining similar sources, including contrary findings, extensive search terms, and a two-stage interview-partner selection process.
  • The paper states its scope combines AI, ML, and DL without differentiating them and excludes special cases such as federated learning or transfer learning; evident from the text, the interview sample is small (nine participants), four of nine participants’ organizations did not have fully automated pipelines, and the TFX mapping shows TFX does not explicitly handle software development-specific tasks such as packaging or system-level quality assurance.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1016/j.jss.2023.111615

Cites

Statistics

Nothing to recompute.

Claims the paper makes

0 of 441 cited sources checked · 0 supported · 0 flagged

  • Findingown data
    § Abstract · p. 1

    The consolidated pipeline for continuous development of AI consists of four stages: Data Handling, Model Learning, Software Development, and System Operations.

    “This pipeline consists of four stages: Data Handling , Model Learning , Software Development and System Operations .”
    • Also stated in: § 3.2. Taxonomy Creation · p. 15 · plain
      “The identified facets comprise the stages Data Handling , Model Learning , Software Development and System Operations .”
    Quote in the paperSubstantial claim
  • Findingown data
    § Abstract · p. 1

    The paper's MLR consolidated 151 relevant formal and informal sources.

    “This paper includes a Multivocal Literature Review (MLR) where we consolidated 151 relevant formal and informal sources.”
    Quote in the paperSubstantial claim
  • Findingown data
    § 3.1. MLR · p. 12

    The final data set comprises 79 papers.

    “We derived the final data set that comprises 79 papers between the 15th of April and the 30th of May 2021.”
    Quote in the paperNot a substantial claim
  • Findingreasoning
    § 2.2. Related Work · p. 8

    Our study covers the full scope of AI models, independently of a specific application context.

    “In contrast, our study covers the full scope of AI models, independently of a specific application context.”
    Quote in the paperSubstantial claim
  • Findingnone given
    § 2.2. Related Work · p. 9

    No information regarding triggers of the pipeline for continuous development is covered by the related SLRs and MLRs.

    “No information regarding triggers of the pipeline for the con- tinuous development is covered by the related SLRs and MLRs.”
    Quote in the paperNot a substantial claim
  • Classified as probably not a substantial claim. 290
  • May combine two claims. 101
  • The quote only partly states this claim. 1

  • Classified as probably not a substantial claim.

    Findingown data
    § Abstract · p. 1

    The paper maps challenges regarding pipeline implementation, adaption, and usage for continuous development of AI to the four stages.

    “Moreover, we map challenges regarding pipeline implementation, adaption, and usage for the continuous development of AI to these four stages.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data
    § 4.1. Terminologies · p. 18

    According to three interviewed participants, the terms describe the process of adapting the standard software lifecycle to AI modelling in order to minimize the time between iterations and to ease the whole process of continuously developing and deploying AI models.

    “According to three of the interviewed participants, the terms describe the process of adapting the standard software lifecycle to AI modelling in order to minimize the time between iterations and to ease the whole process of continuously developing and deploying AI models.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The nine interviewed participants are representative of current practice in the continuous development of AI.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingreasoning
    § 2.2. Related Work · p. 8

    Definitions for related terms such as CI/CD for AI are not considered in the identified related work.

    “Definitions for related terms (e.g., CI/CD for AI) are not considered.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data
    § nation with a number. · p. 13

    Theoretical saturation was not achieved in seven out of twenty cases.

    “Theoretical saturation was not achieved in seven out of twenty cases.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Findingnone given
    § 2.2. Related Work · p. 8

    SLR or MLR based papers in the identified related work do not elaborate on the definition of terms for continuous development of AI.

    “SLR or MLR based papers in the identified related work do not elaborate on the definition of terms for continuous development of AI.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingreasoning
    § 3.1. MLR · p. 12

    We derived 37 formal sources as the start data set.

    “Based on these criteria we derived 37 formal sources as the start data set.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingnone given
    § 3.1. MLR · p. 12

    We derived informal sources between 31 May and 26 June 2021.

    “Based on these criteria we derived informal sources between the 31st of May and the 26th of June 2021.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data
    § 3.3. Qualitative Analysis · p. 15

    We identified nine participants.

    “Based on the selection criteria, we identified nine participants.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingown data
    § nation with a number. · p. 14

    The MLR identified 151 relevant sources, out of which 79 papers (approximately 53%) were formal sources and 72 informal sources.

    “The MLR identified 151 relevant sources, out of which 79 papers (approximately 53%) were formal sources and 72 informal sources.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data
    § 4.1. Terminologies · p. 18

    The differences between the terms DevOps for AI, CI/CD for AI, MLOps, lifecycle management, and CD4ML are unknown in practice.

    “The interviewees indicated that the differences between the terms are unknown in practice.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The nine interviewed participants are representative of current practice in the continuous development of AI.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data
    § 4.1.3. MLOps · p. 19

    MLOps is an extension of DevOps.

    “participant Z and R confirm that MLOps is an extension of DevOps.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The nine interviewed participants are representative of current practice in the continuous development of AI.

  • May combine two claims.

    Findingcitation
    § 4.2.1. Feedback and Alert Systems · p. 22

    Three interviewees identified data events as a potential trigger, whereas information extracted from literature also covers data and model changes.

    “Three interviewees identified data events as a potential trigger, whereas information extracted from literature also covers data and model changes [F59, I63, I52].”
    Quote in the paperNot a substantial claim0 of 3 sources checked
    • cites F59

      Not checked in the free run.

    • cites I63

      Not checked in the free run.

    • cites I52

      Not checked in the free run.

    • Rests on an assumption: The nine interviewed participants are representative of current practice in the continuous development of AI.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data
    § 4.2.1. Feedback and Alert Systems · p. 23

    It is ambiguous for participant D what the appropriate amount of new data to change the outcome of a model is.

    “Similar to the results obtained from the literature, it is ambiguous for participant D what the appropriate amount of new data to change the outcome of a model is.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The nine interviewed participants are representative of current practice in the continuous development of AI.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingnone given
    § 4.2.1. Feedback and Alert Systems · p. 23

    According to participants T, R and D, data updates should improve the model.

    “According to participants T, R and D, data updates should improve the model.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The nine interviewed participants are representative of current practice in the continuous development of AI.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data
    § 4.2.1. Feedback and Alert Systems · p. 23

    Event streaming platforms, such as Apache Kafka or other event hubs, may be used to semi-automate the triggering process.

    “Not mentioned in the extracted literature's information, however, indicated by participant R, is that event streaming platforms, such as Apache Kafka or other event hubs, may be used to semi-automate the triggering process.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The nine interviewed participants are representative of current practice in the continuous development of AI.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingcitation
    § 4.1.5. CD4ML · p. 21

    The term CD4ML is proposed, promoted and heavily influenced by Thoughtworks.

    “Participant T as well as the extracted information from the literature identify that this term is proposed, promoted and heavily influenced by Thoughtworks [F44, I59, I10]”
    Quote in the paperNot a substantial claim0 of 3 sources checked
    • cites F44

      Not checked in the free run.

    • cites I59

      Not checked in the free run.

    • cites I10

      Not checked in the free run.

    • Rests on an assumption: The nine interviewed participants are representative of current practice in the continuous development of AI.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data
    § 4.2. Triggers · p. 22

    It is a trial and error process to minimize resource consumption where several different team members identify the appropriate approach.

    “Participants R and D indicated that it is a trial and error process to minimize resource consumption where several different team members identify the appropriate approach.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The nine interviewed participants are representative of current practice in the continuous development of AI.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data
    § 4.2.1. Feedback and Alert Systems · p. 23

    According to participant R, the changes need to be extensive enough to significantly impact the model.

    “According to participant R, the changes need to be extensive enough to significantly impact the model.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The nine interviewed participants are representative of current practice in the continuous development of AI.

  • Probably not a substantial claim: may not pass S1 (asserted), S3 (load-bearing).

    Findingown data
    § 4.2.1. Feedback and Alert Systems · p. 22

    Due to privacy restrictions and data regulation requirements in Europe, users have the right that their associated data is forgotten.

    “For example, participant T mentioned that due to privacy restrictions and the data regulation requirements in Europe, users have the right that their associated data is forgotten.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The nine interviewed participants are representative of current practice in the continuous development of AI.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingcitation
    § 4.3. Pipeline · p. 24

    The pipeline and task execution strongly depends on the individual context, e.g. organizational policies for running the pipeline, and whether implementing the tasks outweigh the costs.

    “However, Garcia et al. [F25] as well as two participants (P and T) emphasized that the pipeline and task execution strongly depends on the individual context, e.g. organizational policies for running the pipeline, and whether implementing the tasks outweigh the costs.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites F25

      Not checked in the free run.

    • Rests on an assumption: The nine interviewed participants are representative of current practice in the continuous development of AI.

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Findingnone given
    § 4.3. Pipeline · p. 26

    The scope and types of tests strongly vary from implementation to implementation.

    “In addition, the scope and types of tests strongly vary from implementation to implementation.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data
    § 4.2.4. Other Triggers · p. 24

    Although manual triggers are sparsely elaborated in the collected literature, three out of nine participants use triggers that involve human interaction to identify if sufficient data is available.

    “Although manual triggers are sparsely elaborated in the collected literature [F42, I42, I31, I5], three out of nine participants use triggers that involve human interaction to identify if sufficient data is available.”
    Quote in the paperNot a substantial claim0 of 3 sources checked
    • cites F42

      Not checked in the free run.

    • cites I42

      Not checked in the free run.

    • cites I31

      Not checked in the free run.

    • Rests on an assumption: The nine interviewed participants are representative of current practice in the continuous development of AI.

  • May combine two claims.

    Findingcitation
    § 4.3. Pipeline · p. 26

    The pipeline is not linear but relies on feedback loops throughout the process.

    “As depicted in Figure 4, the pipeline is not linear but relies on feedback loops throughout the process.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingown data
    § 4.2.4. Other Triggers · p. 24

    Another reason for manual triggers is that humans can better estimate if their model requires retraining and if the improved quality still satisfies the user's needs.

    “Another reason is that humans can better estimate if their model requires retraining and if the improved quality still satisfies the user's needs.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The nine interviewed participants are representative of current practice in the continuous development of AI.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingcitation
    § 4.3.1. Data Handling · p. 26

    A study conducted by Hummer et al. indicates that the data handling uses 7% of the total execution time, but this time can be reduced due to parallelized computing procedures.

    “A study conducted by Hummer et al. [F28] indicates that the data handling uses 7% of the total execution time, but this time can be reduced due to parallelized computing procedures [F54, F7, F49, F28, F16].”
    Quote in the paperNot a substantial claim0 of 3 sources checked
    • cites F28

      Not checked in the free run.

    • cites F54

      Not checked in the free run.

    • cites F7

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingnone given
    § 4.3.1. Data Handling · p. 26

    The stage Data Handling covers the end-to-end lifecycle of data curation.

    “The stage Data Handling covers the end-to-end lifecycle of data curation.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data
    § 4.2.2. Orchestration Service and Schedule · p. 23

    One participant triggered their pipeline once a week.

    “For example, one participant triggered their pipeline once a week.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The nine interviewed participants are representative of current practice in the continuous development of AI.

  • Probably not a substantial claim: may not pass S1 (asserted), S3 (load-bearing).

    Findingown data
    § 4.2.3. Repository · p. 23

    When using repository triggers, the data set and the appropriate code should be in the same system and under the same source control.

    “When using this approach, participant T indicated that the data set and the appropriate code should be in the same system and under the same source control.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The nine interviewed participants are representative of current practice in the continuous development of AI.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingown data
    § Data Preprocessing · p. 27

    If too much data is available, the data set is reduced, and participant A mentioned the risk of introducing bias.

    “If too much data is available, the data set is reduced where participant A mentioned the risk of introducing bias.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The nine interviewed participants are representative of current practice in the continuous development of AI.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingnone given
    § Data Quality Assurance · p. 28

    The better the data quality is, the better the model results are.

    “Participant R and B emphasized that the better the data quality is, the better the model results are.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The nine interviewed participants are representative of current practice in the continuous development of AI.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data
    § Data Preprocessing · p. 27

    In cases where not enough data can be extracted, three interviewees (P, B and V) stated that they synthetically generate data to balance the data set.

    “In cases where not enough data can be extracted, three interviewees (P, B and V) stated that they synthetically generate data to balance the data set.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The nine interviewed participants are representative of current practice in the continuous development of AI.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data
    § Data Preprocessing · p. 28

    Data labelling is necessary for supervised learning.

    “Data labelling is necessary for supervised learning as indicated by participant P and C.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The nine interviewed participants are representative of current practice in the continuous development of AI.

  • May combine two claims.

    Findingreasoning
    § Data Versioning · p. 30

    Only the latest version is stored in participant P, T and Z's case and the previous versions are suspended.

    “Thus, only the latest version is stored in participant P, T and Z's case and the previous versions are suspended.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The nine interviewed participants are representative of current practice in the continuous development of AI.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data
    § Data Versioning · p. 30

    Participant V and D stored their data in-house due to the user's preference and regulations regarding private data, such as patient records.

    “Participant V and D indicated that they stored their data in-house due to the user's preference and regulations regarding for private data, such as patient records.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The nine interviewed participants are representative of current practice in the continuous development of AI.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingnone given
    § Data Quality Assurance · p. 28

    Participant A and R check the right shape of data as common validation practice.

    “Participant A and R check the right shape of data as common validation practice.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The nine interviewed participants are representative of current practice in the continuous development of AI.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data
    § Data Versioning · p. 30

    Participant C, V and D highlighted the need for a full data lineage if important models are based on this data.

    “Participant C, V and D highlighted the need for a full data lineage if important models are based on this data.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The nine interviewed participants are representative of current practice in the continuous development of AI.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data
    § Data Quality Assurance · p. 28

    Participant C uses manually or automatically created data schemas.

    “Participant C also uses manually or automatically created data schemas.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The nine interviewed participants are representative of current practice in the continuous development of AI.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data
    § AI Model Quality Assurance · p. 33

    Participant P executed quality assurance tests manually due to missing configurations in the pipeline.

    “Participant P also stated that they executed quality assurance tests manually due missing configurations in the pipeline”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingnone given
    § 4.3.2. Model Learning · p. 32

    Further tasks in the Model Learning stage comprise metadata capturing, versioning of the model and its dependencies, and documentation.

    “Further tasks comprise metadata capturing, versioning of the model and its dependencies, and documentation.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingnone given
    § 4.3.2. Model Learning · p. 32

    After data handling, the pipeline executes tasks associated with the AI model learning, such as model design, model training, quality assurance and improvement.

    “After data handling, the pipeline executes tasks associated with the AI model learning, such as model design, model training, quality assurance and improvement.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingcitation
    § Data Versioning · p. 31

    Miao et al. and participant T suggest versioning the data processing steps including applied code and metadata.

    “Miao et al. [F50] and participant T suggest versioning the data processing steps including applied code and metadata [F42, F2, I59].”
    Quote in the paperNot a substantial claim0 of 3 sources checked
    • cites F50

      Not checked in the free run.

    • cites F42

      Not checked in the free run.

    • cites F2

      Not checked in the free run.

    • Rests on an assumption: The nine interviewed participants are representative of current practice in the continuous development of AI.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data
    § Design · p. 32

    Participant C illustrated that modeld.io eases the decision making process by providing AutoML.

    “Participant C illustrated that modeld.io ease the decision making process by providing AutoML.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The nine interviewed participants are representative of current practice in the continuous development of AI.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data
    § AI Model Quality Assurance · p. 33

    Participant D indicated that experts had to validate the model's decisions individually.

    “For instance, participant D indicated that experts had to validate the model's decisions individually.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingnone given
    § Data Documentation · p. 31

    Although documentation is essential, in practice only one-third of the participants rely on manual data documentation, whereas the others do not document their data related information due to missing guidelines and software support.

    “Although documentation is essential, in practice only one-third of the participants rely on manual data documentation, whereas the others do not document their data related information due to missing guidelines and software support.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The nine interviewed participants are representative of current practice in the continuous development of AI.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data
    § Model Training · p. 32

    Two participants identified that TFX and MLFlow provide the necessary implementations or support for using AI libraries.

    “Two participants identified that TFX and MLFlow provide the necessary implementations or support for using AI libraries.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The nine interviewed participants are representative of current practice in the continuous development of AI.

  • May combine two claims.

    Findingcitation
    § Model Improvement · p. 34

    Pipelines support model compression or pruning, model hardening, and hyper-parameter optimization.

    “For instance, pipelines support model compression or pruning [F28, F34, F12], model hardening [F27, I11, I20], and hyper-parameter optimization [F68, F31, F12, F30, I12].”
    Quote in the paperNot a substantial claim0 of 3 sources checked
    • cites F28

      Not checked in the free run.

    • cites F34

      Not checked in the free run.

    • cites F12

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data
    § Model Versioning · p. 35

    During the interviews, participants proposed storage facilities of MLFlow, H2O, DataRobot and Git Large File Storage.

    “During the interviews, participants proposed storage facilities of MLFlow, H2O, DataRobot and Git Large File Storage.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingown data
    § Model Documentation · p. 35

    Documentation is required to receive certifications, which is especially important in the healthcare sector according to participant D.

    “This is required to receive certifications, which is especially important in the healthcare sector according to participant D.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: Documentation proves that the model adheres to the regulations and restrictions and works as expected.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data
    § AI Model Quality Assurance · p. 33

    Participant A mentioned using Normalized Discounted Cumulative Gain for measuring the effectiveness of their search engine model.

    “Another approach mentioned by participant A is a Normalized Discounted Cumulative Gain for measuring the effectiveness for their search engine model.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data
    § AI Model Quality Assurance · p. 33

    Participants A and P endorsed using traditional quality metrics for AI, such as F1 scores, precision and accuracy.

    “Participant A and P endorsed to use traditional quality metrics for AI, such as F1 scores, precision and accuracy.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data
    § Documentation · p. 38

    Participant T raised the research gap on software tools automatically handling and updating the documentation.

    “The participant raised the research gap on software tools automatically handling and updating the documentation.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingown data
    § Documentation · p. 38

    Documenting information about the development stage was not handled by any paper from the literature review but was mentioned by participant T.

    “Documenting information about the development stage was not handled by any paper from the literature review but was mentioned by participant T.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data
    § Documentation · p. 38

    Participant T proposed to document the quality assurance process and associated outcome and any additional information necessary for a software release.

    “He proposed to document the quality assurance process and associated outcome and any additional information necessary for a software release.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingcitation
    § Software-level Quality Assurance · p. 37

    The pipeline can automatically execute compatibility checks between interfaces and API endpoints, which is essential according to participant R when AI models are integrated as microservices.

    “The pipeline can automatically execute compatibility checks between interfaces and the Application Programming Interface (API) endpoints [I10, I49], which is essential according to participant R when AI models are integrated as microservices.”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites I10

      Not checked in the free run.

    • cites I49

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingcitation
    § Model Documentation · p. 36

    The MLR already established the information necessary in the model documentation.

    “The MLR however, already established the information necessary in the model documentation.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data
    § 4.3.3. Software Development · p. 37

    Three interviewees (participant R, C and V) confirm using Docker extensively.

    “In addition, three interviewees (participant R, C and V) confirm using Docker extensively.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data
    § Model Documentation · p. 35

    Seven out of nine participants identified model documentation as a good practice however they hardly ever create and maintain an up-to-date and consistent documentation.

    “However, seven out of nine participants identified model documentation as a good practice however they hardly ever create and maintain an up-to-date and consistent documentation.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingown data
    § Software-level Quality Assurance · p. 37

    Participants D and R agreed on the importance of software-level quality assurance, whereas participant C expressed concerns because the integration and respective testing are not model related and should be handled via a different CI/CD pipeline.

    “Participants D and R agreed on the importance of this type of quality assurance whereas participant C expressed some concerns because the integration and respective testing are not model related and should be handled via a different CI/CD pipeline.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingcitation
    § 4.3.3. Software Development · p. 37

    Research on other containerization approaches is sparse, however essential for increasing efficiency in cloud computing.

    “Research on other approaches is sparse, however essential for increasing efficiency in cloud computing [I11].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites I11

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingnone given
    § Deployment Strategy · p. 40

    Participant A uses these deployment strategies to identify potential network effects.

    “Participant A uses these deployment strategies to identify potential network effects.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S1 (asserted), S3 (load-bearing). May combine two claims.

    Findingown data
    § Monitoring · p. 40

    According to participant P, operational telemetry data is essential because although the model improves, the subjective user perception of the model may diminish.

    “According to participant P, operational telemetry data is essential because although the model improves, the subjective user perception of the model may diminish.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data
    § Deploy · p. 39

    According to the interview participants, cloud solutions are preferred.

    “According to the interview participants, also cloud solutions are preferred.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data
    § Deployment Strategy · p. 40

    If issues are encountered, rollbacks to already packaged models are necessary as identified during the interviews (R, A and T).

    “If issues are encountered, rollbacks to already packaged models are necessary as identified during the interviews (R, A and T).”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data
    § Monitoring · p. 40

    Participants A, P and T use Google Error Logs and Postman to identify the correct behaviour and performance of API requests.

    “To monitor the previously mentioned aspects, participants A, P and T use Google Error Logs and Postman to identify the correct behaviour and performance of API requests.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingnone given
    § 4.4.4. System Operations · p. 48

    Specific challenges during System Operations include ensuring that the model is compatible with the consuming application, cross-model inference and comparing models.

    “Specific challenges occurring during the stage System Operations elaborate challenges regarding the deployment, such as ensuring that the model is compatible with the consuming application, cross-model inference and comparing models.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingown data
    § 4.4.5. General Requirements for the Pipeline · p. 49

    Participant V uses TFX for AI development and discovered that infrastructure handling is very difficult for them and their customers.

    “Participant V uses TFX for their AI development and they discovered that the infrastructure handling is very difficult for them and for their customers.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Findingnone given
    § 4.4.1. Data Handling · p. 47

    The most often mentioned data stage issue is the data collection and integration with over 16 mentioned cases.

    “The most often mentioned data stage issue is the data collection and integration with over 16 mentioned cases.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingcitation
    § 4.4.2. Model Learning · p. 47

    Most often mentioned was that versioning should depict the model evolution and ensure reproducibility.

    “In addition, most often mentioned was that versioning should depict the model evolution and ensure reproducibility [I51, I58].”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites I51

      Not checked in the free run.

    • cites I58

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Findingnone given
    § 5. Discussion · p. 50

    The most often mentioned pipeline collected via the MLR is TFX.

    “The most often mentioned pipeline collected via the MLR is TFX.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingnone given
    § 4.4.4. System Operations · p. 48

    Monitoring related challenges focus on what aspects are worth monitoring and hidden feedback loops.

    “Monitoring related challenges focus on what aspects are worth monitoring and hidden feedback loops.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data
    § 4.4.5. General Requirements for the Pipeline · p. 49

    More than half of the participants (R, A, P, Z and C) confirmed the need for flexibility and customizability because pipelines are context-specific and strongly depend on the use case.

    “The need for flexibility and customizability were confirmed by more than half of the participants (R, A, P, Z and C) because pipelines are very context-specific and strongly depend on the use case.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingnone given
    § 4.4.2. Model Learning · p. 47

    Specific challenges during Model Learning include finding the optimal time to update a model to save time and resources, splitting the data set, testing updated models, potential overfitting, and introduced latency that slows down the pipeline.

    “Specific challenges occurring during the stage Model Learning elaborate challenges regarding finding the optimal time to update a model to save time and resources, model quality assurance faces the challenges to split the data set, test updated and the potential overfitting and the introduced latency that slows down the pipeline.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingown data
    § 4.4.5. General Requirements for the Pipeline · p. 49

    Participant V's customers wanted to execute the TFX pipeline themselves but did not have a Linux-based infrastructure.

    “Participant V, indicated that his customers wanted to execute the TFX pipeline by themselves but did not have an infrastructure based on Linux.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

    Findingnone given
    § 4.4.5. General Requirements for the Pipeline · p. 49

    The required multi-tenant setting, integration into the already existing security landscape, and missing regulations pose challenges.

    “The required multi-tenant setting, integration into the already existing security landscape, and missing regulations pose challenges as well.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data
    § 4.4.3. Software Development · p. 48

    Specific challenges during Software Development include packaging model and software independently to provide scalability, which was challenging for participant P.

    “Specific challenges occurring during the stage Software Development elaborate challenges regarding packaging model and software independently to provide scalability which was challenging for participant P.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingnone given
    § 4.4.5. General Requirements for the Pipeline · p. 49

    Elastic scaling is a pipeline-specific challenge because capacity changes.

    “Elastic scaling is also a pipeline specific challenge because capacity changes.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data
    § 4.4.1. Data Handling · p. 47

    Participants R and D stated that the pipeline should handle data governance and ownership constraints.

    “In addition, participants R and D stated that the pipeline should handle data governance and ownership constraints.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingnone given
    § 7. Conclusion · p. 57

    The Data Handling stage comprises the repetitive end-to-end lifecycle of data-related tasks, such as pre-processing, quality assurance, versioning, and documentation.

    “The stage Data Handling comprises the repetitive end-to-end lifecycle of data-related tasks, such as pre-processing, quality assurance, versioning, and documentation.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Findingnone given
    § 7. Conclusion · p. 57

    These triggers start the execution of the pipeline, which consists of four stages: Data Handling, Model Learning, Software Development, and System Operations.

    “These triggers start the execution of the pipeline , which consists of four stages: (1) Data Handling , (2) Model Learning , (3) Software Development and (4) System Operations .”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingnone given
    § 7. Conclusion · p. 57

    The final stage System Operations deploys the AI model to a specific environment via different deployment strategies and monitors the system.

    “The final stage System Operations deploys the AI model to a specific environment via different deployment strategies and monitors the system.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingnone given
    § 7. Conclusion · p. 57

    After the pipeline handles model learning, the Software Development stage prepares the model for deployment via packaging, software level quality assurance, and system versioning.

    “After the the pipeline handles the model learning, the stage Software Development prepares the model for deployment via packaging, software level quality assurance, and system versioning.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Findingnone given
    § 7. Conclusion · p. 57

    Software Development requires the pipeline to provide backward and forward compatibility of the model and data schema.

    “Software Development requires the pipeline to provide backward and forward compatibility of the model and data schema.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingnone given
    § 7. Conclusion · p. 57

    Challenges resulting from pipeline requirements, such as the need for a flexible, customizable, and scalable pipeline, are elaborated.

    “Ultimately challenges resulting from the pipeline requirements, such as the need for a flexible, customizable, and scalable pipeline are elaborated.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Findingnone given
    § 7. Conclusion · p. 57

    System Operations needs to handle multiple environments where cross-model inferences may occur.

    “The stage System Operations needs to handle multiple environments where cross-model inferences may occur.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

    Findingnone given
    § 7. Conclusion · p. 57

    Model Learning related challenges identify that versioning should depict the model evolution and ensure reproducibility.

    “Model Learning related challenges identify that versioning should depict the model evolution and ensure reproducibility.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Findingnone given
    § 7. Conclusion · p. 57

    Data Handling related challenges encompass data collection and integration.

    “Data Handling related challenges encompass data collection and integration.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Findingnone given
    § 7. Conclusion · p. 57

    The paper maps 25 potential challenges regarding the implementation, adaption, and usage of pipelines for the continuous development of AI to the four stages.

    “Furthermore, the paper maps 25 potential challenges regarding the implementation, adaption, and usage of pipelines for the continuous development of AI to the previously mentioned four stages.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingnone given
    § 7. Conclusion · p. 57

    The Model Learning stage uses the output of data handling and includes tasks associated with model development, such as model design, training, quality assurance, model improvement metadata capture, versioning, and documentation.

    “The stage Model Learning uses the output of the data handling and illustrates the tasks associated with the model development, such as model design, training, quality assurance, model improvement metadata capture, versioning, and documentation.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Study design and conductown data
    § Abstract · p. 1

    Nine semi-structured interviews with participants from academia and industry verified and extended the information obtained from the MLR.

    “In addition, nine-semi structured interviews with participants from academia and industry verified and extended the obtained information.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S2 (checkable). May combine two claims.

    Premisecitation
    § 1. Introduction · p. 3

    It is necessary to deploy and integrate AI models into production systems and assure quality of continuously evolving and self-adapting systems to harness the power of AI.

    “To harness the power of AI, it is necessary to deploy and integrate AI models into production systems and to assure the quality of the resulting continuously evolving and self-adapting systems [F24, 46, 47, 38].”
    Quote in the paperNot a substantial claim0 of 3 sources checked
    • cites F24

      Not checked in the free run.

    • cites [46] Stone, 2016, Artificial intelligence and life in 2030: One hundred year study on artificial intelligence: Report of the 2015–2016 study panel, Stanford University

      Not checked in the free run.

    • cites [47] Tao, 2019, Testing and quality validation for AI software–perspectives, issues, and practices, IEEE Access source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S2 (checkable). May combine two claims.

    Premisecitation
    § 1. Introduction · p. 3

    Quality assurance requires thorough attention to guarantee safe and reliable behaviour and increase accountability and responsibility of involved AI systems.

    “However, quality assurance requires thorough attention to guarantee safe and reliable behaviour and increase the accountability and responsibility of the involved AI systems [38, 47, 13, 23].”
    Quote in the paperNot a substantial claim0 of 3 sources checked
    • cites [38] Pieters, 2011, Explanation and trust: what to tell the user in security and AI?, Ethics Inform. Technol. source

      Not checked in the free run.

    • cites [47] Tao, 2019, Testing and quality validation for AI software–perspectives, issues, and practices, IEEE Access source

      Not checked in the free run.

    • cites [13] Hand, 2020, Validating and verifying AI systems, Patterns (New York, N.Y.)

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductreasoning
    § nation with a number. · p. 14

    We added additional categories via an iterative process because the predefined set of categories did not cover all tasks handled with the pipeline.

    “Because the predefined set of categories did not cover all tasks handled with the pipeline for the continuous development of AI, we added additional categories via an iterative process.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Study design and conductcitation
    § 3.3. Qualitative Analysis · p. 15

    We analysed the interviews according to Mayring with summarizing qualitative content analysis.

    “We analysed the interviews according to Mayring [28] with the summarizing qualitative content analysis .”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [28] Mayring, 2015, Qualitative Inhaltsanalyse : Grundlagen und Techniken

      Not checked in the free run.

  • Classified as probably not a substantial claim.

    Study design and conductcitation
    § 3.3. Qualitative Analysis · p. 15

    We conducted interviews because they allowed us to explore and understand individual experiences from a sample by outlining the complexity and diversity of the observed environment.

    “For this, we conducted interviews because they allowed us to explore and understand individual experiences from a sample by outlining the complexity and diversity of the observed environment [31].”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [31] Miles, 2014

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S2 (checkable).

    Premisecitation
    § 1. Introduction · p. 3

    These pipelines need more research when adapting them to AI models.

    “These pipelines are well established in traditional software development however need more research when adapting them to AI models because these pipelines not only need to handle code but also data and the model itself in addition to a large system-level complexity [7, I18].”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites [7] Fischer, 2020, AI system engineering key challenges and lessons learned, Mach. Learn. Knowl. Extraction source

      Not checked in the free run.

    • cites I18

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationown data
    § 4.1. Terminologies · p. 18

    All terms (DevOps for AI, CI/CD for AI, MLOps, lifecycle management, and CD4ML) share a common understanding and describe the automation for the continuous development and improvement of AI models via pipelines.

    “All terms share a common understanding and describe the automation for the continuous development and improvement of AI models via pipelines.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The literature identified by the MLR is representative of current research on the continuous development of AI.

  • Classified as probably not a substantial claim. May combine two claims.

    Study design and conductcitation
    § nation with a number. · p. 13

    We adopted quality assessment criteria to select sources based on relevance and to check whether informal literature search results are valid and free of bias.

    “We adopt quality assessment criteria to select the sources based on their relevance and to check whether the informal literature search results are valid and free of bias [10].”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [10] Garousi, 2016, The need for multivocal literature reviews in software engineering

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given
    § 3.3. Qualitative Analysis · p. 15

    We adopted the deductive category definition to extract information regarding the pipeline.

    “We adopted the deductive category definition to extract information regarding the pipeline.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductcitation
    § 3.3. Qualitative Analysis · p. 15

    We adopted a stratified sampling technique with three groups to select interview participants.

    “To select the interview participants, we adopt a stratified sampling technique [40] with the three groups illustrated in Table 5.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [40] Robinson, 2014, Sampling in interview-based qualitative research: A theoretical and practical guide, Qual. Res. Psychol. source

      Not checked in the free run.

  • Classified as probably not a substantial claim.

    Study design and conductown data
    § 3.3. Qualitative Analysis · p. 15

    We check via a qualitative approach if the derived information from the literature is correct and comprehensive enough to provide a thorough depiction of existing knowledge.

    “We check via a qualitative approach if the derived information from the literature is correct and comprehensive enough to provide a thorough depiction of existing knowledge.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Study design and conductnone given
    § 3.2. Taxonomy Creation · p. 15

    We defined units of classes/categories based on DevOps phases because they are commonly accepted within the field.

    “Firstly, we defined the units of the classes/categories which are based on DevOps phases because they are commonly accepted within the field.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductcitation
    § 3.1. MLR · p. 12

    We used Google Scholar as the search engine to retrieve an unbiased start data set.

    “We used Google Scholar as the search engine to retrieve an unbiased start data set.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [55] Yasin, 2020, On using grey literature and google scholar in systematic literature reviews in software engineering, IEEE Access source

      Not checked in the free run.

  • May combine two claims.

    Premisecitation
    § 1. Introduction · p. 3

    These pipelines need to handle code, data, the model itself, and large system-level complexity.

    “These pipelines are well established in traditional software development however need more research when adapting them to AI models because these pipelines not only need to handle code but also data and the model itself in addition to a large system-level complexity [7, I18].”
    Quote in the paperSubstantial claim0 of 2 sources checked
    • cites [7] Fischer, 2020, AI system engineering key challenges and lessons learned, Mach. Learn. Knowl. Extraction source

      Not checked in the free run.

    • cites I18

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductcitation
    § nation with a number. · p. 15

    We executed a test-retest process to evaluate the rater's data extraction consistency.

    “After selecting the source and extracting necessary categories, we executed a test-retest process proposed by [20] to evaluate the rater's data extraction consistency.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [20] Kitchenham, 2007

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § 4.1.1. DevOps for AI · p. 18

    DevOps for AI is seen as a standard for modern software development to ensure higher data and code quality.

    “It is seen as a standard for modern software development to ensure higher data, as well as code quality [F59, I8].”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites F59

      Not checked in the free run.

    • cites I8

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductcitation
    § nation with a number. · p. 14

    The extraction process and retrieved literature were documented in a systematic map available online.

    “The extraction process and the retrieved formal and informal literature were documented in a systematic map which is available online [44].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [44] Steidl, 2022

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § 4.1.1. DevOps for AI · p. 18

    DevOps for AI is used by some authors to describe methods for managing the software lifecycle.

    “Some authors use the term DevOps for AI to describe '[...] methods for managing the software lifecycle' [F59] as stated by Rausch et al.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites F59

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductcitation
    § 3.3. Qualitative Analysis · p. 15

    We unambiguously assign the participants' statements to the identified categories.

    “By doing so we unambiguously assign the participants' statements to the identified categories [28].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [28] Mayring, 2015, Qualitative Inhaltsanalyse : Grundlagen und Techniken

      Not checked in the free run.

  • Classified as probably not a substantial claim.

    Study design and conductreasoning
    § 3.2. Taxonomy Creation · p. 15

    We use a facet approach for classification structure because research on pipelines for continuous development of AI applications is still a new and evolving field.

    “For the classification structure type, we use a a facet approach because research on pipelines for the continuous development of AI applications is still a new and evolving field.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Study design and conductnone given
    § nation with a number. · p. 14

    We base the categories on a previous pilot study.

    “We base the categories on a previous pilot study.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § 4.1.3. MLOps · p. 20

    MLOps helps with the continuous, quick, seamless and reliable deployment of multiple AI versions which are deployed in a heterogeneous and distributed environment via infrastructure and tools.

    “Similar to DevOps for AI, MLOps helps with the continuous, quick, seamless and reliable deployment of multiple AI versions which are deployed in a heterogeneous and distributed environment via infrastructure and tools [F58, F14, F46, F24].”
    Quote in the paperNot a substantial claim0 of 3 sources checked
    • cites F58

      Not checked in the free run.

    • cites F14

      Not checked in the free run.

    • cites F46

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation
    § 4.1.3. MLOps · p. 20

    AI model quality is strongly dependent on the used data sets and the model is only a small part of the entire software system.

    “The AI model quality is strongly dependent on the used data sets and the model is only a small part of the entire software system [I69, I68, F44, I22, I39, I1, I44].”
    Quote in the paperNot a substantial claim0 of 3 sources checked
    • cites I69

      Not checked in the free run.

    • cites I68

      Not checked in the free run.

    • cites F44

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § 4.1.2. CI/CD for AI · p. 18

    CI/CD supports deployment to the infrastructure used to serve models in production.

    “According to Karlas et al. CI/CD supports the '[...] deployment to the infrastructure used to serve models in production' [F33].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites F33

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § 4.1.3. MLOps · p. 19

    MLOps applies DevOps principles to ML systems.

    “According to Google Cloud MLOps '[...] appl[ies] DevOps principles to ML systems' [I34]”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites I34

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § 4.1.3. MLOps · p. 19

    MLOps is an emerging method to fuse machine learning engineering with software development.

    “According to Raj et al. it is an '[...] emerging method to fuse machine learning engineering with software development' [F58].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites F58

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § 4.1.3. MLOps · p. 20

    MLOps can be divided into three different levels of maturity depending on the degree of automation.

    “According to the Google Cloud documentation [I34], MLOps can be divided into three different levels of maturity depending on the degree of automation.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites I34

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premiseown data
    § 4.1.3. MLOps · p. 20

    Microsoft defines five levels of technical implementation of MLOps.

    “Another definition by Microsoft was identified during the interviews where five levels of technical implementation of MLOps are defined 1 .”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation
    § 4.1.3. MLOps · p. 19

    MLOps is a set of practices that combines ML, DevOps and Data Engineering, which aims to deploy and maintain ML systems in production reliably and efficiently.

    “Breuel defines it as following: 'MLOps is a set of practices that combines ML, DevOps and Data Engineering, which aims to deploy and maintain ML systems in production reliably and efficiently' [I8].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites I8

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § 4.1.4. (End-to-End) Lifecycle Management · p. 20

    It is crucial to have correct values for the features that correspond to the timestamp of the labels.

    “For example, it is crucial for Brumbaugh et al. to '[...] have correct values for the features that correspond to the timestamp of the labels' [F16].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites F16

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § 4.1.2. CI/CD for AI · p. 18

    CI/CD handles (semi-)automatic and iterative validation and testing of data, data schemas, and models, in addition to validating and testing code and components.

    “Thus, CI/CD takes into consideration not only validating and testing code and components, but also handles the (semi-)automatic and iterative validation and testing of data, data schemas, and models [I34, F33].”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites I34

      Not checked in the free run.

    • cites F33

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § 4.1.3. MLOps · p. 20

    Continuous Training (CT) is concerned with automatically retraining and serving the models.

    “A new practice, called Continuous Training (CT) is introduced that according to Google Cloud '[...] is concerned with automatically retraining and serving the models' [I34].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites I34

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § 4.1.4. (End-to-End) Lifecycle Management · p. 20

    The term (end-to-end) lifecycle management includes the word management, which describes the handling of specific tasks of the continuous development of AI.

    “This term includes the word management , which describes the handling of specific tasks of the continuous development of AI [F70, I26].”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites F70

      Not checked in the free run.

    • cites I26

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Interpretationcitation
    § 4.1.3. MLOps · p. 19

    The key difference between MLOps and other terms is that MLOps strongly takes into consideration the company's culture and illustrates how cross-functional teams collaborate via a harmonized process.

    “The key difference between other terms is that MLOps strongly takes into consideration the company's culture and illustrates how cross-functional teams, such as data analysts, system operators, as well as data and software engineers collaborate via a harmonized process [F64, F54, F32, F46, I67, I59, I61, I27].”
    Quote in the paperNot a substantial claim0 of 3 sources checked
    • cites F64

      Not checked in the free run.

    • cites F54

      Not checked in the free run.

    • cites F32

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § 4.1.4. (End-to-End) Lifecycle Management · p. 20

    Model management tasks include tracking, storing and indexing large numbers of machine learning models so they may subsequently be shared, queried and analyzed.

    “Vartak et al. further specifies tasks included in the model management such as '[...] tracking, storing and indexing large numbers of machine learning models so they may subsequently be shared, queried and analyzed' [F70].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites F70

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

    Premisecitation
    § 4.1.4. (End-to-End) Lifecycle Management · p. 20

    The (end-to-end) lifecycle for AI models describes a well-fitted pipeline that should achieve the best possible quality and stability of the AI components via several iterations until the model cannot be improved any further.

    “Based on these tasks, the (end-to-end) lifecycle for AI models describes a well-fitted pipeline that should achieve the best possible quality and stability of the AI components via several iterations until the model cannot be improved any further [F33, F16, F49, F50].”
    Quote in the paperNot a substantial claim0 of 3 sources checked
    • cites F33

      Not checked in the free run.

    • cites F16

      Not checked in the free run.

    • cites F49

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § 4.1.1. DevOps for AI · p. 18

    DevOps for AI is a concept and agile practice to reduce time and resources between deployment iteration cycles.

    “DevOps for AI is a concept and agile practice to reduce time and resources between deployment iteration cycles [F59, F33, I8].”
    Quote in the paperNot a substantial claim0 of 3 sources checked
    • cites F59

      Not checked in the free run.

    • cites F33

      Not checked in the free run.

    • cites I8

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § 4.1.5. CD4ML · p. 21

    CD4ML is a technical implementation of the MLOps concept to automate the pipeline for the continuous development of AI.

    “CD4ML is a technical implementation of the MLOps concept to automate the pipeline for the continuous development of AI [F44, I18, I50, I49, I54].”
    Quote in the paperNot a substantial claim0 of 3 sources checked
    • cites F44

      Not checked in the free run.

    • cites I18

      Not checked in the free run.

    • cites I50

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § 4.1.4. (End-to-End) Lifecycle Management · p. 21

    A pipeline of tools automates the collection, preprocessing, cleaning and labelling of data.

    “According to Barrak et al. a '[...] pipeline of tools [...] automate[s] the collection, preprocessing, cleaning and labelling of data.' [F7]”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites F7

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § 4.1.5. CD4ML · p. 21

    CD4ML is defined as a software engineering approach in which a cross-functional team produces machine learning applications based on code, data, and models in small and safe increments, that can be reproduced and reliably released at any time in short adaptation cycles.

    “which defines CD4ML as '[...] a software engineering approach in which a cross-functional team produces machine learning applications based on code, data, and models in small and safe increments, that can be reproduced and reliably released at any time in short adaptation cycles' [I50].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites I50

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation
    § 4.2.1. Feedback and Alert Systems · p. 23

    To avoid triggering the pipeline continuously, triggers may occur periodically or when a specific threshold is attained.

    “To avoid triggering the pipeline continuously, triggers may occur periodically or when a specific threshold is attained [F2, F56, F15].”
    Quote in the paperNot a substantial claim0 of 3 sources checked
    • cites F2

      Not checked in the free run.

    • cites F56

      Not checked in the free run.

    • cites F15

      Not checked in the free run.

  • May combine two claims.

    Premisecitation
    § 4.2.1. Feedback and Alert Systems · p. 23

    Model updates may be triggered due to the deterioration of the model's performance, and scores in production below a specific threshold, also called model or concept drift.

    “Model updates may be triggered due to the deterioration of the model's performance, and scores in production below a specific threshold, also called model or concept drift [F45, F76, F30, F39, I45, I66].”
    Quote in the paperNot a substantial claim0 of 3 sources checked
    • cites F45

      Not checked in the free run.

    • cites F76

      Not checked in the free run.

    • cites F30

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § 4.2.1. Feedback and Alert Systems · p. 22

    Data updates may happen if the shape of the data, such as table or constraint definitions, changes due to schema updates based on software updates, requirement changes or migrations.

    “Data updates may also happen if the shape of the data, such as table or constraint definitions, may change due to schema updates based on software updates, requirement changes or migrations [I16].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites I16

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § 4.1.4. (End-to-End) Lifecycle Management · p. 21

    The term end-to-end lifecycle management uses well-established concepts from software development to cope with many model iterations, such as DevOps in combination with CI/CD and MLOps.

    “The term uses well-established concepts from software development to cope with many model iterations, such as DevOps in combination with CI/CD [F76, F5, I5] and MLOps [I39].”
    Quote in the paperNot a substantial claim0 of 3 sources checked
    • cites F76

      Not checked in the free run.

    • cites F5

      Not checked in the free run.

    • cites I5

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § 4.2.1. Feedback and Alert Systems · p. 22

    Data drifts occur when the distribution within the data set changes.

    “Data drifts occur when the distribution within the data set changes [I16, F71, F25, I13].”
    Quote in the paperNot a substantial claim0 of 3 sources checked
    • cites I16

      Not checked in the free run.

    • cites F71

      Not checked in the free run.

    • cites F25

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § 4.2. Triggers · p. 22

    Triggers may take into consideration the optimal threshold where the benefits obtained by an updated (i.e., retrained) model outweigh the effort involved in the retraining.

    “For example, triggers may take into consideration the optimal threshold where the benefits obtained by an updated (i.e., retrained) model outweigh the effort involved in the retraining [F9, I14, F35, F65, I43].”
    Quote in the paperNot a substantial claim0 of 3 sources checked
    • cites F9

      Not checked in the free run.

    • cites I14

      Not checked in the free run.

    • cites F35

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation
    § 4.2.1. Feedback and Alert Systems · p. 22

    Data drifts occur when data varies due to seasonal changes, or any other insertion, deletion or update of data values.

    “This occurs when data varies due to seasonal changes, or any other insertion, deletion or update of data values [I7, F45, F38, F9, F76].”
    Quote in the paperNot a substantial claim0 of 3 sources checked
    • cites I7

      Not checked in the free run.

    • cites F45

      Not checked in the free run.

    • cites F38

      Not checked in the free run.

  • May combine two claims.

    Premisecitation
    § 4.1.5. CD4ML · p. 21

    CD principles are used to span the AI lifecycle management and apply them to AI applications.

    “Therefore, CD principles are used to span the AI lifecycle management and apply them to AI applications [F44, I16, I65].”
    Quote in the paperNot a substantial claim0 of 3 sources checked
    • cites F44

      Not checked in the free run.

    • cites I16

      Not checked in the free run.

    • cites I65

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

    Premisereasoning
    § 4.1.4. (End-to-End) Lifecycle Management · p. 21

    The term end-to-end lifecycle management benefits from the idea of automation for the whole model lifecycle.

    “The term end-to-end lifecycle management benefits from the idea of automation for the whole model lifecycle.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § 4.1.4. (End-to-End) Lifecycle Management · p. 21

    In contrast to MLOps, lifecycle management does not focus on the interpersonal collaboration between different teams.

    “However, in contrast to MLOps, lifecycle management does not focus on the interpersonal collaboration between different teams [F48].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites F48

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § 4.2. Triggers · p. 22

    Triggers combining different approaches may also be feasible.

    “Thus, triggers combining different approaches may also be feasible [F59, F58].”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites F59

      Not checked in the free run.

    • cites F58

      Not checked in the free run.

  • May combine two claims.

    Premisecitation
    § 4.3. Pipeline · p. 26

    Feedback loops are a key characteristic in agile development and allow to include continuous feedback to improve the AI model and the collection of relevant data from production.

    “This is a key characteristic in agile development and allows to include continuous feedback to improve the AI model and the collection of relevant data from production [F2].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites F2

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § 4.3.1. Data Handling · p. 26

    The pipeline allows handling tasks more efficiently.

    “Not only allows the pipeline to handle tasks more efficiently, but also the quality of a AI model strongly depends on the data availability, quality, and preprocessing techniques [F2, F57, F6, I63].”
    Quote in the paperNot a substantial claim0 of 3 sources checked
    • cites F2

      Not checked in the free run.

    • cites F57

      Not checked in the free run.

    • cites F6

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § Data Preprocessing · p. 26

    Some data preprocessing steps may be skipped if the data set was already preprocessed in previous iterations.

    “Some steps may be skipped, if the data set was already preprocessed in previous iterations [I46].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites I46

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation
    § 4.2.1. Feedback and Alert Systems · p. 23

    Concept drift occurs when the problem the model was designed to solve changes, and this problem needs to be reformulated.

    “It occurs when the problem the model was designed to solve changes, and this problem needs to be reformulated [I63, I58, I46].”
    Quote in the paperNot a substantial claim0 of 3 sources checked
    • cites I63

      Not checked in the free run.

    • cites I58

      Not checked in the free run.

    • cites I46

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § 4.3. Pipeline · p. 26

    Ensuring quality is an integral part of the whole engineering procedure carried out in several steps such as data, model and system-specific tests.

    “Ensuring quality is an integral part of the whole engineering procedure carried out in several steps such as data, model and system-specific tests [F45, I32].”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites F45

      Not checked in the free run.

    • cites I32

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisenone given
    § 4.2.3. Repository · p. 23

    Repository updates are used as traditional triggers to guarantee that the latest changes are tested and available to the users.

    “Repository updates are used as traditional triggers to guarantee that the latest changes are tested and available to the users.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Premisecitation
    § 4.2.2. Orchestration Service and Schedule · p. 23

    Schedules help to optimize the retraining frequency, allocation of computing resources and execution order of pipeline jobs, which is especially important if edge and cloud resources are involved.

    “On the other hand, schedules help to optimize the retraining frequency, allocation of computing resources and execution order of pipeline jobs, which is especially important if edge and cloud resources are involved [F11, I34, F39, F60].”
    Quote in the paperNot a substantial claim0 of 3 sources checked
    • cites F11

      Not checked in the free run.

    • cites I34

      Not checked in the free run.

    • cites F39

      Not checked in the free run.

  • May combine two claims.

    Premisecitation
    § 4.2.2. Orchestration Service and Schedule · p. 23

    Additional triggers are an orchestration service or a scheduled time.

    “Additional triggers are an orchestration service or a scheduled time [F59, F16, I42].”
    Quote in the paperNot a substantial claim0 of 3 sources checked
    • cites F59

      Not checked in the free run.

    • cites F16

      Not checked in the free run.

    • cites I42

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § Data Preprocessing · p. 26

    The data preprocessing step is often defined in form of rules part of a script that defines how the raw data should be manipulated, transformed and compared.

    “This step is often defined in form of rules part of a script that defines how the raw data should be manipulated, transformed and compared [F45, F42, I63].”
    Quote in the paperNot a substantial claim0 of 3 sources checked
    • cites F45

      Not checked in the free run.

    • cites F42

      Not checked in the free run.

    • cites I63

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § Data Preprocessing · p. 26

    Initially, data is prepared for model design and training.

    “Initially, data is prepared for model design and training [I63].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites I63

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § 4.2.3. Repository · p. 23

    Pull requests to the repository as a commit or merge requests identify changes to the data, model or code.

    “For example, pull requests to the repository as a commit or merge requests identify changes to the data, model or code [I5, I54, I10, I13, I38, I55].”
    Quote in the paperNot a substantial claim0 of 3 sources checked
    • cites I5

      Not checked in the free run.

    • cites I54

      Not checked in the free run.

    • cites I10

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § 4.3.1. Data Handling · p. 26

    The data pipeline manipulates the initial data via intertwined tasks, such as data preprocessing, testing, versioning, and documentation, until the data can be used for model training.

    “The data pipeline manipulates the initial data via intertwined tasks, such as data preprocessing, testing, versioning, and documentation, until the data can be used for model training [F57].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites F57

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisereasoning
    § Data Preprocessing · p. 26

    Data collection including data injection, preparation, labelling and feature extraction needs to be executed to transform the raw data.

    “Therefore, data collection including data injection, preparation, labelling and feature extraction needs to be executed to transform the raw data.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Premisecitation
    § 4.3. Pipeline · p. 26

    Collecting data in production improves the training data set which ultimately increases model quality and efficacy.

    “For example, collecting data in production improves the training data set which ultimately increases model quality and efficacy [I29].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites I29

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § Data Preprocessing · p. 27

    Data can be collected from multiple distributed on-premise data centres, external public or private cloud storage.

    “Data can be collected from multiple distributed on-premise data centres, external public or private cloud storage [F64, F40, F6, F16, F57].”
    Quote in the paperNot a substantial claim0 of 3 sources checked
    • cites F64

      Not checked in the free run.

    • cites F40

      Not checked in the free run.

    • cites F6

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation
    § Data Preprocessing · p. 28

    For supervised learning, extracting relevant features guides the model training and identifies which features are worth exploring as input for the AI model.

    “For supervised learning, extracting relevant features guides the model training and identifies which features are worth exploring as input for the AI model [F33, F6, F31, F55].”
    Quote in the paperNot a substantial claim0 of 3 sources checked
    • cites F33

      Not checked in the free run.

    • cites F6

      Not checked in the free run.

    • cites F31

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation
    § Data Quality Assurance · p. 28

    Single-batch validation assumes that data batches collected in succession do not differ drastically and comply with a specific shape.

    “Single-batch validation assumes that data batches collected in succession do not differ drastically and comply with a specific shape [F15, F55].”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites F15

      Not checked in the free run.

    • cites F55

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § Data Preprocessing · p. 27

    Numeric data may be normalized via feature scaling.

    “In addition, numeric data may be normalized via feature scaling [I42, F57].”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites I42

      Not checked in the free run.

    • cites F57

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § Data Preprocessing · p. 28

    Automatic feature selection techniques include particle swarm optimization, recursive feature elimination, principal component analysis, and auto encoders.

    “Automatic feature selection techniques include, e.g., particle swarm optimization [54], recursive feature elimination, principal component analysis [17], and auto encoders [35].”
    Quote in the paperNot a substantial claim0 of 3 sources checked
    • cites [54] Xue, 2013, Particle swarm optimization for feature selection in classification: a multi-objective approach, IEEE Trans. Cybern. source

      Not checked in the free run.

    • cites [17] Jolliffe, 2005, Principal component analysis

      Not checked in the free run.

    • cites [35] Ng, 2011, Sparse autoencoder, CS294A Lecture Notes

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisereasoning
    § Data Preprocessing · p. 28

    Each record receives a meaningful ground truth label indicating the expected output.

    “Therefore, each record receives a meaningful ground truth label indicating the expected output.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § Data Preprocessing · p. 28

    Other learning techniques, such as reinforcement learning, use demonstrations as labelled data.

    “Other learning techniques, such as reinforcement learning, use demonstrations as labelled data [F2, F32].”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites F2

      Not checked in the free run.

    • cites F32

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation
    § Data Preprocessing · p. 27

    Data sets may be already available (e.g., open source or internally available) or need to be collected from multiple devices where the data may be stored in different formats, such as tabular data, logs, key-value stores or input files.

    “The data sets may be already available (e.g., open source or internally available) or needs to be collected from multiple devices where the data may be stored in different formats, such as tabular data, logs, key-value stores or input files [F2, F32, F16, F1, F33, F29, F55].”
    Quote in the paperNot a substantial claim0 of 3 sources checked
    • cites F2

      Not checked in the free run.

    • cites F32

      Not checked in the free run.

    • cites F16

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation
    § Data Preprocessing · p. 26

    Three authors proposed to use TFX to avoid the training-serving skew by exporting the tasks for data transformations that are again used in production and the training and serving pipeline does not need syncing.

    “Three authors proposed to use TFX, an end-to-end lifecycle management platform provided by Google, to avoid the training-serving skew by exporting the tasks for data transformations that are again used in production and the training and serving pipeline does not need syncing [F9, F56, F52, I70].”
    Quote in the paperNot a substantial claim0 of 3 sources checked
    • cites F9

      Not checked in the free run.

    • cites F56

      Not checked in the free run.

    • cites F52

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation
    § Data Preprocessing · p. 28

    Unsupervised algorithms do not require labels, thus the data labelling task can be skipped.

    “Unsupervised algorithms do not require labels, thus this task can be skipped [F57, F27].”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites F57

      Not checked in the free run.

    • cites F27

      Not checked in the free run.

  • May combine two claims.

    Premisecitation
    § Data Quality Assurance · p. 28

    The pipeline runs continuously, thus, reusable components for testing data are necessary to avoid introducing bugs in the data and propagating bugs down to the model training.

    “The pipeline runs continuously, thus, reusable components for testing data are necessary to avoid introducing bugs in the data and propagating bugs down to the model training [F9, I64, F15, F4].”
    Quote in the paperNot a substantial claim0 of 3 sources checked
    • cites F9

      Not checked in the free run.

    • cites I64

      Not checked in the free run.

    • cites F15

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation
    § Data Preprocessing · p. 27

    Pipelines may discard incomplete or irrelevant data or outliers and noisy records, anonymize data, aggregate data, or decompress and resize images, or filter and tokenize text.

    “For example, pipelines may discard incomplete or irrelevant data or outliers and noisy records [F58, F2, F32, F63, F51, I51], anonymize data [F51], (windowed/bucketed) aggregate data [F16], or decompress and resize images, of filter and tokenize text [F58, F64, F15].”
    Quote in the paperNot a substantial claim0 of 3 sources checked
    • cites F58

      Not checked in the free run.

    • cites F2

      Not checked in the free run.

    • cites F32

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing). May combine two claims.

    Premisereasoning
    § Data Preprocessing · p. 28

    The pipeline should provide consistent feature extractions of the offline and online inference environment, which may be achieved by a point-in-time correctness.

    “One important part the pipeline should provide consistent feature extractions of the offline and online inference environment, which may be achieved by a point-in-time correctness.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § Data Preprocessing · p. 28

    To automate data labelling, the lifecycle management platform ease.ml uses a model runner which identifies corresponding labels for input features.

    “To automate this process, the lifecycle management platform ease.ml for example uses a model runner which identifies corresponding labels for input features [F33].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites F33

      Not checked in the free run.

  • May combine two claims.

    Premisecitation
    § Data Preprocessing · p. 26

    Sometimes deviations in the data preprocessing pipelines are desirable because the data set and its size differ, persistent data stores provide the data for training whereas data in production is non-static where data needs to be processed fast.

    “However, sometimes deviations in the data preprocessing pipelines is desirable because data set and its size differs, persistent data stores provide the data for training whereas data in production is non-static where data needs to be processed fast [I31, I64].”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites I31

      Not checked in the free run.

    • cites I64

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisenone given
    § Data Quality Assurance · p. 28

    Data and feature validation validates batches of data.

    “Data and feature validation validates batches of data.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisenone given
    § Data Quality Assurance · p. 28

    Data batches can either be evaluated via a single-batch or inter-batch validation.

    “These batches can either be evaluated via a single-batch or inter-batch validation.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisereasoning
    § Data Quality Assurance · p. 28

    Statistical characteristics are discovered via calculating the distribution distance between training and serving data.

    “Statistical characteristics are discovered via calculating the distribution distance between training and serving data.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Premisereasoning
    § Data Versioning · p. 30

    Not only data and its dependencies but also the data processing steps need to be versioned.

    “Hence, not only data and its dependencies but also the data processing steps need to be versioned.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Premisereasoning
    § Data Versioning · p. 31

    DVC can store large files due to an external storage which can be combined with Git via a lightweight metadata file including a hash to indicate the data set version and data set location.

    “DVC can store large files due to an external storage which can be combined with Git via a lightweight metadata file including a hash to indicate the data set version and data set location.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § Data Quality Assurance · p. 28

    Inter-batch validation identifies differences between different batches, such as training and serving data or successive training data sets.

    “Inter-batch validation identifies differences between different batches, such as training and serving data or successive training data sets [F15].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites F15

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation
    § Data Quality Assurance · p. 29

    If ease.ml identifies the data set as insufficient, the user can clean up the data manually based on a list of dirty examples, or collect more data.

    “If ease.ml identifies the data set as insufficient, the user can clean up the data manually based on a list of dirty examples, or collect more data [F1, F61, F62, I44].”
    Quote in the paperNot a substantial claim0 of 3 sources checked
    • cites F1

      Not checked in the free run.

    • cites F61

      Not checked in the free run.

    • cites F62

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

    Premisenone given
    § Data Quality Assurance · p. 29

    Accuracy defines how well the data is suited and certified to execute a specific task.

    “Accuracy defines how well the data is suited and certified to execute a specific task.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisereasoning
    § Data Quality Assurance · p. 28

    A distance threshold is calculated to identify if the deviation results in an error.

    “Therefore, a distance threshold is calculated to identify if the deviation results in an error.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § Data Quality Assurance · p. 29

    The automatic feasibility analysis calculates a Bayes error based on already extracted features which are used for training.

    “It calculates a bayes error based on already extracted features which are used for training.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation
    § Data Quality Assurance · p. 28

    Information provided in a data schema includes expected type, presence, valency, and distribution of categorical values.

    “Information provided in this schema are the expected type, presence, valency [F9, F15, F18], and distribution of categorical values [F55, I7, I64].”
    Quote in the paperNot a substantial claim0 of 3 sources checked
    • cites F9

      Not checked in the free run.

    • cites F15

      Not checked in the free run.

    • cites F18

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisereasoning
    § Data Quality Assurance · p. 29

    The Bayes error is compared to a desired target performance, such as the accuracy.

    “Then, this error is compared to a desired target performance, such as the accuracy.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § Data Quality Assurance · p. 28

    The distance is calculated via two distributions including its probabilities.

    “The distance is calculated via two distributions including its probabilities [F18].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites F18

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisenone given
    § Data Quality Assurance · p. 29

    Consistency identifies the extent to which semantic rules of a data set are violated.

    “Consistency identifies the extend to which semantic rules of a data set are violated.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Premisenone given
    § Data Quality Assurance · p. 29

    The Bayes error estimator identifies the minimum error rate achievable by any classifier.

    “The Bayes error estimator identifies the minimum error rate achievable by any classifier.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § Data Quality Assurance · p. 30

    Unit tests identify if there are differences between the schema and the assumptions in the code.

    “These tests identify if there are differences between the schema and the assumptions in the code [F15].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites F15

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

    Premisenone given
    § Data Quality Assurance · p. 29

    Completeness identifies how well the used fraction of the data set represents the corresponding main data set or real-world data.

    “Completeness identifies how well the used fraction of the data set represents the corresponding main data set or real-world data.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § Data Quality Assurance · p. 28

    A stable description, such as a data schema, identifies expected features, feature types and values, and the correlation of different features.

    “Thus, a stable description, such as a data schema, identifies expected features, feature types and values, and the correlation of different features [F9, F56, F15, I48].”
    Quote in the paperNot a substantial claim0 of 3 sources checked
    • cites F9

      Not checked in the free run.

    • cites F56

      Not checked in the free run.

    • cites F15

      Not checked in the free run.

  • May combine two claims.

    Premisecitation
    § Data Versioning · p. 31

    Traditional version control systems, such as Git, cannot handle the amount of data, so new systems and tools, such as Data Version Control (DVC), have been introduced.

    “Because traditional version control systems, such as Git, cannot handle the amount of data [F7, F20, F30], new systems and tools, such as Data Version Control (DVC) have been introduced.”
    Quote in the paperNot a substantial claim0 of 3 sources checked
    • cites F7

      Not checked in the free run.

    • cites F20

      Not checked in the free run.

    • cites F30

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § Data Quality Assurance · p. 29

    Data can also be validated via a data quality measurement which identifies how suitable the data is to meet the users' requirements.

    “Data can also be validated via a data quality measurement which identifies how suitable the data is to meet the users' requirements [F4].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites F4

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § Data Quality Assurance · p. 30

    Unit tests can consist of tests that verify that the data ingestion works correctly.

    “Unit tests can consist of tests that verify that the data ingestion works correctly [F40].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites F40

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisenone given
    § Data Quality Assurance · p. 29

    Uniqueness states that no duplicates occur in the data set.

    “Uniqueness states that no duplicates occur in the data set.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § Data Quality Assurance · p. 29

    Data quality can be measured via the six dimensions of data quality, namely completeness, uniqueness, consistency, validity, accuracy and timeliness.

    “Data quality can be measured via the six dimensions of data quality, namely completeness, uniqueness, consistency, validity, accuracy and timeliness [F64, F2, F4, I44].”
    Quote in the paperNot a substantial claim0 of 3 sources checked
    • cites F64

      Not checked in the free run.

    • cites F2

      Not checked in the free run.

    • cites F4

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § Data Quality Assurance · p. 29

    Timeliness describes if the data set is up-to-date for a task.

    “Timeliness describes if the data set is up-to-date for a task [I44].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites I44

      Not checked in the free run.

  • May combine two claims.

    Premisecitation
    § Data Versioning · p. 30

    Data is stored via uniquely identified data snapshots or via reference to the original raw data set.

    “Data is stored via uniquely identified data snapshots or via reference to the original raw data set [F22, F33, F2, F20, F50].”
    Quote in the paperNot a substantial claim0 of 3 sources checked
    • cites F22

      Not checked in the free run.

    • cites F33

      Not checked in the free run.

    • cites F2

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § Data Versioning · p. 31

    The metadata file is then tracked via Git.

    “The metadata file is then tracked via Git [F7, I49, I37].”
    Quote in the paperNot a substantial claim0 of 3 sources checked
    • cites F7

      Not checked in the free run.

    • cites I49

      Not checked in the free run.

    • cites I37

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisereasoning
    § Data Quality Assurance · p. 29

    Another way of identifying if the available data is sufficient to meet the user requirements is an automatic feasibility analysis proposed by ease.ml.

    “Another way of identifying if the available data is sufficient to meet the user requirements is an automatic feasibility analysis proposed by ease.ml.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisenone given
    § Data Quality Assurance · p. 29

    Validity identifies if all data values comply with a specific format.

    “Validity identifies if all data values comply with a specific format.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Premisenone given
    § Data Quality Assurance · p. 28

    Other examples for calculating the statistical characteristics are Kullback-Leibler divergence, cosine similarity, or statistical goodness-of-fit tests.

    “Other examples for calculating the statistical characteristics are Kullback-Leibler divergence, cosine similarity, or statistical goodness-of-fit tests.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation
    § Data Quality Assurance · p. 30

    Unit tests can either state the input, execute a transformation and state the expected output or use data schemas.

    “Unit tests can either state the input, execute a transformation and state the expected output [I48] or use data schemas.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites I48

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § Data Versioning · p. 30

    If all snapshots cannot be stored due to space constraints, the delta in the data set can be versioned.

    “If all snapshots cannot be stored due to space constraints, the delta in the data set can be versioned [I32].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites I32

      Not checked in the free run.

  • May combine two claims.

    Premisecitation
    § Data Versioning · p. 30

    AI models require massive amounts of data for model training which needs to be stored and versioned to guarantee traceability and compliance with regulations, such as the General Data Protection Regulation.

    “AI models require massive amounts of data for model training which needs to be stored and versioned to guarantee traceability and compliance with regulations, such as the General Data Protection Regulation [F76, F69].”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites F76

      Not checked in the free run.

    • cites F69

      Not checked in the free run.

  • Classified as probably not a substantial claim. May combine two claims.

    Premisecitation
    § Design · p. 32

    The pipeline should support taking decisions regarding the model design, such as the appropriate hypothesis or regarding the selection of reusable model components which align with the problem domain.

    “Firstly, the pipeline should support taking decisions regarding the model design, such as the appropriate hypothesis [F47, I41] or regarding the selection of reusable model components which align with the problem domain [F27, F47, I64].”
    Quote in the paperSubstantial claim0 of 3 sources checked
    • cites F47

      Not checked in the free run.

    • cites I41

      Not checked in the free run.

    • cites F27

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based).

    Premisenone given
    § Data Documentation · p. 31

    Documentation should be extensive enough to support auditability of AI models.

    “Documentation should be extensive enough to support auditability of AI models.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable).

    Premisecitation
    § AI Model Quality Assurance · p. 32

    Fail-safe measures need to be introduced via relevant, sufficient and repeatable tests to cover all potential cases due to the continuous pipeline.

    “Thus, fail-safe measures need to be introduced via relevant, sufficient and repeatable tests to cover all potential cases due to the continuous pipeline [F23, F43, F23, F8]”
    Quote in the paperNot a substantial claim0 of 3 sources checked
    • cites F23

      Not checked in the free run.

    • cites F43

      Not checked in the free run.

    • cites F8

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § 4.3.2. Model Learning · p. 32

    According to Zhou et al., the Model Learning stage and its respective tasks are most essential throughout the pipeline for the continuous development of AI.

    “According to Zhou et al. [F76], this stage and its respective tasks are most essential throughout the pipeline for the continuous development of AI.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites F76

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Premisenone given
    § Data Versioning · p. 31

    Several space constraints occur with edge devices where data transfer costs should be avoided.

    “Several space constraints occur with edge devices where data transfer costs should be avoided.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Premisecitation
    § Data Documentation · p. 31

    Auditability for AI still is an emerging topic that is missing generally established mechanism and regulations.

    “Auditability for AI still is an emerging topic that is missing generally established mechanism and regulations [F14].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites F14

      Not checked in the free run.

  • Classified as probably not a substantial claim. May combine two claims.

    Premisecitation
    § AI Model Quality Assurance · p. 32

    The pipeline for continuous development of AI aims to provide faultless, reliable and secure software and to guarantee a trusted decision space.

    “The pipeline for the continuous development of AI aims to provide faultless, reliable and secure software and to guarantee a trusted decision space [F58, F40, F39].”
    Quote in the paperSubstantial claim0 of 3 sources checked
    • cites F58

      Not checked in the free run.

    • cites F40

      Not checked in the free run.

    • cites F39

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S2 (checkable).

    Premisecitation
    § Design · p. 32

    Pipelines should support context-specific decisions on the best suitable algorithm, feature selection, hyper-parameter setting, data set split, and potential reduction of the training data set.

    “Pipelines should support context-specific decisions on the best suitable algorithm, feature selection, hyper-parameter setting, data set split, and potential reduction of the training data set [F44, F2, F32, I64, I57].”
    Quote in the paperNot a substantial claim0 of 3 sources checked
    • cites F44

      Not checked in the free run.

    • cites F2

      Not checked in the free run.

    • cites F32

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). The quote only partly states this claim.

    Study design and conductown data
    § 3.3. Qualitative Analysis · p. 16

    The interview participants were selected using defined categories for stratified sampling across area of research, experience with AI, experience with lifecycle pipelines, and general experience.

    “Table 5: Defined categories for the stratified sampling”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based).

    Premisenone given
    § Data Documentation · p. 31

    Auditability defines a reviewing process where responsibilities and potential risks associated with the usage of an AI model are identified and root causes can be analyzed in case of a failure.

    “Auditability defines a reviewing process where responsibilities and potential risks associated with the usage of an AI model are identified and root causes can be analyzed in case of a failure.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing). May combine two claims.

    Premisereasoning
    § Data Versioning · p. 31

    Dependencies, data processing steps and extracted features should be versioned.

    “In addition, dependencies , data processing steps and extracted features should be versioned.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § AI Model Quality Assurance · p. 33

    The pipeline compares statistical evaluation metrics with metrics from previous model versions.

    “For instance, the pipeline compares statistical evaluation metrics with metrics from previous model versions [F51, F74, I47, F5, I48].”
    Quote in the paperNot a substantial claim0 of 3 sources checked
    • cites F51

      Not checked in the free run.

    • cites F74

      Not checked in the free run.

    • cites I47

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § AI Model Quality Assurance · p. 33

    Learnability is measured via five SCORE learnability facets: solution quality, convergence, overhead, robustness and effect.

    “Learnability is measured via five SCORE learnability facets, such as solution quality, convergence, overhead, robustness and effect.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites F26

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § Model Versioning · p. 35

    Granlund et al. stated that they cannot store model versions due to strict guidelines regarding the patient's data and missing isolated, in-house hardware resources.

    “In addition, Granlund et al. [I18] stated that they cannot store model versions due to strict guidelines regarding the patient's data and missing isolated, in-house hardware resources.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites I18

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § Model Versioning · p. 35

    Model dependencies capture the relationship to related elements, such as the associated data set, source code and configuration files.

    “Model dependencies capture the relationship to related elements, such as the associated data set, source code and configuration files.”
    Quote in the paperNot a substantial claim0 of 3 sources checked
    • cites F30

      Not checked in the free run.

    • cites F40

      Not checked in the free run.

    • cites F49

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § Metadata Capture · p. 34

    Feature extractors provided by several lifecycle management tools, such as TFX, automatically extract metadata.

    “Although feature extractors provided by several lifecycle management tools, such as TFX automatically extract metadata [F27, I64].”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites F27

      Not checked in the free run.

    • cites I64

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § Model Versioning · p. 35

    Model versioning captures version model artifacts and model dependencies to backtrack or reproduce different model versions that quickly evolve over time.

    “Model versioning not only captures version model artifacts but also model dependencies to backtrack or reproduce different model versions that quickly evolve over time [F70, F48, F49, F50, F37, I7, F53, I34, F25].”
    Quote in the paperNot a substantial claim0 of 3 sources checked
    • cites F70

      Not checked in the free run.

    • cites F48

      Not checked in the free run.

    • cites F49

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § Metadata Capture · p. 34

    Model specific metadata comprises relevant information on the execution and deployment of AI models, network architecture, training, instance metadata and runtime metadata.

    “For example, model specific metadata comprises relevant information on the execution and deployment of AI models, network architecture [F19], training [F40, F42, F19, F70], instance metadata and runtime metadata [I31, F28, F75, F19].”
    Quote in the paperNot a substantial claim0 of 3 sources checked
    • cites F19

      Not checked in the free run.

    • cites F40

      Not checked in the free run.

    • cites F42

      Not checked in the free run.

  • May combine two claims.

    Premisecitation
    § Model Versioning · p. 35

    AI model versioning is more complex and requires more storage capacities due to the continuous development, so standard version control systems, such as Git cannot be used as model repositories.

    “Because AI model versioning is more complex and requires more storage capacities due to the continuous development, standard version control systems, such as Git cannot be used as model repositories [I8].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites I8

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation
    § Metadata Capture · p. 34

    Metadata comprise provenance information which is necessary to govern the modelling lifecycle as well as to receive additional information on the data set and AI model.

    “Metadata comprise provenance information which is necessary to govern the modelling lifecycle as well as to receive additional information on the data set and AI model [F48, F27, F5].”
    Quote in the paperNot a substantial claim0 of 3 sources checked
    • cites F48

      Not checked in the free run.

    • cites F27

      Not checked in the free run.

    • cites F5

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § AI Model Quality Assurance · p. 33

    The ethics-by-design metric uses feature explainability approaches (LIME, PDP, SHAP) to identify whether and how much each feature contributes to the final prediction.

    “This metric uses approaches for feature explainability 2 , such as Local Interpretable Model Agnostic Explanations (LIME), Partial Dependency Plots (PDP), and SHapley Additive exPlanations (SHAP), to identify whether and how much each feature contributes to the final prediction [F14, F73, I49].”
    Quote in the paperNot a substantial claim0 of 3 sources checked
    • cites F14

      Not checked in the free run.

    • cites F73

      Not checked in the free run.

    • cites I49

      Not checked in the free run.

  • May combine two claims.

    Premisecitation
    § Model Versioning · p. 35

    Potential alternatives to standard version control systems are container registries where images are versioned or model repositories that store model versions including code, metadata, test results and dependencies.

    “Potential alternatives are container registries where images are versioned [I56] or model repositories that store model versions including code, metadata, test results and dependencies [F19, I59, I35].”
    Quote in the paperNot a substantial claim0 of 3 sources checked
    • cites I56

      Not checked in the free run.

    • cites F19

      Not checked in the free run.

    • cites I59

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S2 (checkable).

    Premisecitation
    § AI Model Quality Assurance · p. 33

    The financial technology sector requires fair models.

    “The financial technology sector requires fair models [I24, I1]”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites I24

      Not checked in the free run.

    • cites I1

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation
    § Metadata Capture · p. 34

    Further metadata includes information about the model's location, registration data, and information on the start and end data.

    “Further metadata includes information about the model's location, registration data, and information on the start and end data [F40, F42, F45, F58, F39, F16].”
    Quote in the paperNot a substantial claim0 of 3 sources checked
    • cites F40

      Not checked in the free run.

    • cites F42

      Not checked in the free run.

    • cites F45

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation
    § 4.3.3. Software Development · p. 36

    PMML describes statistical and data mining formats and transforms them into an XML configuration file.

    “Another possible format is PMML 4 , which describes statistical and data mining formats and transforms them into an Extensible Markup Language (XML) configuration file [F17, F74, I63].”
    Quote in the paperNot a substantial claim0 of 3 sources checked
    • cites F17

      Not checked in the free run.

    • cites F74

      Not checked in the free run.

    • cites I63

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § Versioning · p. 38

    The used pipeline version references the associated model versions or vice versa.

    “Moreover, the used pipeline version references the associated model versions or vice versa [F16].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites F16

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § Versioning · p. 38

    ModelDB stores the pipeline as a sequence of actions in a relational database.

    “Additionally, ModelDB stores the pipeline as a sequence of actions in a relational database [F70].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites F70

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § Software-level Quality Assurance · p. 37

    According to the paper 'Hidden Technical Debt in Machine Learning Systems', AI models represent only a small part of the whole system landscape where many data sources and software applications interact.

    “According to the paper 'Hidden Technical Debt in Machine Learning Systems' [42], AI models represent only a small part of the whole system landscape where many data sources and software applications interact [F9, I24, I18, F31].”
    Quote in the paperNot a substantial claim0 of 3 sources checked
    • cites [42] Sculley, 2015, Hidden technical debt in machine learning systems

      Not checked in the free run.

    • cites F9

      Not checked in the free run.

    • cites I24

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § Versioning · p. 38

    It is necessary to version the pipeline and its associated tasks.

    “In addition, it is also necessary to version the pipeline and its associated tasks [F28, F16, I59].”
    Quote in the paperNot a substantial claim0 of 3 sources checked
    • cites F28

      Not checked in the free run.

    • cites F16

      Not checked in the free run.

    • cites I59

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation
    § Software-level Quality Assurance · p. 38

    QFD collects customers' expectations and needs and generates a test coverage matrix where a support vector machine generates test cases.

    “This strategy collects customers' expectations and needs and generates a test coverage matrix where a support vector machine generates test cases [F23].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites F23

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § Versioning · p. 38

    TFX versions the pipeline as an artifact or source code regarding the implemented pipeline tasks.

    “For instance, TFX versions the pipeline as an artifact or source code regarding the implemented pipeline tasks [I2, I50]”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites I2

      Not checked in the free run.

    • cites I50

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation
    § Software-level Quality Assurance · p. 37

    Uber's end-to-end lifecycle management platform Michelangelo uses an internal benchmarking system to profile certain software parts, allowing measurement of how quickly inferences are run for a specific model based on real-life data.

    “For example, Uber's end-to-end lifecycle management platform Michelangelo uses an internal benchmarking system to profile certain software parts. This allows measuring how quickly inferences are run for a specific model based on real-life data [I19].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites I19

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisenone given
    § 4.3.3. Software Development · p. 36

    Potential formats established in the pipeline packaging task are ONNX, SavedModel, TensorRT, PMML, and PFA.

    “Potential formats established in the pipeline packaging task are ONNX, SavedModel, TensorRT, Predictive Model Markup Language (PMML), and Portable Format for Analytics (PFA).”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § 4.3.3. Software Development · p. 36

    PFA is a model interchange format that provides a safe execution environment for AI algorithms.

    “The PFA 5 is a model interchange format that provides a safe execution environment for AI algorithms [I63].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites I63

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § 4.3.3. Software Development · p. 36

    ONNX is an open, standardized format that represents AI models by using a common set of operators and file format to allow interoperability and serialization of models.

    “ONNX 3 is an open, standardized format built that represents AI models by using a common set of operators and file format to allow interoperability and serialization of models [I42, I63].”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites I42

      Not checked in the free run.

    • cites I63

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation
    § Software-level Quality Assurance · p. 37

    Integration tests check if different services work together correctly, or check whether the obtained model prediction is correctly transferred to the whole system by comparing this test to results of the model quality assurance.

    “For instance, integration tests check if different services work together correctly [I40], or check whether the obtained model prediction is correctly transferred to the whole system by comparing this test to results of the model quality assurance [F40, I33, I28].”
    Quote in the paperNot a substantial claim0 of 3 sources checked
    • cites I40

      Not checked in the free run.

    • cites F40

      Not checked in the free run.

    • cites I33

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § 4.3.3. Software Development · p. 36

    Several authors propose transforming the registered model into a systemindependent, deployable and hardware-optimized format.

    “Several authors propose transforming the registered model into a systemindependent, deployable and hardware-optimized format [F43, F75, F17].”
    Quote in the paperNot a substantial claim0 of 3 sources checked
    • cites F43

      Not checked in the free run.

    • cites F75

      Not checked in the free run.

    • cites F17

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § Versioning · p. 38

    GitHub repositories may store an Azure DevOps pipeline.

    “GitHub repositories may store an Azure DevOps pipeline [I46].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites I46

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § Model Documentation · p. 36

    Documented assumptions explain the reached decisions.

    “Additionally, documented assumptions explain the reached decisions [I21].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites I21

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation
    § Deployment Strategy · p. 40

    Examples of continuous experimentation are A/B tests, canary releases and shadow deployments.

    “Examples are A/B tests [F21, I56], canary releases [F9, F52] and shadow deployments [I51, I65, I49] [I32, I11, I14].”
    Quote in the paperNot a substantial claim0 of 3 sources checked
    • cites F21

      Not checked in the free run.

    • cites I56

      Not checked in the free run.

    • cites F9

      Not checked in the free run.

  • May combine two claims.

    Premisecitation
    § Monitoring · p. 40

    The pipeline monitors the AI model in production and collects necessary information to improve the non-deterministic model over time.

    “The pipeline also monitors the AI model in production and collects necessary information to improve the non-deterministic model over time [I59, I10, F36, I64, I17].”
    Quote in the paperNot a substantial claim0 of 3 sources checked
    • cites I59

      Not checked in the free run.

    • cites I10

      Not checked in the free run.

    • cites F36

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § Deploy · p. 39

    One approach to guarantee comparability is to use an unseen dataset.

    “One approach to guarantee comparability is to use an unseen dataset [F1].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites F1

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation
    § Monitoring · p. 40

    KPIs consist of response time, minimal latency and throughput or resource usage and network statistics, which are especially important if the AI application is deployed as a Service.

    “For instance, KPIs consist of response time, minimal latency and throughput [F40, F37, I40] or resource usage and network statistics, which are especially important if the AI application is deployed as a Service [F75, I7].”
    Quote in the paperNot a substantial claim0 of 3 sources checked
    • cites F40

      Not checked in the free run.

    • cites F37

      Not checked in the free run.

    • cites I40

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § Deployment Strategy · p. 40

    Continuous experimentation is a deployment strategy that allows gathering (user) feedback during runtime.

    “Continuous experimentation is a deployment strategy that allows gathering (user) feedback during runtime [F21, F6, F9].”
    Quote in the paperNot a substantial claim0 of 3 sources checked
    • cites F21

      Not checked in the free run.

    • cites F6

      Not checked in the free run.

    • cites F9

      Not checked in the free run.

  • Classified as probably not a substantial claim. May combine two claims.

    Premisecitation
    § Deploy · p. 39

    The pipeline needs to be aware of whether the validation uses the same data and if the evaluation remains the same.

    “Not only does the pipeline need to be aware of whether the validation uses the same data but also if the evaluation remains the same [F25, F34, F65].”
    Quote in the paperSubstantial claim0 of 3 sources checked
    • cites F25

      Not checked in the free run.

    • cites F34

      Not checked in the free run.

    • cites F65

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

    Interpretationnone given
    § 5. Discussion · p. 50

    The implemented pipeline may vary in practice due to resource constraints, difficulties in specific implementations, and because a task's benefit does not outweigh the costs.

    “the implemented pipeline may vary in practice. For instance, pipelines may vary due to resource constraints, difficulties in specific implementations and because a task's benefit does not outweigh the costs.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation
    § 4.4.2. Model Learning · p. 47

    Versioning is a static procedure, however, the pipeline needs to be capable of tracking constantly learning models and the conceptual coupling between the respective data sets, model and pipeline.

    “Versioning is a static procedure, however, the pipeline needs to be capable of tracking constantly learning models and the conceptual coupling between the respective data sets, model and pipeline [I49, F7, F5, F7, I50].”
    Quote in the paperNot a substantial claim0 of 3 sources checked
    • cites I49

      Not checked in the free run.

    • cites F7

      Not checked in the free run.

    • cites F5

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation
    § Environment and Infrastructure Handling · p. 41

    SageMaker expects a custom Docker image including predefined entry points, Watson Machine Learning requires a zip archive which includes the code for training a model.

    “For example, SageMaker expects a custom Docker image including predefined entry points, Watson Machine Learning requires a zip archive which includes the code for training a model.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites F28

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisenone given
    § 5.1. Data Handling · p. 52

    Transform converts ExampleGen's data set via SchemaGen's data schema using TensorFlow Transform.

    “Transform converts the ExampleGen's data set via SchemaGen's data schema using TensorFlow Transform.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisenone given
    § 5. Discussion · p. 50

    TFX combines different components to enable a flexible and customizable pipeline.

    “TFX combines different components to enable a flexible and customizable pipeline.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisereasoning
    § 5.1. Data Handling · p. 52

    SchemaGen uses these statistics to construct a schema automatically.

    “SchemaGen uses these statistics to construct a schema automatically.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisenone given
    § 5. Discussion · p. 50

    An orchestrator, such as Kubeflow, manages and triggers TFX's components.

    “An orchestrator, such as Kubeflow, manages and triggers these components.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisenone given
    § 5.2. Model Learning · p. 53

    Tuner requires the training and evaluation data set, tuning logic such as the model definition, hyperparameter search space, and Protocol buffers that include instructions for serializing structured data.

    “As input, Tuner requires the training and evaluation data set, tuning logic such as the model definition, hyperparameter search space and the Protocol buffers that include instructions for serializing structured data.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

    Premisecitation
    § 4.4.3. Software Development · p. 48

    The pipeline should also allow for backwards and forwards compatibility of a data schema.

    “The pipeline should also allow for for backwards and forwards compatibility of a data schema [I49, F65]”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites I49

      Not checked in the free run.

    • cites F65

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisenone given
    § 5.1. Data Handling · p. 52

    ExampleGen splits each version into training and evaluation data sets.

    “ExampleGen then splits each version into training and evaluation data sets.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Interpretationnone given
    § 5. Discussion · p. 50

    The execution order of tasks strongly depends on the specific context, such as organizational policies for running the pipeline.

    “In addition, the execution order of the tasks strongly depends on the specific context, such as organizational policies for running the pipeline.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation
    § 4.4.4. System Operations · p. 48

    Edge devices provide different hardware architecture, such as arm64X or x86 or sensor setups, thus the pipeline needs to provide personalized docker containers automatically tailored to the architecture.

    “For instance, edge devices provide different hardware architecture, such as arm64X or x86 or sensor setups, thus the pipeline needs to provide personalized docker containers automatically tailored to the architecture.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites I42

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

    Premisenone given
    § 4.4.5. General Requirements for the Pipeline · p. 49

    TFX requires Kubeflow, which only works on Linux.

    “For example, TFX requires Kubeflow which only works on Linux.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisereasoning
    § 5.1. Data Handling · p. 52

    ExampleGen handles file-based data, messaging, file-system-based data, databases, and others.

    “For instance, ExampleGen handles file-based data (e.g. CSV, Amazon S3, Google Cloud Storage, general text files), Messaging (e.g. Apache Kafka, Kinesis streams, JMS, Amazon Simple Queue Service), FileSystem-based data (e.g. Hadoop, Google Cloud Storage), Databases (e.g. Apache Cassandra, MongoDB, Google Cloud Datastore) and others.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisenone given
    § 5.2. Model Learning · p. 53

    TFX improves the model by tuning hyperparameters via its Tuner, which uses Python KerasTuner.

    “Regarding model learning, TFX improves the model by tuning hyperparameters via their Tuner that uses the Python KerasTuner.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisenone given
    § 5.1. Data Handling · p. 52

    Developers can modify and adapt the generated schema.

    “Developers can modify and adapt the generated schema.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisenone given
    § 5.1. Data Handling · p. 52

    Opposed to the proposed framework, Transform handles feature engineering after data validation.

    “Opposed to the proposed framework, Transform handles feature engineering after the data validation.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation
    § 5.1. Data Handling · p. 52

    As suggested in the MLR, ExampleValidator compares data statistics against the constructed schema, compares training and serving data to identify training-serving skews, and identifies data drifts by evaluating a series of data.

    “As suggested in the MLR, it compares the data statistics against the constructed schema, compares training and serving data to identify training-serving skews, and identifies data drifts by evaluating a series of data.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

    Premisecitation
    § 4.4.2. Model Learning · p. 47

    The pipeline's versioning should provide enough information to identify the reason for performance and prediction changes.

    “Additionally, the pipeline's versioning should provide enough information to identify the reason for performance and prediction changes [F73].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites F73

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § 4.4.1. Data Handling · p. 47

    The healthcare sector may require to keep the data sets within the organization.

    “For example, the healthcare sector may require to keep the data sets within the organization.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites I18

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § 4.4.4. System Operations · p. 48

    This should allow improving hardware stability.

    “This should allow improving hardware stability [I42].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites I42

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

    Premisereasoning
    § Environment and Infrastructure Handling · p. 41

    If multiple computing platforms use the same AI application, the pipeline should support cross-platform abstraction.

    “If for instance, multiple computing platforms use the same AI application, the pipeline should support cross-platform abstraction.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisenone given
    § 5.1. Data Handling · p. 52

    Feature transformations include embedding, vocabulary generation, value normalization, and enriching text features.

    “Feature transformations include embedding (mapping features to a low dimensional space), vocabulary generation (convert non-numeric features into integers), value normalization (scale numeric data without distorting differences in its range), and enriching text feature (extract features from raw data, e.g. n-grams).”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisereasoning
    § 5.1. Data Handling · p. 52

    With Apache Beam, ExampleGen can read different data formats and sources.

    “With the help of Apache Beam, a unified programming model for defining and executing data processing workflow, ExampleGen can read different data formats and sources.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisenone given
    § 5.1. Data Handling · p. 52

    TFX's ExampleGen transforms ingested data to customizable spans, versions and splits.

    “TFX's ExampleGen transforms the ingested data to customizable spans, versions and splits.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisenone given
    § 5.1. Data Handling · p. 52

    ExampleValidator uses the generated schema as input to validate the data set via TensorFlow Data Validation to ensure data quality.

    “ExampleValidator uses the generated schema as input to validate the data set via the TensorFlow Data Validation to ensure data quality.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisenone given
    § 5.1. Data Handling · p. 52

    StatisticsGen uses ExampleGen's output as input and calculates descriptive statistics for the data set.

    “StatisticsGen uses ExampleGen's output as input and calculates descriptive statistics for the data set.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationreasoning
    § Environment and Infrastructure Handling · p. 41

    Cross-platform abstraction allows abstracting the configuration management and low-level API calls.

    “This allows abstracting the configuration management and low-level API calls.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § Environment and Infrastructure Handling · p. 41

    The zip archive is then deployed in a container in the cloud.

    “This zip archive is then deployed in a container in the cloud [F28].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites F28

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation
    § Deploy · p. 39

    The deployment is an essential task to make the model available to others, enable collaboration and avoid knowledge silos.

    “The deployment is an essential task to make the model available to others, enable collaboration and avoid knowledge silos [F39, F69].”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites F39

      Not checked in the free run.

    • cites F69

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisenone given
    § 5.1. Data Handling · p. 52

    TFX's ExampleGen component stores consumed data from different file types in an appropriate format for following components.

    “TFX's ExampleGen component stores consumed data from different file types (e.g., csv) in an appropriate format for the following components.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisereasoning
    § 5.1. Data Handling · p. 52

    The schema includes information about data types, ranges, categories, distribution, etc.

    “The schema includes information about the (absence/presence of) data types, ranges, categories, distribution etc.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisenone given
    § 5.2. Model Learning · p. 53

    Opposed to the proposed framework, TFX firstly improves the model via hyperparameter tuning, then trains the model, and then validates it.

    “Opposed to the proposed framework, TFX firstly improves the model via hyperparameter tuning, then trains the model and then validates it.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductreasoning
    § 6. Threats to validity · p. 56

    Contrary findings were also included.

    “(5) Contrary findings were also included as well as”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Study design and conductown data
    § 6. Threats to validity · p. 56

    These interviews helped to evaluate and extend the results obtained from the MLR to provide a general depiction of tasks necessary for the continuous development of AI.

    “These interviews also helped to evaluate and extend the results obtained from the MLR to provide a general depiction of tasks necessary for the continuous development of AI.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given
    § 6. Threats to validity · p. 55

    The studies were assigned a 5-point Likert scale to identify the amount of contribution.

    “(2) the studies were assigned a 5-point Likert scale to identify the amount of contribution.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisereasoning
    § 5.4. System Operations · p. 54

    TFX uses an ML Metadata Store, a relational database that stores properties of trained models and the respective data set, evaluation results, execution records, and provenance of data objects.

    “Therefore, it uses a ML Metadata Store, a relational database that stores the properties of trained models and the respective data set, evaluation results, execution records and the provenance of data objects.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductnone given
    § 6. Threats to validity · p. 56

    Interview partners from industry can already use a continuous end-to-end lifecycle management pipeline for AI or are start-ups that develop AI applications and the associated lifecycle pipeline.

    “Interview partners from the industry can already use a continuous end-to-end lifecycle management pipeline for AI or are start-ups that develop AI applications and the associated lifecycle pipeline.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given
    § 6. Threats to validity · p. 55

    To improve the interviews' reliability of treatment implementation, the authors implemented and tested an interview plan.

    “To improve the interviews' reliability of treatment implementation, we implemented and tested an interview plan.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Premisenone given
    § 5.2. Model Learning · p. 53

    Standard Keras metrics calculate accuracy, precision, recall, etc.

    “Moreover, standard Keras metrics calculate the accuracy, precision, recall etc.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisenone given
    § 5.2. Model Learning · p. 53

    Trainer trains an AI model via Python's TensorFlow API.

    “Trainer trains an AI model via Python's TensorFlow API.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisenone given
    § 5.4. System Operations · p. 54

    TFX's InfraValidator checks whether the model complies with the production environment.

    “TFX'S InfraValidator checks whether the model complies with the production environment.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Premisereasoning
    § 5.4. System Operations · p. 54

    The metadata store allows identifying which features impacted the evaluation metrics.

    “Thus, the metadata store allows identifying which features impacted the evaluation metrics.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisereasoning
    § 5.2. Model Learning · p. 53

    Trainer uses ExampleGen's data set and transformed features to train two models.

    “This task uses ExampleGen's data set and the transformed features to train two models.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given
    § 6. Threats to validity · p. 55

    Internal validity may occur because only one researcher conducted the MLR.

    “Internal validity may occur because only one researcher conducted the MLR.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Study design and conductown data
    § 6. Threats to validity · p. 56

    Regarding external validity, the authors applied appropriate countermeasures, such as extensive search terms, including sources from academic studies as well as grey literature provided by industry, in addition to a two-stage selection process of the interview partners.

    “Regarding the external validity , we applied appropriate countermeasures, such as extensive search terms, including sources from academic studies as well as grey literature provided by industry, in addition to a two-stage selection process of the interview partners.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisenone given
    § 5.2. Model Learning · p. 53

    TFX's Evaluator performs analysis on training results and allows looking at the model's behaviour for individual slices of data sets.

    “TFX's Evaluator performs analysis on the training results and allows to look at the model's behaviour for individual slices of data sets.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Study design and conductnone given
    § 6. Threats to validity · p. 56

    The scope of the study is to cover a comprehensive framework for the continuous development of AI models.

    “The scope of this study is to cover a comprehensive framework for the continuous development of AI models.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisenone given
    § 5.4. System Operations · p. 54

    Depending on the deployment target, Pusher supports model repositories such as Tensorflow Hub, Javascript environments, native mobile applications, or server farms.

    “Depending on the deployment target Pusher supports model repositories such as Tensorflow Hub, Javascript environments (tensorflow.js), native mobile applications (TensorFlow Light), or server farms (Tensorflow Serving).”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Study design and conductown data
    § 6. Threats to validity · p. 55

    Some categories only had one supporting source during the test, and during the retest, another source was identified, which is a 100% increase from test to retest.

    “however, some categories only had one supporting source during the test, and during the retest, we identified another source which is a 100% increase from test to retest.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Study design and conductnone given
    § 6. Threats to validity · p. 55

    The selection process was rigorously documented.

    “(1) The selection process was rigorously documented”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Study design and conductreasoning
    § 6. Threats to validity · p. 55

    To minimize construct validity, the selected literature resources were carefully evaluated via two quality checklists proposed by Kitchenham and Charters [20] for formal literature and Garousi et al.'s checklist [11] for grey literature.

    “Thus, to minimize construct validity, the selected literature resources were carefully evaluated via two quality checklists proposed by Kitchenham and Charters' [20] for formal literature and Garousi et al.'s checklist [11] for grey literature.”
    Quote in the paperSubstantial claim0 of 2 sources checked
    • cites [20] Kitchenham, 2007

      Not checked in the free run.

    • cites [11] Garousi, 2019, Guidelines for including grey literature and conducting multivocal literature reviews in software engineering, Inf. Softw. Technol. source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given
    § 6. Threats to validity · p. 55

    The semi-structured approach allowed the authors to ask follow-up and clarification questions to reduce misunderstandings and ensure a thorough understanding.

    “In addition, the semi-structured approach allowed us to ask follow-up and clarification questions to reduce misunderstandings and ensure a thorough understanding.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

    Premisenone given
    § 5.3. Software Development · p. 53

    ML Metadata versions and documents software-specific information.

    “ML Metadata versions and documents software-specific information.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

    Premisenone given
    § 5.2. Model Learning · p. 53

    This ensures identifying well-performing models for the entire data set but poorly for a data point.

    “This ensures identifying well-performing models for the entire data set but poorly for a data point.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisenone given
    § 5.2. Model Learning · p. 53

    Evaluator allows comparing models to find the optimum model, which does not necessarily need to be the lastly trained model.

    “This task also allows comparing models to find the optimum model that does not necessarily need to be the lastly trained model.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Study design and conductown data
    § 6. Threats to validity · p. 55

    The random sample comprised 30 papers, where only two papers were excluded instead of included, resulting in a consistency of 93.3%.

    “The random sample comprised 30 papers where only two papers were excluded instead of included. This results in a consistency of 93,3%.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given
    § 6. Threats to validity · p. 56

    The categories comprise people from academia and industry.

    “The categories comprise people from academia and industry.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductreasoning
    § 6. Threats to validity · p. 56

    The target group consists of three different categories of interview partners to incorporate several points of view.

    “Regarding the two-stage selection process, the target group consists of three different categories of interview partners to incorporate several points of view.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisenone given
    § 5.4. System Operations · p. 54

    TFX versions all artifacts produced by every component over several executions.

    “TFX versions all the artifacts produced by every component over several executions.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisereasoning
    § 5.2. Model Learning · p. 53

    During model training, developers can simultaneously compare multiple model runs due to saved information in the ML Metadata Store.

    “In addition, during model training, developers can simultaneously compare multiple model runs, due to saved information in the ML Metadata Store.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Study design and conductown data
    § 6. Threats to validity · p. 55

    The authors calculated the test-retest value for categorizing the data of the included sources by extracting information from the 30 randomly selected sources again.

    “We calculate the Test-retests value for categorizing the data of the included sources. Therefore, we once more extracted the information from these 30 randomly selected sources.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Premisenone given
    § 5.3. Software Development · p. 53

    TFX does not explicitly handle any software development-specific tasks in its primary pipeline, such as packaging or system-level quality assurance.

    “TFX does not explicitly handle any software development-specific tasks in their primary pipeline, such as packaging or system-level quality assurance.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductreasoning
    § 6. Threats to validity · p. 56

    The described checklists identified unwarranted sources which were excluded.

    “(4) The described checklists identified unwarranted sources which were excluded.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Study design and conductcitation
    § 7. Conclusion · p. 57

    The authors extracted 151 relevant formal and informal sources via a Multivocal Literature Review (MLR) and executed nine semi-structured interviews.

    “To achieve this goal, we extracted 151 relevant formal and informal sources via a Multivocal Literature Review (MLR) and we executed nine semi-structured interviews [44].”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [44] Steidl, 2022

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisereasoning
    § 5.4. System Operations · p. 54

    The metadata store allows only rerunning necessary components, such as skipping data processing when the developer only adapts the model hyperparameters.

    “It also allows only rerunning necessary components, such as skipping data processing when the developer only adapts the model hyperparameters.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Study design and conductnone given
    § 6. Threats to validity · p. 55

    Conclusion validity is restricted because a qualitative approach was chosen.

    “Conclusion validity is restricted because a qualitative approach was chosen.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Study design and conductreasoning
    § 6. Threats to validity · p. 56

    The paper combines research on AI, ML and DL and does not differentiate between them because results from the MLR treated them similarly and only the implementation of tasks in data handling or model learning slightly varies.

    “Thus, this paper combines research on AI, ML and DL. The paper does not differentiate between them because results from the MLR treated them similarly and only the implementation of tasks in data handling or model learning slightly varies.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisenone given
    § 5.2. Model Learning · p. 53

    One model gets deployed to production for inference, and TFX uses one model for evaluation.

    “One model gets deployed to production for inference, and TFX uses one model for evaluation.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisenone given
    § 5.4. System Operations · p. 54

    After successfully evaluating the model concerning the specific production serving environment, TFX's Pusher pushes the validated model to the appropriate deployment environment.

    “After successfully evaluating the model concerning the specific production serving environment, TFX's Pusher pushes the validated model to the appropriate deployment environment.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisenone given
    § 5.2. Model Learning · p. 53

    TFX allows warm-starting training by using an existing model for further training.

    “TFX also allows warm-starting the training by using an existing model for further training.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given
    § 6. Threats to validity · p. 56

    The scope comprises different learning types, such as supervised, unsupervised and reinforcement learning.

    “In addition, the scope comprises different learning types, such as supervised, unsupervised and reinforcement learning.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductown data
    § 6. Threats to validity · p. 56

    The authors did not consider special cases of learning techniques, such as federated learning or transfer learning.

    “However, we did not consider special cases of learning techniques, such as federated learning or transfer learning.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisenone given
    § 5.4. System Operations · p. 54

    InfraValidator launches a sand-boxed model server based on manually configured environment details to evaluate compatibility between the model server binary and the model that should get deployed.

    “To do so, it launches a sand-boxed model server based on manually configured environment details, such as type of CPU, memory, or accelerators to evaluate the compatibility between the model server binary and the model that should get deployed.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Study design and conductnone given
    § 6. Threats to validity · p. 56

    The search strings allowed to include many relevant study findings.

    “(6) the search strings allowed to include many relevant study findings.”
    Quote in the paperNot a substantial claim
321 of 321

  • The paper relies on this without support.

    The nine interviewed participants are representative of current practice in the continuous development of AI.

    “Table 6: Overview and background of interviewed participants”
    “In four out of nine participants' organizations (A, B, C and V), the pipelines are not fully automated.”

  • The paper relies on this without support.

    The literature identified by the MLR is representative of current research on the continuous development of AI.

    “This section presents the information obtained from the literature and interviews.”
    “The MLR only extracted the z-transformation and Box-Cox transformations [F6] for data preparation but did not extract further information on the technical details, algorithms or implementations of the data preparation tasks.”

  • The paper relies on this without support.

    Documentation proves that the model adheres to the regulations and restrictions and works as expected.

    “Documentation proves that the model adheres to the regulations and restrictions and works as expected.”

  • The paper relies on this without support.

    Commonly, decisions made in the model creation need some rework.

    “Commonly, decisions made in the model creation need some rework.”

  • The paper relies on this without support.

    The continuity of the pipeline requires automated software-level quality assurance tests to frequently execute the tests and compare the results.

    “The continuity of the pipeline requires automated software-level quality assurance tests to frequently execute the tests and compare the results.”

  • The paper relies on this without support.

    AI applications may be deployed in several different environments or operational stages.

    “AI applications may be deployed in several different environments or operational stages.”
6 of 6

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Steidl et al. [22] reported a mixed-method investigation (i.e., SLR and a semi-structured review study) focusing specifically on continuous development pipelines of AI models, which is one aspect of MLOps practices.

    “Steidl et al. [22] reported a mixed-method investigation (i.e., SLR and a semi-structured review study) focusing specifically on continuous development pipelines of AI models, which is one aspect of MLOps practices.”

    From A Multivocal Review of MLOps Practices, Challenges and Open Issues (2026) · cites this paper as [22]

    The Abstract (p. 1) and Conclusion (p. 57) describe a study of continuous-development pipelines for AI models using a multivocal literature review and nine semi-structured interviews. This supports the topic and mixed-method characterization, but the claim misidentifies the methods as an SLR and a ‘semi-structured review study.’ (LLM, conf=0.88)

    Experimental — accuracy not yet measured

  • Steidl et al. [22] focused on analyzing definitions of various terms related to continuous development pipelines for AI, tasks handled by these pipelines, and the challenges related to implementation, adaption and usage of pipelines for continuous development of AI.

    “This work focused on analyzing definitions of various terms related to continuous development pipelines for AI such as DevOps for AI, CI/CD for AI, MLOPs, and so on, tasks handled by these pipelines, and the challenges related to implementation, adaption and usage of pipelines for continuous development of AI.”

    From A Multivocal Review of MLOps Practices, Challenges and Open Issues (2026) · cites this paper as [22]

The citation lists haven’t been fetched yet

References 178 · Cited by 126

Research works printed but not linked: 6

  1. Dellinger, 2005, Validity and the review of literature, Res. Schools, 12
  2. Figalist, 2020, An end-to-end framework for productive use of machine learning in software analytics and business intelligence solutions, vol. 12562, 217
  3. Ng, 2011, Sparse autoencoder, CS294A Lecture Notes, 72, 1
  4. Stone, 2016, Artificial intelligence and life in 2030: One hundred year study on artificial intelligence: Report of the 2015–2016 study panel, Stanford University
  5. Zaharia, 2018, Accelerating the machine learning lifecycle with mlflow, IEEE Data Eng. Bull., 41, 39
  6. Rosenbaum, Sasha, 2020. In: InfoQ (Ed.), CI/CD for Machine Learning. QCon, URL.

Authors
Steidl, M., Felderer, M., & Ramler, R.
Affiliations
  • Universität Innsbruck (AT)
  • Software Competence Center Hagenberg (Austria) (AT)
Corresponding author
Steidl, Monika
ORCID
Venue
Journal of Systems and Software
Publisher
Elsevier BV
Published
2023-05
Volume
199
Pages
111615
ISSN
0164-1212 (ISSN-L)
DOI
10.1016/j.jss.2023.111615
OpenAlex
W4316126954
Cited by
126
References
178
Citations by year
2026: 48 · 2025: 48 · 2024: 20 · 2023: 10
FWCI
29.52
Type
journal article
Language
en
Open access
Licence
Locations
  • Journal of Systems and Software (publishedVersion, cc-by)
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • KTH Publication Database DiVA (KTH Royal Institute of Technology) (submittedVersion, cc-by)
  • arXiv (Cornell University) (cc-by)
Funders
Österreichische Forschungsförderungsgesellschaft
Topics
  • Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
  • Physical Sciences › Computer Science › Information Systems › Software Engineering Research
  • Physical Sciences › Computer Science › Computer Networks and Communications › Software System Performance and Reliability
Keywords
AI pipelines, MLOps, continuous integration, data handling, model learning, software development, system operation, taxonomy (general classification scheme)

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-10.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-10.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review