CoChat
Sign Up
Cited by 34Software Engineering Techniques and PracticesSoftware Engineering ResearchSoftware System Performance and Reliability

Operationalizing machine learning models: a systematic literature review

Open access
Kolltveit, Ask Berstad, Li, Jingyue·2022·Proceedings of the 1st Workshop on Software Engineering for Responsible AI
DOI 10.1145/3526073.3527584
Open this paper in CoChat — free accountOpen PDF

Abstract

Deploying machine learning (ML) models to production with the same level of rigor and automation as traditional software systems has shown itself to be a non-trivial task, requiring extra care and infrastructure to deal with the additional challenges. Although many studies focus on adapting ML software engineering (SE) approaches and techniques, few studies have summarized the status and challenges of operationalizing ML models. Model operationalization encompasses all steps after model training and evaluation, including packaging the model in a format appropriate for deployment, publishing to a model registry or storage, integrating the model into a broader software system, serving, and monitoring. This study is the first systematic literature review investigating the techniques, tools, and infrastructures to operationalize ML models. After reviewing 24 primary studies, the results show that there are a number of tools for most use cases to operationalize ML models and cloud deployment in particular. The review also revealed several research opportunities, such as dynamic model-switching, continuous model-monitoring, and efficient edge ML deployments.

AI snapshot

Written from the paper’s full text.

TL;DR

This systematic literature review examines how machine learning models are operationalized after training, focusing on techniques, tools, infrastructure, and feature gaps. After reviewing 24 primary studies, it finds many tools for most operationalization use cases, especially cloud deployment, while identifying research opportunities in dynamic model-switching, continuous model-monitoring, and efficient edge ML deployments.

Key findings

  • The review analyzed 24 primary studies and found that there are a number of tools for most use cases to operationalize ML models, with cloud deployment in particular being well supported.
  • Reported operationalization techniques include packaging models in containers, serializing models (e.g., ONNX, HDF5, Joblib), deploying via push/pull patterns, serving through REST or RPC APIs, and monitoring runtime performance and predictions.
  • Main challenges cluster around serving performance, edge deployment, and monitoring/logging; for example, [44] observed that 33% of application latency resulted from internal communication frameworks, and [43] observed Kubernetes scheduler performance degrading with 5,000 nodes and fewer than 10 constraints, struggling to process more than 15 functions per second.
  • Research gaps include automatic model-switching to meet service level objectives, tools to monitor data and model drift, and efficient solutions for deploying ML models to edge devices.

Methods

  • The authors conducted a systematic literature review following guidelines from Kitchenham and Charters and Wohlin, using snowballing after string-based searches in Oria failed to produce sufficiently relevant articles.
  • From a starting set of 19 papers, 8 satisfied selection criteria and minimum quality score; backward snowballing yielded 21 candidates with 6 selected, and forward snowballing yielded 29 candidates with 10 selected, totaling 24 primary studies.
  • Data extraction used a spreadsheet, and thematic analysis was performed with TreeSheets, reorganizing data from author-oriented to code-oriented hierarchies to identify commonalities and differences.

Limitations

  • The paper states that snowballing was limited to a single iteration of forward and backward snowballing instead of continuing until no new studies were found, so the review is quite probably not exhaustive.
  • The paper states that qualitative data analysis is closely tied to the researchers’ background, identity, assumptions, and beliefs, which was addressed by discussing results between authors to reach consensus when needed.
  • The paper states that the initial string-based search possibly failed because too many aspects were attempted in just a few keywords, and that the study potentially misses practitioner knowledge, with a GLR suggested as a natural next step.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1145/3526073.3527584

Cites

40 of 40 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

0 of 161 cited sources checked · 0 supported · 0 flagged

  • Findingnone given

    Only a few of the reviewed studies focused on operationalization aspects of ML models.

    “Only a few of the studies reviewed, e.g., [4,23,33,38,41], focused on the operationalization aspects of ML models.”
    Quote in the paperSubstantial claim
  • Findingreasoning

    The authors of [24] did not study tools and infrastructures for operationalizing ML models.

    “However, the authors of [24] did not study tools and infrastructures for operationalizing ML models.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [24] source

      Not checked in the free run.

  • Findingown data

    The articles found by string-based searches did not report deployment details related to the operationalization aspect of SE for ML.

    “The articles found did not report deployment details related to the operationalization aspect of SE for ML, with which this study is concerned.”
    Quote in the paperSubstantial claim
  • Findingnone given

    None of the reviewed studies focused on related tooling and infrastructure issues.

    “However, none of the studies focused on related tooling and infrastructure issues.”
    Quote in the paperSubstantial claim
  • Findingown data

    String-based searches in Oria failed to produce sufficiently relevant articles on ML deployment/MLOps deployment.

    “However, our string-based searches (using strings such as “machine learning deployment, ” “MLOps deployment, ” etc.) in Oria [1], a search engine aggregating research papers from scientific databases, including IEEE Xplore, Springer, ACM Digital library, and Scopus, in late September 2021 failed to produce sufficiently relevant articles on the topic.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [1]

      Not checked in the free run.

  • Classified as probably not a substantial claim. 69
  • May combine two claims. 34

  • Probably not a substantial claim: may not pass S4 (based).

    Findingnone given

    Earlier studies have focused mostly on mapping out general SE challenges and practices for ML.

    “Earlier studies have focused mostly on mapping out general SE challenges and practices for ML.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Findingown data

    The review revealed research opportunities in dynamic model-switching, continuous model-monitoring, and efficient edge ML deployments.

    “The review also revealed several research opportunities, such as dynamic model-switching, continuous model-monitoring, and efficient edge ML deployments.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingcitation

    Predictions are logged, and when available joined with actual outcomes.

    “Predictions are logged, and when available joined with actual outcomes [28].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [28]

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

    Findingnone given

    Serialized models may simply be loaded into memory by the application, potentially leading to downtime.

    “On the other hand, serialized models may simply be loaded into memory by the application, potentially leading to downtime.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingcitation

    Models intended for batch predictions may be plugged into a computing pipeline such as Apache Spark, computing predictions and storing them in a database or data warehouse.

    “Models intended for batch predictions may simply be plugged into a computing pipeline such as Apache Spark 8 , computing predictions and storing them in a database or data warehouse [28].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [28]

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingcitation

    [12] noted several challenges in deploying ML frameworks: interface inconsistency across frameworks, changes over time in what is the best framework for the job, and frameworks that are not optimized for deployment.

    “[12] noted several challenges in deploying ML frameworks: interface inconsistency across frameworks, changes over time in what is the best framework for the job, and frameworks that are not optimized for deployment.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [12]

      Not checked in the free run.

  • May combine two claims.

    Findingcitation

    Runtime monitoring is important for operationalized ML in order to increase trust and detect performance degradations.

    “Runtime monitoring is important for operationalized ML in order to increase trust and detect performance degradations [28,42].”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites [28]

      Not checked in the free run.

    • cites [42] source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingcitation

    Model warmup can be achieved either through a method provided by the ML framework or through issuing an empty query against the model.

    “Model warmup can be achieved either through a method provided by the ML framework [28] or more generally through issuing an empty query against the model [16].”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites [28]

      Not checked in the free run.

    • cites [16] source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingcitation

    Model accuracy should be used to determine when a new model is needed.

    “Model accuracy should be used to determine when a new model is needed [39].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [39] source

      Not checked in the free run.

  • May combine two claims.

    Findingnone given

    Models may be deployed in a target environment different from where they are packaged, requiring a model transfer through either a push or pull pattern.

    “However, models may be deployed in a target environment that is different from where they are packaged, in which case a model transfer must occur, which may happen through either a push-or a pull-pattern.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingcitation

    If the hosting application is containerized, model redeployment may be orchestrated by deploying an entirely new application instance with the new model data alongside the old instance; when the new instance is initialized, it can start serving requests and the old instance may be evicted.

    “If the hosting application is containerized, model redeployment may be orchestrated by deploying an entirely new application instance with the new model data alongside the old instance.When the new application instance is finished initializing, it can start serving requests, and the old instance may be evicted [40].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [40] source

      Not checked in the free run.

  • May combine two claims.

    Findingcitation

    The model may be integrated directly into the application code, traditionally first having to be rewritten for production.

    “The model may be integrated directly into the application code [19,29,45], traditionally first having to be rewritten for production [21].”
    Quote in the paperNot a substantial claim0 of 4 sources checked
    • cites [19] source

      Not checked in the free run.

    • cites [29] source

      Not checked in the free run.

    • cites [45] source

      Not checked in the free run.

    • cites [21] source

      Not checked in the free run.

  • May combine two claims.

    Findingcitation

    ML frameworks are not all created equal, and some may be better suited for research than production or vice versa, typically because of performance characteristics versus support for rapid experimentation and debugging.

    “ML frameworks are not all created equal, and some may be better suited for research than production or vice versa, typically because of performance characteristics versus support for rapid experimentation and debugging.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingcitation

    Another reported technique is using adaptive batching queues with a timeout, so queries are batched together in batch sizes tuned to the individual model and framework.

    “Another is using adaptive batching queues with a timeout, so that queries are batched together in batch sizes that are tuned to the individual model and framework.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [12]

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing). May combine two claims.

    Findingcitation

    To avoid cold-start issues caused by model loading and initialization, models should be warmed up at deployment and kept perpetually warm over their lives.

    “In order to avoid cold-start issues caused by model loading and initialization, models should be warmed up at deployment and be kept perpetually warm over the course of their lives [54].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [54]

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingcitation

    In a pull-pattern deployment, the target environment periodically polls for model updates and downloads when available.

    “In a pull-pattern deployment, the target environment (host application running on, e.g., server or edge device) periodically polls for model updates and downloads when available [28,39,40].”
    Quote in the paperNot a substantial claim0 of 3 sources checked
    • cites [28]

      Not checked in the free run.

    • cites [39] source

      Not checked in the free run.

    • cites [40] source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

    Findingcitation

    The application should be validated against predefined key performance indicators of the project.

    “The application should be validated against predefined key performance indicators of the project [45].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [45] source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingcitation

    A cache layer may be put on top of the model to reduce computation.

    “Additionally, a cache layer may be put on top of the model to reduce computation [12].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [12]

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingcitation

    The model could also be packaged in a format specific to an end-to-end framework such as MLflow.

    “The model could also be packaged in a format specific to an end-to-end framework such as MLflowfoot_4 [10].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [10] source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingcitation

    To avoid high response latencies for the first query, models may be queried with an empty request or an explicit warmup function call to force lazy-loaded model components to initialize.

    “To avoid high response latencies for the first query (cold-start issues), models may be queried with an empty request (or an explicit warmup function call may be used if available) which forces lazy-loaded model components to initialize [28].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [28]

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingcitation

    When using an ML deployment service such as SageMaker, the service may handle the deployment of serialized models.

    “When using an ML deployment service (e.g., SageMaker 7 ), the service may handle the deployment of serialized models [9].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [9] source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Findingnone given

    For models packaged as standalone containers, a container orchestration service can roll out updated models without downtime.

    “For models packaged as standalone containers, a container orchestration service (e.g., Kubernetesfoot_5 ) can roll out updated models without downtime.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingcitation

    Monitoring ML models is challenging but is required in order to detect performance degradations and build confidence in the models.

    “Monitoring ML models is challenging [6,28,41], but is required in order to detect performance degradations and build confidence in the models [7,10].”
    Quote in the paperNot a substantial claim0 of 5 sources checked
    • cites [6] source

      Not checked in the free run.

    • cites [28]

      Not checked in the free run.

    • cites [41] source

      Not checked in the free run.

    • cites [7] source

      Not checked in the free run.

    • cites [10] source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingcitation

    [7] reported the extensive use of glue code as a challenge with which companies struggle.

    “[7] reported the extensive use of glue code as a challenge with which companies struggle.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [7] source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingcitation

    Data pre- and post-processing code in streaming ML systems could soon become another performance bottleneck.

    “[44] warns that the data pre-and post-processing code in streaming ML systems could soon become another performance bottleneck.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [44] source

      Not checked in the free run.

    • Rests on an assumption: ML models typically do not make inferences based on raw data but may require data transformations both before and after inference.

  • Classified as probably not a substantial claim. May combine two claims.

    Findingcitation

    Currently, the options for meeting performance SLOs during high-demand scenarios are either to accept degraded throughput or latency, or to scale up hardware resources and incur increased costs.

    “The authors noted that currently, the options are to either accept degraded throughput or latency during high-demand scenarios, or to scale up the hardware resources (as suggested by [6]) and thereby incurring increased costs.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [6] source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based).

    Findingnone given

    Edge deployment brings a whole host of its own challenges.

    “However, edge deployment brings a whole host of its own challenges, which will be discussed in Subsubsection 5.2.3.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingcitation

    The integration of data-driven models with computation-driven software components is a potentially non-trivial task.

    “The authors also highlight the integration of data-driven models with computationdriven software components as a potentially non-trivial task.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [7] source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingcitation

    Increasing the number of broker node instances used in the messaging system clears up the congestion issue.

    “The authors found that a solution to the problem could be to increase the number of broker node instances used in the messaging system, which clears up the congestion issue.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [44] source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingcitation

    Model-switching was proposed to solve the problem of upholding SLOs while minimizing costs.

    “The model-switching technique was proposed by [54] to solve the problem of upholding SLOs while minimizing costs.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [54]

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingcitation

    [12] used decoupling as a solution, but opted for adding an abstraction layer with a simple interface on top of all models; this solution is more general, as it is not specific to neural networks.

    “[12] also used decoupling as a solution to this problem, but opted for adding an abstraction layer with a simple interface on top of all models.This solution is more general, as it is not specific to neural networks.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [12]

      Not checked in the free run.

  • May combine two claims.

    Findingcitation

    Serving-related challenges are some of the ones most often reported in the literature, usually from a performance perspective.

    “Serving-related challenges are some of the ones most often reported in the literature [5-7, 9, 11, 12, 16, 20, 28, 34, 44, 53, 54], usually from a performance perspective.”
    Quote in the paperNot a substantial claim0 of 10 sources checked
    • cites [5] source

      Not checked in the free run.

    • cites [6] source

      Not checked in the free run.

    • cites [7] source

      Not checked in the free run.

    • cites [9] source

      Not checked in the free run.

    • cites [11] source

      Not checked in the free run.

    • cites [12]

      Not checked in the free run.

    • cites [16] source

      Not checked in the free run.

    • cites [20] source

      Not checked in the free run.

    • cites [28]

      Not checked in the free run.

    • cites [34] source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingnone given

    There are problems with the solution of keeping models perpetually warm.

    “There are, however, problems with this solution, which will be discussed in Subsection 5.4.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingcitation

    The messaging system becomes congested when faced with an eightfold increase in inference performance.

    “by experimentation, found that the messaging system becomes congested when faced with an eightfold increase in inference performance.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [44] source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingcitation

    In model-switching, high-accuracy/low-performance models are traded for lower-accuracy/high-performance models during load spikes to meet the effective accuracy SLO.

    “In this technique, high-accuracy/low-performance models are traded for lower-accuracy/high-performance models during load spikes in order to meet the effective accuracy SLO, i.e., the average accuracy of predictions that meet latency requirements.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [54]

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingcitation

    In a system utilizing publish/subscribe messaging services, 33% of application latency is a result of internal communication frameworks.

    “In a system utilizing publish/subscribe messaging services, [44] observed that 33% of application latency is a result of internal communication frameworks, and by experimentation, found that the messaging system becomes congested when faced with an eightfold increase in inference performance.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [44] source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingcitation

    Increasing hardware instances and network bandwidth both improve inference latencies.

    “The difficulty in achieving cost-effective and performant inference was also corroborated by [9], in which the authors observed that increasing hardware instances and network bandwidth both improve inference latencies.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [9] source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingcitation

    Deploying ML models to the edge can reduce network latency.

    “[20] suggested deploying ML models to the edge in order to reduce network latency.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [20] source

      Not checked in the free run.

  • May combine two claims.

    Findingcitation

    Apache Kafka is a commonly reported messaging and communications framework, used for logging predictions, notifying model hosts of updates, and general inter-container communication.

    “A commonly reported messaging and communications framework is Apache Kafka 28 , used for logging predictions [28], notifying model hosts of updates [16,29] and general inter-container communication [44].”
    Quote in the paperNot a substantial claim0 of 4 sources checked
    • cites [28]

      Not checked in the free run.

    • cites [16] source

      Not checked in the free run.

    • cites [29] source

      Not checked in the free run.

    • cites [44] source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingcitation

    Several batching techniques have been reported for ML serving systems.

    “Several batching techniques have been reported for ML serving systems.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingcitation

    Serverless frameworks like Apache OpenWhisk and Knative may be used to deploy models as functions and provide horizontal scaling, both building on top of Kubernetes.

    “Serverless frameworks like Apache OpenWhisk 25 and Knativefoot_11 may be used to deploy models as functions and provide horizontal scaling [16,43], both building on top of Kubernetes.”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites [16] source

      Not checked in the free run.

    • cites [43]

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingcitation

    One solution used in practice is to look at the distribution of predictions and determine if there is a significant deviation from what the response distribution of an ideal model would look like.

    “One solution used in practice is to look at the distribution of predictions and determine if there is a significant deviation from what the response distribution of an ideal model would look like [6].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [6] source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingcitation

    A Kubernetes-as-a-service tool such as Rancher may be used to manage multiple clusters of containers.

    “In order to manage multiple clusters of containers, a Kubernetes-as-a-service tool such as Rancher 24 may be used [40].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [40] source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingcitation

    Services running on Kubernetes and K3s may be defined with service descriptors like Helm.

    “Services running on Kubernetes and K3s may be defined with service descriptors like Helm 23 [40].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [40] source

      Not checked in the free run.

  • May combine two claims.

    Findingcitation

    After an inference is made, the true label may not be available for an extended period of time, making it difficult to monitor prediction quality.

    “After an inference is made, the true label may not be available for an extended period of time, making it difficult to monitor prediction quality [6].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [6] source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    [38] identified practical challenges in 16 different steps of ML deployment, and suggested further research in tooling and services for individual challenges, as well as new holistic approaches for dealing with ML systems engineering.

    “The authors identified practical challenges in 16 different steps of ML deployment, and suggested further research to be done in the areas of tooling and services for individual challenges, as well as new holistic approaches for dealing with ML systems engineering.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [38]

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    [35] noted that areas such as privacy and safety, evaluation, and deployment are missing solutions.

    “The authors also identified solutions to some of the challenges but noted that areas such as privacy and safety, evaluation, and deployment are missing solutions.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [35] source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    [27] found that challenges in SE for ML are found in all KAs, and that safety/security and non-technical areas have received a lot of attention.

    “The author found that challenges in SE for ML are found in all KAs, and that safety/security and non-technical areas, in particular, have received a lot of attention in the literature.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [27] source

      Not checked in the free run.

  • Classified as probably not a substantial claim.

    Study design and conductnone given

    Sources with an average of 0.5 points or less were excluded.

    “Sources with an average of 0.5 points or less were excluded.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    [24] concludes that successful AI/ML operationalization ensures proper model deployment and updating in different environments.

    “[24] concludes that successful AI/ML operationalization ensures proper model deployment and updating in different environments.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [24] source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    [37] found that studies from laboratory environments are mainly concerned with building and testing models, which could indicate more industry literature should be studied when reviewing state of art and practice of deploying models.

    “The authors found that studies from laboratory environments are mainly concerned with building and testing models, which could indicate that more literature produced by industry practitioners should be studied when looking to review the state of the art and state of the practice of deploying models to production.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [37]

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    [31] identified phases, techniques, gaps and trends in ML development lifecycle, and deployment aspect was only briefly mentioned.

    “The deployment aspect was only briefly mentioned.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [31]

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    [37] highlighted that deployment is one of the areas with the fewest suggested practices in published literature.

    “This was further illustrated when the authors highlighted that deployment is one of the areas with the fewest suggested practices in published literature.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [37]

      Not checked in the free run.

  • May combine two claims.

    Premisecitation

    ML models have an additional data dependency, in that their behavior is learned from data, and they have been characterized as non-deterministic.

    “ML models have an additional data dependency, in the sense that their behavior is learned from data, and they have even been characterized as non-deterministic [18,36].”
    Quote in the paperSubstantial claim0 of 2 sources checked
    • cites [18] source

      Not checked in the free run.

    • cites [36]

      Not checked in the free run.

  • Classified as probably not a substantial claim.

    Study design and conductcitation

    The quality of included studies was assessed using criteria adapted from [18] and suggested by [17].

    “The quality of included studies was assessed using the criteria found in Table 2, adapted from criteria used by [18] and suggested by [17].”
    Quote in the paperSubstantial claim0 of 2 sources checked
    • cites [18] source

      Not checked in the free run.

    • cites [17] source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    [52] found that AI model deployment was underexplored.

    “[52] was a SMS on the lifecycle of the AI models and found that AI model deployment was underexplored.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [52] source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    [4] found that practitioners' experiences generally confirmed challenges reported in the academic literature.

    “The authors found that the practitioners' experiences generally confirmed challenges reported in the academic literature.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [4]

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    MLOps is a subset of SE for ML and a superset/extension of DevOps, focusing on adopting DevOps practices when developing and operating ML systems.

    “DevOps for Machine Learning, named MLOps, is a subset of SE for ML and a superset/extension of DevOps, focusing on adopting DevOps practices when developing and operating ML systems [50].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [50] source

      Not checked in the free run.

  • May combine two claims.

    Interpretationcitation

    Developers have to choose either using a suboptimal framework or incurring technical debt to support and integrate multiple frameworks.

    “Developers then have to make the choice of either using a suboptimal framework or incurring technical debt in order to support and integrate multiple frameworks.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    Keeping all models warm at all times is prohibitively expensive and scales poorly.

    “As [54] has pointed out, keeping all models warm at all times is prohibitively expensive and scales poorly.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [54]

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    There is a strong desire for continuous monitoring and validation of AI systems post deployment for responsible AI requirements, where current MLOps practices provide limited guidance.

    “[32] observes that “There is a strong desire for continuous monitoring and validation of AI systems post deployment for responsible AI requirements, where current MLOps practices provide limited guidance.””
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [32]

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S4 (based). May combine two claims.

    Interpretationreasoning

    The number of edge devices may reach the thousands and beyond in an IoT scenario, suggesting that the scheduler may become a performance bottleneck in the future.

    “It is conceivable that the number of edge devices may reach the thousands and beyond in an IoT scenario, e.g., in industry 4.0, suggesting that the scheduler may become a performance bottleneck in the future.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationnone given

    The serving system must take into account the current state of the system, which is continually changing, in addition to the set of possible hardware resources and model variants.

    “In addition to the set of possible hardware resources and model variants, the serving system must also take into account the current state of the system, which is continually changing.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Premisecitation

    No out-of-the-box solution currently solves the issues of edge resource management from the cloud.

    “The authors reported that no out-of-the-box solution currently solves these issues, and request research into transparent edge-cloud resource consolidation/orchestration without the use of point-to-point integrations like VPNs.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [43]

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    The performance of the Kubernetes scheduler begins degrading with 5,000 nodes and fewer than 10 constraints, struggling to process more than 15 functions per second.

    “[43] observed that the performance of the Kubernetes scheduler begins degrading with 5,000 nodes and fewer than 10 constraints, struggling to process more than 15 functions per second.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [43]

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    An adaptive batching scheme specifically for neural networks was proposed, reducing latency during low-traffic scenarios.

    “while [11] proposed an adaptive batching scheme specifically for neural networks, reducing latency during low-traffic scenarios.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [11] source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Clipper used a batching technique with a static maximum batch size with a timeout window tuned on a per-model basis.

    “[12] used a batching technique with a static maximum batch size with a timeout window which is both tuned on a per-model basis”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [12]

      Not checked in the free run.

  • Classified as probably not a substantial claim.

    Premisecitation

    There is a lack of generic solutions for deploying ML models to embedded and edge devices.

    “[7] and [42] reported a lack of generic solutions for deploying ML models to embedded and edge devices.”
    Quote in the paperSubstantial claim0 of 2 sources checked
    • cites [7] source

      Not checked in the free run.

    • cites [42] source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S4 (based). May combine two claims.

    Interpretationreasoning

    Model management operations add latency to the query response, affecting the system's ability to meet SLOs.

    “The model management operations add latency to the query response, affecting the system's ability to meet SLOs.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Edge resource management from the cloud is complex, partly because nodes may be on private networks or behind firewalls.

    “[43] pointed out that edge resource management from the cloud is complex, partly because nodes may be on private networks or behind firewalls.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [43]

      Not checked in the free run.

  • May combine two claims.

    Interpretationcitation

    The Kubernetes-based nature of much ML infrastructure and tooling could be a barrier to MLOps adoption because setting up a Kubernetes cluster is an expensive task.

    “This could be a barrier to MLOps adoption, as [45] has noted that setting up a Kubernetes cluster is an expensive task.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [45] source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

    Study design and conductnone given

    Qualitative data analysis is closely tied to the researchers' background, identity, assumptions and beliefs.

    “Furthermore, qualitative data analysis is closely tied to the researchers' background, identity, assumptions and beliefs.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Study design and conductreasoning

    The starting set included papers spanning multiple years (2017-2021), venues (IEEE, ACM, Sciendo, Frontiers, Sciencedirect, Zenodo, IGI Global, Wiley, MDPI, Springer) and authors.

    “To mitigate this, the starting set included papers spanning multiple years (2017-2021), venues (IEEE, ACM, Sciendo, Frontiers, Sciencedirect, Zenodo, IGI Global, Wiley, MDPI, Springer) and authors.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationcitation

    Retraining triggers require that monitoring data be continuously streamed from the edge devices to the (possibly centralized) ML pipeline.

    “Retraining triggers require that monitoring data be continuously streamed from the edge devices to the (possibly centralized) ML pipeline.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

    Study design and conductnone given

    The study potentially misses out on practitioner knowledge.

    “The study potentially misses out on practitioner knowledge.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Conventional IoT provisioning frameworks are not suited for deploying ML models.

    “[42] reported that conventional IoT provisioning frameworks are not suited for deploying ML models”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [42] source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Some metrics, such as concept drift and model drift, require continuous access to the original training set.

    “First, some metrics, such as concept drift and model drift, require continuous access to the original training set.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [42] source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Support for non-x86 architectures is lacking.

    “[43] reported that support for non-x86 architectures is lacking.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [43]

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    There is a lack of solutions for deploying to edge devices that do not support containerization techniques.

    “while [20] reported that there is a lack of solutions for deploying to edge devices that do not support containerization techniques.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [20] source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    The initial string-based search possibly failed because too many aspects were attempted covered in just a few keywords, instead of using a more specific search string.

    “The initial string-based search possibly failed because too many aspects were attempted covered in just a few keywords, instead of using a more specific search string (such as”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductreasoning

    The data analysis results were discussed between the authors to arrive at a consensus when needed.

    “To address this issue, the data analysis results were discussed between the authors to arrive at a consensus when needed.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    There are certain inherent threats to validity in an SLR, such as the possibility that not all relevant literature is found or included.

    “There are certain inherent threats to validity in an SLR, such as the possibility that not all relevant literature is found or included.”
    Quote in the paperNot a substantial claim
82 of 82

  • The paper relies on this without support.

    The difficulty for most companies and practitioners now is not to find new algorithms and optimizations in training, but rather how to actually deploy models to production to deliver tangible business value.

    “With all the research that has been done on the training and evaluation of machine learning models, the difficulty for most companies and practitioners now is not to find new algorithms and optimizations in training, but rather how to actually deploy models to production in order to deliver tangible business value.”

  • The paper relies on this without support.

    Software engineering for ML systems is still a young and immature knowledge area.

    “While traditional SE for non-ML systems is a mature and wellunderstood field, software engineering for ML systems is still a young and immature knowledge area.”

  • The paper relies on this without support.

    Tackling operationalization challenges requires adopting good practices and utilizing suitable tooling.

    “Tackling operationalization challenges requires adopting good practices and utilizing suitable tooling.”

  • The paper relies on this without support.

    ML models typically do not make inferences based on raw data but may require data transformations both before and after inference.

    “However, ML models typically do not make inferences based on raw data but may require data transformations both before and after inference.”

  • The paper relies on this without support.

    Edge resources may have unreliable or limited network access, making it challenging to maintain constant communication channels.

    “In addition, edge resources may have unreliable or limited network access, making it challenging to maintain constant communication channels.”

  • The paper relies on this without support.

    CPUs are currently the most widely used for inference in existing ML as a service (MLaaS) platforms.

    “CPUs are currently the most widely used for inference in existing ML as a service (MLaaS) platforms.”

  • The paper relies on this without support.

    Other types of hardware exist that are suited for neural network inference, such as GPUs, TPUs, etc.

    “Other types of hardware exist that are suited for neural network inference, such as GPUs, TPUs, etc.”

  • The paper relies on this without support.

    Edge devices present greater heterogeneity in computing capabilities, storage and networking.

    “Edge devices present greater heterogeneity in computing capabilities, storage and networking.”

  • The paper relies on this without support.

    In the context of having multiple models deployed in an edge architecture, some models may have to be evicted from time to time in order to free up resources for new deployments.

    “In the context of having multiple models deployed in an edge architecture, some models may have to be evicted from time to time in order to free up resources for new deployments.”

  • The paper relies on this without support.

    Many processes in ML pipelines contain multiple subtasks, where changing a single step may cause regressions.

    “Many processes in ML pipelines contain multiple subtasks, where changing a single step may cause regressions.”

  • The paper relies on this without support.

    The choice of model placement depends on individual application requirements, e.g., whether the inference is online or offline, what the SLOs are.

    “The choice will depend on the individual application requirements, e.g., whether the inference is online or offline, what the SLOs are, etc.”
11 of 11

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Kolltveit et al. [13] reported an SLR on the operationalization of ML systems, which covers all steps following model training and evaluation.

    “Kolltveit et al. [13] reported an SLR on the operationalization of ML systems, which covers all steps following model training and evaluation.”

    From A Multivocal Review of MLOps Practices, Challenges and Open Issues (2026) · cites this paper as [13]

  • Kolltveit et al. [13] answered four RQs focused on tooling and infrastructure related aspects of MLOps.

    “Specifically, they answered four RQs focused on tooling and infrastructure related aspects of MLOps such as; how are ML models operationalized, what are the main challenges in operationalizing ML models, what tools and software infrastructure are used to operationalize ML models, what are the feature gaps in the tooling and infrastructure used to operationalize ML models.”

    From A Multivocal Review of MLOps Practices, Challenges and Open Issues (2026) · cites this paper as [13]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • Earliest publication year conflicts with selection criterion

    Minor

    The methods say 2015 was chosen as the earliest publication year, but Table 1 requires studies to have been published after 2015, which excludes publications from that year.

1 of 1

Cites

Flagged first, then by citations

Research works printed but not linked: 11

  1. 2020. The state of AI in 2020 . https://www.mckinsey.com/business-functions/mckinsey-analytics/our-insights/global-survey-the-state-of-ai-in-2020 2020. The state of AI in 2020. https://www.mckinsey.com/business-functions/mckinsey-analytics/our-insights/global-survey-the-state-of-ai-in-2020
  2. Lucas Baier , Fabian Jöhren , and Stefan Seebacher . 2019 . Challenges in the deployment and operation of machine learning in practice . In ECIS 2019 - 27th European Conference on Information Systems. https://www.researchgate.net/publication/332996647_CHALLENGES_IN_THE_DEPLOYMENT_AND_OPERATION_OF_MACHINE_LEARNING_IN_PRACTICE Lucas Baier, Fabian Jöhren, and Stefan Seebacher. 2019. Challenges in the deployment and operation of machine learning in practice. In ECIS 2019 - 27th European Conference on Information Systems. https://www.researchgate.net/publication/332996647_CHALLENGES_IN_THE_DEPLOYMENT_AND_OPERATION_OF_MACHINE_LEARNING_IN_PRACTICE
  3. Daniel Crankshaw , Xin Wang , Guilio Zhou , Michael J. Franklin , Joseph E. Gonzalez , and Ion Stoica . 2017 . Clipper: A Low-Latency Online Prediction Serving System. In 14th USENIX Symposium on Networked Systems Design and Implementation (NSDI 17) . USENIX Association, Boston, MA, 613--627. https://www.usenix.org/conference/nsdi17/technical-sessions/presentation/crankshaw Daniel Crankshaw, Xin Wang, Guilio Zhou, Michael J. Franklin, Joseph E. Gonzalez, and Ion Stoica. 2017. Clipper: A Low-Latency Online Prediction Serving System. In 14th USENIX Symposium on Networked Systems Design and Implementation (NSDI 17). USENIX Association, Boston, MA, 613--627. https://www.usenix.org/conference/nsdi17/technical-sessions/presentation/crankshaw
  4. Sweenor David Hillion Steven Rope Dan Kannabiran Dev Hill Thomas O'Connell Michael and Safari an O'Reilly Media Company. 2020. ML Ops : Operationalizing Data Science. https://go.oreilly.com/queensland-university-of-technology/library/view/-/9781492074663/?ar Sweenor David Hillion Steven Rope Dan Kannabiran Dev Hill Thomas O'Connell Michael and Safari an O'Reilly Media Company. 2020. ML Ops : Operationalizing Data Science. https://go.oreilly.com/queensland-university-of-technology/library/view/-/9781492074663/?ar
  5. Li Erran Li , Eric Chen , Jeremy Hermann , Pusheng Zhang , and Luming Wang . 2017 . Scaling Machine Learning as a Service . In Proceedings of The 3rd International Conference on Predictive Applications and APIs (Proceedings of Machine Learning Research , Vol. 67), Claire Hardgrove, Louis Dorard, Keiran Thompson, and Florian Douetteau (Eds.). PMLR, 14-- 29 . https://proceedings.mlr.press/v67/li17a.html Li Erran Li, Eric Chen, Jeremy Hermann, Pusheng Zhang, and Luming Wang. 2017. Scaling Machine Learning as a Service. In Proceedings of The 3rd International Conference on Predictive Applications and APIs (Proceedings of Machine Learning Research, Vol. 67), Claire Hardgrove, Louis Dorard, Keiran Thompson, and Florian Douetteau (Eds.). PMLR, 14--29. https://proceedings.mlr.press/v67/li17a.html
  6. Qinghua Lu Liming Zhu Xiwei Xu Jon Whittle David Douglas and Conrad Sanderson. 2021. Software Engineering for Responsible AI: An Empirical Study and Operationalised Patterns. https://doi.org/abs/2111.09478 Qinghua Lu Liming Zhu Xiwei Xu Jon Whittle David Douglas and Conrad Sanderson. 2021. Software Engineering for Responsible AI: An Empirical Study and Operationalised Patterns. https://doi.org/abs/2111.09478
  7. Silverio Martínez-Fernández , Justus Bogner , Xavier Franch , Marc Oriol , Julien Siebert , Adam Trendowicz , Anna Maria Vollmer, and Stefan Wagner . 2021 . Software Engineering for AI-Based Systems: A Survey . (May 2021). arXiv:2105.01984 [cs.SE] Silverio Martínez-Fernández, Justus Bogner, Xavier Franch, Marc Oriol, Julien Siebert, Adam Trendowicz, Anna Maria Vollmer, and Stefan Wagner. 2021. Software Engineering for AI-Based Systems: A Survey. (May 2021). arXiv:2105.01984 [cs.SE] · arXiv
  8. Thomas Rausch , Waldemar Hummer , Vinod Muthusamy , Alexander Rashed , and Schahram Dustdar . 2019 . Towards a Serverless Platform for Edge AI . In 2nd USENIX Workshop on Hot Topics in Edge Computing (HotEdge 19) . USENIX Association, Renton, WA. https://www.usenix.org/conference/hotedge19/presentation/rausch Thomas Rausch, Waldemar Hummer, Vinod Muthusamy, Alexander Rashed, and Schahram Dustdar. 2019. Towards a Serverless Platform for Edge AI. In 2nd USENIX Workshop on Hot Topics in Edge Computing (HotEdge 19). USENIX Association, Renton, WA. https://www.usenix.org/conference/hotedge19/presentation/rausch
  9. D. Sculley , Gary Holt , Daniel Golovin , Eugene Davydov , Todd Phillips , Dietmar Ebner , Vinay Chaudhary , Michael Young , Jean-François Crespo , and Dan Dennison . 2015 . Hidden Technical Debt in Machine Learning Systems. In Advances in Neural Information Processing Systems, C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R . Garnett (Eds.) , Vol. 28 . D. Sculley, Gary Holt, Daniel Golovin, Eugene Davydov, Todd Phillips, Dietmar Ebner, Vinay Chaudhary, Michael Young, Jean-François Crespo, and Dan Dennison. 2015. Hidden Technical Debt in Machine Learning Systems. In Advances in Neural Information Processing Systems, C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Garnett (Eds.), Vol. 28.
  10. Alex Serban and Joost Visser . 2021. An Empirical Study of Software Architecture for Machine Learning. (May 2021 ). arXiv:2105.12422 [cs.SE] Alex Serban and Joost Visser. 2021. An Empirical Study of Software Architecture for Machine Learning. (May 2021). arXiv:2105.12422 [cs.SE] · arXiv
  11. Jeff Zhang , Sameh Elnikety , Shuayb Zarar , Atul Gupta , and Siddharth Garg . 2020 . Model-Switching: Dealing with Fluctuating Workloads in Machine-Learning-asa-Service Systems. In 12th USENIX Workshop on Hot Topics in Cloud Computing (HotCloud 20) . USENIX Association. https://www.usenix.org/conference/hotcloud20/presentation/zhang Jeff Zhang, Sameh Elnikety, Shuayb Zarar, Atul Gupta, and Siddharth Garg. 2020. Model-Switching: Dealing with Fluctuating Workloads in Machine-Learning-asa-Service Systems. In 12th USENIX Workshop on Hot Topics in Cloud Computing (HotCloud 20). USENIX Association. https://www.usenix.org/conference/hotcloud20/presentation/zhang

Cited by

Sorted by citations

Related

Authors
Kolltveit, A. B., & Li, J.
Affiliations
Norwegian University of Science and Technology (NO)
ORCID
Li, Jingyue
Venue
Proceedings of the 1st Workshop on Software Engineering for Responsible AI
Publisher
ACM
Published
2022-05-19
Pages
1-8
DOI
10.1145/3526073.3527584
OpenAlex
W4319150613
Semantic Scholar
c82e39cc3b49bcb9868bad539376ddff34730ada
Cited by
34
References
40
Citations by year
2026: 4 · 2025: 15 · 2024: 10 · 2023: 4 · 2022: 1
FWCI
6.71
Type
conference paper
Language
en
Open access
Licence
www.acm.org/publications/policies/copyright_policy#Background (vor)
Topics
  • Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
  • Physical Sciences › Computer Science › Information Systems › Software Engineering Research
  • Physical Sciences › Computer Science › Computer Networks and Communications › Software System Performance and Reliability
Keywords
model deployment, model serving, model registry, model monitoring, cloud deployment, edge machine learning, continuous monitoring, MLOps

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-10.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-10.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review