A Case for Managed and Model-less Inference Serving
Abstract
The number of applications relying on inference from machine learning models, especially neural networks, is already large and expected to keep growing. For instance, Facebook applications issue tens-of-trillions of inference queries per day with varying performance, accuracy, and cost constraints. Unfortunately, today's inference serving systems are neither easy to use nor cost effective. Developers must manually match the performance, accuracy, and cost constraints of their applications to a large design space that includes decisions such as selecting the right model and model optimizations, selecting the right hardware architecture, selecting the right scale-out factor, and avoiding cold-start effects. These interacting decisions are difficult to make, especially when the application load varies over time, applications evolve over time, and the available resources vary over time.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Start-up/cold-start latency is a typical serving challenge for ML models.
“Start-up/cold-start latency is a typical serving challenge for ML models [53].”
From Operationalizing machine learning models: a systematic literature review (2022) · cites this paper as [53]
Edge deployments are often reported as being more challenging than cloud deployments because of hardware resource heterogeneity, the typically distributed nature of edge devices, and possibly varying network connection quality.
“Edge deployments are often reported as being more challenging than cloud deployments because of hardware resource heterogeneity, the typically distributed nature of edge devices, and possibly varying network connection quality [7,10,20,22,39,53].”
From Operationalizing machine learning models: a systematic literature review (2022) · cites this paper as [53]
Meeting performance SLOs during significant variations in model query traffic is a problem.
“[54] discussed the problem of meeting performance SLOs during significant variations in model query traffic, a problem also observed by [53].”
From Operationalizing machine learning models: a systematic literature review (2022) · cites this paper as [53]
Cites
SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <0.5MB model size
Iandola, F. N., Han, S., Moskewicz, M. W., Ashraf, K., Dally, W. J., & Keutzer, K. (2016) · arXiv · doi:10.48550/arxiv.1602.07360 · Cited by 6,201 · see citing works
In-Datacenter Performance Analysis of a Tensor Processing Unit
Jouppi, N. P., Young, C., Patil, N., Patterson, D., Agrawal, G., Bajwa, R., Bates, S., Bhatia, S., Boden, N., Borchers, A., et al. (2017) · Proceedings of the 44th Annual International Symposium on Computer Architecture · doi:10.1145/3079856.3080246 · Cited by 4,537 · see citing works
The emergence of edge computing
Satyanarayanan, M. (2017) · Computer · doi:10.1109/mc.2017.9 · Cited by 2,832 · see citing works
Hogwild: A Lock-Free Approach to Parallelizing Stochastic Gradient Descent
Niu, F., Recht, B., Re, C., & Wright, S. J. (2011) · arXiv · doi:10.48550/arxiv.1106.5730 · Cited by 1,211 · see citing works
An Analysis of Deep Neural Network Models for Practical Applications
Canziani, A., Paszke, A., & Culurciello, E. (2016) · arXiv · doi:10.48550/arxiv.1605.07678 · Cited by 969 · see citing works
Neurosurgeon: Collaborative Intelligence Between the Cloud and Mobile Edge
Kang, Y., Hauswald, J., Gao, C., Rovinski, A., Mudge, T., Mars, J., & Tang, L. (2017) · Proceedings of the Twenty-Second International Conference on Architectural Support for Programming Languages and Operating Systems · doi:10.1145/3037697.3037698 · Cited by 673 · see citing works
Multi-Scale Dense Networks for Resource Efficient Image ClassificationPreprint
Huang, G. M., Chen, D., Li, T., Wu, F., van der Maaten, L., & Weinberger, K. Q. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1703.09844 · Cited by 638 · see citing works
Applied Machine Learning at Facebook: A Datacenter Infrastructure Perspective
Hazelwood, K., Bird, S., Brooks, D., Chintala, S., Diril, U., Dzhulgakov, D., Fawzy, M., Jia, B., Jia, Y., Kalro, A., et al. (2018) · 2018 IEEE International Symposium on High Performance Computer Architecture (HPCA) · doi:10.1109/hpca.2018.00059 · Cited by 590 · see citing works
A Configurable Cloud-Scale DNN Processor for Real-Time AI
Fowers, J., Ovtcharov, K., Papamichael, M. K., Massengill, T., Liu, M., Lo, D., Alkalay, S., Haselman, M., Adams, L., Ghandi, M., et al. (2018) · Proceedings - International Symposium on Computer Architecture/Proceedings · doi:10.1109/isca.2018.00012 · Cited by 529 · see citing works
Chameleon: scalable adaptation of video analytics
Jiang, J., Ananthanarayanan, G., Bodik, P., Sen, S., & Stoica, I. (2018) · Proceedings of the 2018 Conference of the ACM Special Interest Group on Data Communication · doi:10.1145/3230543.3230574 · Cited by 467 · see citing works
Clipper: A Low-Latency Online Prediction Serving System
Crankshaw, D., Wang, X., Zhou, G., Franklin, M. J., Gonzalez, J. E., & Stoica, I. (2016) · arXiv · doi:10.48550/arxiv.1612.03079 · Cited by 415 · see citing works
Jonas, E., Pu, Q., Venkataraman, S., Stoica, I. G., & Recht, B. (2017) · Symposium on Cloud Computing · doi:10.1145/3127479.3128601 · Cited by 413 · see citing works
Gandhi, A., Harchol‐Balter, M., Raghunathan, R., et al. (2012) · ACM Transactions on Computer Systems · doi:10.1145/2382553.2382556 · Cited by 301 · see citing works
CoCoA: A General Framework for Communication-Efficient Distributed OptimizationPreprint
Smith, V., Forte, S., Ma, C.-X., Takác, M., Jordan, M. I., & Jaggi, M. (2016) · Journal of machine learning research · doi:10.48550/arxiv.1611.02189 · Cited by 286 · see citing works
SAND: Towards High-Performance Serverless Computing.
Akkuş, İ. E., Chen, R., Rimac, I., Stein, M., Satzke, K., Beck, A. T., Aditya, P., & Hilt, V. (2018) · Cited by 263 · see citing works
Flexpoint: An Adaptive Numerical Format for Efficient Training of Deep Neural NetworksPreprint
Köster, U., Webb, T. J., Wang, X., Nassar, M., Bansal, A. K., Constable, W., Elibol, O. H., Gray, S., Hall, S., Hornof, L., et al. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1711.02213 · Cited by 236 · see citing works
CoCoA: A General Framework for Communication-Efficient Distributed OptimizationPreprint
Smith, V., Forte, S., Ma, C., Takáč, M., Jordan, M. I., & Jaggi, M. (2016) · arXiv (Cornell University) · doi:10.3929/ethz-b-000282738 · Cited by 204 · see citing works
8th Annual Conference of the International Speech Communication Association
Murray, G., & Renals, S. J. (2007) · Cited by 166 · see citing works
Cache Telepathy: Leveraging Shared Resource Attacks to Learn DNN ArchitecturesPreprint
Yan, M., Fletcher, C. W., & Torrellas, J. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1808.04761 · Cited by 152 · see citing works
SparkNet: Training Deep Networks in SparkPreprint
Moritz, P., Nishihara, R., Stoica, I. G., & Jordan, M. I. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1511.06051 · Cited by 126 · see citing works
Gujarati, A., Elnikety, S., He, Y., McKinley, K. S., & Brandenburg, B. B. (2017) · International Middleware Conference · doi:10.1145/3135974.3135993 · Cited by 99 · see citing works
Wang, W., Gao, J., Zhang, M., Wang, S., Chen, G., Ng, T. K., Ooi, B. C., Shao, J., & Reyad, M. (2018) · Proceedings of the VLDB Endowment · doi:10.14778/3282495.3282499 · Cited by 93 · see citing works
Latte: a language, compiler, and runtime for elegant and efficient deep neural networks
Truong, L., Barik, R., Totoni, E., et al. (2016) · Proceedings of the ACM SIGPLAN Conference on Programming Language Design and Implementation · doi:10.1145/2908080.2908105 · Cited by 51 · see citing works
Multilingual Multi-class Sentiment Classification Using Convolutional Neural Networks
Attia, M. A., Samih, Y., Elkahky, A. M., & Kallmeyer, L. (2018) · Language Resources and Evaluation Conference (LREC) · doi:10.63317/4suqsh39s5kg · Cited by 39 · see citing works
Semantic Lattice Processing in Contextual Automatic Speech Recognition for Google Assistant
Velikovich, L., Williams, I., Scheiner, J., et al. (2018) · Interspeech (USB) · doi:10.21437/interspeech.2018-2453 · Cited by 30 · see citing works
Performance Analysis of Cloud Applications
Ardelean, D., Diwan, A., & Erdman, C. (2018) · Networked Systems Design and Implementation · Cited by 29 · see citing works
Dynamic Space-Time Scheduling for GPU InferencePreprint
Jain, P., Mo, X., Jain, A., et al. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1901.00041 · Cited by 26 · see citing works
A portable, automatic data qantizer for deep neural networks
Oh, Y. H., Quan, Q., Kim, D.-Y., Kim, S., Heo, J., Jung, S., Jang, J.-Y., & Lee, J. W. (2018) · International Conference on Parallel Architectures and Compilation Techniques (PACT) · doi:10.1145/3243176.3243180 · Cited by 24 · see citing works
AbdelBaky, M., Zou, M., Zamani, A. R., et al. (2017) · IEEE International Conference on Distributed Computing Systems (ICDCS) · doi:10.1109/icdcs.2017.323 · Cited by 16 · see citing works
LIT: Block-wise Intermediate Representation Training for Model CompressionPreprint
Koratana, A., Kang, D. D., Bailis, P. D., & Zaharia, M. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1810.01937 · Cited by 12 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Operationalizing machine learning models: a systematic literature review
Kolltveit, A. B., & Li, J. (2022) · Proceedings of the 1st Workshop on Software Engineering for Responsible AI · doi:10.1145/3526073.3527584 · Cited by 34 · see citing works
- Authors
- Yadwadkar, N. J., Romero, F., Li, Q., & Kozyrakis, C.
- Affiliations
- Stanford University (US)
- Venue
- Proceedings of the Workshop on Hot Topics in Operating Systems
- Publisher
- ACM
- Published
- 2019-05-13
- Pages
- 184-191
- OpenAlex
- W2944614352
- Cited by
- 31
- References
- 31
- Citations by year
- 2025: 6 · 2024: 4 · 2023: 2 · 2022: 5 · 2021: 8 · 2020: 3 · 2019: 3
- FWCI
- 3.78
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- Workshop on Hot Topics in Operating Systems (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Information Systems › Cloud Computing and Resource Management
- Physical Sciences › Computer Science › Hardware and Architecture › Parallel Computing and Optimization Techniques
- Physical Sciences › Computer Science › Computer Networks and Communications › Software System Performance and Reliability
- Keywords
- inference serving, machine learning inference, artificial neural networks, model selection, model optimization, horizontal scaling, cost constraints, dynamic resource allocation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
