CoChat
Sign Up
PreprintCited by 415Cloud Computing and Resource ManagementData Stream Mining TechniquesSoftware System Performance and Reliability

Clipper: A Low-Latency Online Prediction Serving System

Open access
Crankshaw, Daniel, Wang, Xin, Zhou, Giulio, Franklin, Michael J., Gonzalez, Joseph E., Stoica, Ion·2016·arXiv
DOI 10.48550/arxiv.1612.03079arXiv 1612.03079
Open this paper in CoChat — free accountOpen PDF

Abstract

Machine learning is being deployed in a growing number of applications which demand real-time, accurate, and robust predictions under heavy query load. However, most machine learning frameworks and systems only address model training and not deployment. In this paper, we introduce Clipper, a general-purpose low-latency prediction serving system. Interposing between end-user applications and a wide range of machine learning frameworks, Clipper introduces a modular architecture to simplify model deployment across frameworks and applications. Furthermore, by introducing caching, batching, and adaptive model selection techniques, Clipper reduces prediction latency and improves prediction throughput, accuracy, and robustness without modifying the underlying machine learning frameworks. We evaluate Clipper on four common machine learning benchmark datasets and demonstrate its ability to meet the latency, accuracy, and throughput demands of online serving applications. Finally, we compare Clipper to the TensorFlow Serving system and demonstrate that we are able to achieve comparable throughput and latency while enabling model composition and online learning to improve accuracy and render more robust predictions.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.1612.03079

Cites

3 of 3 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • General model serving systems fail to take into account the autoregressive property and token state of LLM inference, resulting in missed opportunities for optimization.

    “However, these general systems fail to take into account the autoregressive property and token state of LLM inference, resulting in missed opportunities for optimization.”

    From Efficient Memory Management for Large Language Model Serving with PagedAttention (2023) · cites this paper as [11]

    Section 7 (Limitations; page not available) says Clipper treats models as black boxes and does not optimize their execution, supporting the general point that its serving architecture misses model-specific optimizations. The paper does not address LLM inference, autoregressive generation, or token state, and it does not establish the claim about general serving systems as a group. (LLM, conf=0.78)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 42

  1. [2][2] A. Agarwal, S. Bird, M. Cozowicz, L. Hoang, J. Lang- ford, S. Lee, J. Li, D. Melamed, G. Oshri, O. Ribas, et al. A multiworld testing decision service. arXiv preprint arXiv:1606.03966, 2016. · arXiv
  2. [3][3] D. Agarwal, B. Long, J. Traupman, D. Xin, and L. Zhang. Laser: A scalable response prediction platform for online advertising. In WSDM, pages 173–182, 2014.
  3. [4][4] S. Agarwal and J. R. Lorch. Matchmaking for online games and other latency-sensitive p2p systems. In ACM SIGCOMM Computer Communication Review, volume 39, pages 315–326. ACM, 2009.
  4. [5][5] A. Ahmed, M. Aly, J. Gonzalez, S. Narayanamurthy, and A. J. Smola. Scalable inference in latent variable models. In WSDM, pages 123–132, 2012.
  5. [6][6] P. Auer, N. Cesa-Bianchi, Y. Freund, and R. E. Schapire. The nonstochastic multiarmed bandit problem. SIAM J. Comput., 32(1):48–77, Jan. 2003.
  6. [7][7] J. Bergstra, O. Breuleux, F. Bastien, P. Lamblin, R. Pas- canu, G. Desjardins, J. Turian, D. Warde-Farley, and Y. Bengio. Theano: a cpu and gpu math expression com- piler. In Proceedings of the Python for scientific computing conference (SciPy), volume 4, page 3. Austin, TX, 2010.
  7. [8][8] C. M. Bishop. Pattern Recognition and Machine Learning. Springer-Verlag, Secaucus, NJ, USA, 2006.
  8. [9][9] L. Breiman. Bagging predictors. Mach. Learn., 24(2):123– 140, Aug. 1996.
  9. [10][10] C. Chelba, D. Bikel, M. Shugrina, P. Nguyen, and S. Ku- mar. Large scale language modeling in automatic speech recognition. arXiv preprint arXiv:1210.8440, 2012. · arXiv
  10. [11][11] J. Chen, R. Monga, S. Bengio, and R. Jozefowicz. Re- visiting Distributed Synchronous SGD. arXiv.org, Apr. 2016.
  11. [12][12] T. Chen and C. Guestrin. XGBoost: A Scalable Tree Boosting System. arXiv.org, Mar. 2016.
  12. [13][13] T. Chen, M. Li, Y. Li, M. Lin, N. Wang, M. Wang, T. Xiao, B. Xu, C. Zhang, and Z. Zhang. Mxnet: A flexible and efficient machine learning library for heterogeneous dis- tributed systems. arXiv preprint arXiv:1512.01274, 2015. · arXiv
  13. [14][14] T. Chilimbi, Y. Suzue, J. Apacible, and K. Kalyanaraman. Project adam: Building an efficient and scalable deep learn- ing training system. In OSDI, pages 571–582, 2014.
  14. [15][15] D.-M. Chiu and R. Jain. Analysis of the increase and decrease algorithms for congestion avoidance in computer networks. Comput. Netw. ISDN Syst., 17(1):1–14, June 1989.
  15. [16][16] R. Collobert, K. Kavukcuoglu, and C. Farabet. Torch7: A matlab-like environment for machine learning. In BigLearn, NIPS Workshop, number EPFL-CONF-192376, 2011.
  16. [20][20] J. Davidson, B. Liebald, J. Liu, P. Nandy, T. Van Vleet, U. Gargi, S. Gupta, Y. He, M. Lambert, B. Livingston, and D. Sampath. The YouTube video recommendation system. RecSys, pages 293–296, 2010.
  17. [21][21] J. Dean, G. Corrado, R. Monga, K. Chen, M. Devin, M. Mao, M. aurelio Ranzato, A. Senior, P. Tucker, K. Yang, Q. V. Le, and A. Y. Ng. Large scale distributed deep net- works. In NIPS, pages 1223–1231. 2012.
  18. [23][23] A. Ganjam, F. Siddiqui, J. Zhan, X. Liu, I. Stoica, J. Jiang, V. Sekar, and H. Zhang. C3: Internet-Scale Control Plane for Video Quality Optimization. NSDI '15, pages 131–144, 2015.
  19. [25][25] J. E. Gonzalez, Y. Low, H. Gu, D. Bickson, and C. Guestrin. Powergraph: Distributed graph-parallel computation on natural graphs. OSDI, pages 17–30, 2012.
  20. [27][27] T. Graepel, J. Q. Candela, T. Borchert, and R. Herbrich. Web-Scale Bayesian Click-Through rate Prediction for Sponsored Search Advertising in Microsoft's Bing Search Engine. ICML, pages 13–20, 2010.
  21. [30][30] G. Hinton, O. Vinyals, and J. Dean. Distilling the Knowl- edge in a Neural Network. arXiv.org, Mar. 2015.
  22. [31][31] Y. Jia, E. Shelhamer, J. Donahue, S. Karayev, J. Long, R. Girshick, S. Guadarrama, and T. Darrell. Caffe: Con- volutional architecture for fast feature embedding. In Pro- ceedings of the ACM International Conference on Multi- media, pages 675–678. ACM, 2014.
  23. [36][36] X. Lei, A. W. Senior, A. Gruenstein, and J. Sorensen. Ac- curate and compact large vocabulary speech recognition on mobile devices. INTERSPEECH, pages 662–665, 2013.
  24. [37][37] R. Lerallut, D. Gasselin, and N. Le Roux. Large-Scale Real-Time Product Recommendation at Criteo. In RecSys, pages 232–232, 2015.
  25. [38][38] M. Li, D. G. Andersen, J. W. Park, A. J. Smola, A. Ahmed, V. Josifovski, J. Long, E. J. Shekita, and B.-Y. Su. Scaling distributed machine learning with the parameter server. In OSDI, pages 583–598, 2014.
  26. [39][39] H. B. McMahan, G. Holt, D. Sculley, M. Young, D. Ebner, J. Grady, L. Nie, T. Phillips, E. Davydov, D. Golovin, S. Chikkerur, D. Liu, M. Wattenberg, A. M. Hrafnkelsson, T. Boulos, and J. Kubica. Ad click prediction: a view from the trenches. In KDD, page 1222, 2013.
  27. [40][40] X. Meng, J. Bradley, B. Yavuz, E. Sparks, S. Venkatara- man, D. Liu, J. Freeman, D. Tsai, M. Amde, S. Owen, D. Xin, R. Xin, M. J. Franklin, R. Zadeh, M. Zaharia, and A. Talwalkar. Mllib: Machine learning in apache spark. Journal of Machine Learning Research, 17(34):1–7, 2016.
  28. [41][41] V. Mnih, N. Heess, A. Graves, et al. Recurrent models of visual attention. In NIPS, pages 2204–2212, 2014.
  29. [43][43] K. P. Murphy. Machine Learning: A Probabilistic Perspec- tive. The MIT Press, 2012.
  30. [44][44] J. Nagle. Congestion control in ip/tcp internetworks. SIG- COMM Comput. Commun. Rev., 14(4):11–17, Oct. 1984.
  31. [46][46] V. S. Pai, P. Druschel, and W. Zwaenepoel. Flash: An effi- cient and portable Web server. USENIX Annual Technical Conference, General Track, pages 199–212, 1999.
  32. [49][49] O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, A. C. Berg, and L. Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Com- puter Vision (IJCV), 115(3):211–252, 2015.
  33. [50][50] D. C. Schmidt. Pattern languages of program design. chap- ter Reactor: An Object Behavioral Pattern for Concurrent Event Demultiplexing and Event Handler Dispatching, pages 529–545. 1995.
  34. [52][52] D. Sculley, G. Holt, D. Golovin, E. Davydov, T. Phillips, D. Ebner, V. Chaudhary, M. Young, J.-F. Crespo, and D. Dennison. Hidden Technical Debt in Machine Learning Systems. NIPS, 2015.
  35. [54][54] K. Simonyan and A. Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556, 2014. · arXiv
  36. [57][57] C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, and A. Rabinovich. Going deeper with convolutions. In CVPR, pages 1–9, 2015.
  37. [58][58] C. Szegedy, V. Vanhoucke, S. Ioffe, J. Shlens, and Z. Wo- jna. Rethinking the inception architecture for computer vision. arXiv preprint arXiv:1512.00567, 2015. · arXiv
  38. [61][61] M. Welsh, D. E. Culler, and E. A. Brewer. SEDA: An Architecture for Well-Conditioned, Scalable Internet Ser- vices. SOSP, pages 230–243, 2001.
  39. [62][62] E. P. Xing, Q. Ho, W. Dai, J.-K. Kim, J. Wei, S. Lee, X. Zheng, P. Xie, A. Kumar, and Y. Yu. Petuum: A new platform for distributed machine learning on big data. In KDD, pages 1335–1344. ACM, 2015.
  40. [63][63] S. J. Young, G. Evermann, M. J. F. Gales, T. Hain, D. Kershaw, G. Moore, J. Odell, D. Ollason, D. Povey, V. Valtchev, and P. C. Woodland. The HTK Book, ver- sion 3.4. Cambridge University Engineering Department, Cambridge, UK, 2006.
  41. [64][64] J.-M. Yun, Y. He, S. Elnikety, and S. Ren. Optimal aggre- gation policy for reducing tail latency of web search. In SIGIR, pages 63–72, 2015.
  42. [65][65] M. Zaharia et al. Resilient distributed datasets: A fault- tolerant abstraction for in-memory cluster computing. In NSDI, 2012. 15 https://www.tensorflow.org/versions/r0.10/tutorials/mnist/pros/index.html https://www.tensorflow.org/versions/r0.10/tutorials/mnist/pros/index.html https://www.tensorflow.org/versions/r0.10/tutorials/mnist/pros/index.html http://nginx.org/en/ http://dmg.org/pfa/index.html http://dmg.org/pfa/index.html http://dmg.org/pmml/v4-2-1/GeneralStructure.html http://dmg.org/pmml/v4-2-1/GeneralStructure.html http://scikit-learn.org http://scikit-learn.org http://www.apple.com/ios/siri/ https://www.skype.com/en/features/skype-translator/ https://www.skype.com/en/features/skype-translator/ https://tensorflow.github.io/serving https://tensorflow.github.io/serving https://turi.com

Cited by

Sorted by citations

Authors
Crankshaw, D., Wang, X., Zhou, G., Franklin, M. J., Gonzalez, J. E., & Stoica, I.
Affiliations
  • University of California, Berkeley (US)
  • University of Chicago (US)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2016
Pages
613-627
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.1612.03079
arXiv
1612.03079
OpenAlex
W2964108773
Cited by
415
References
3
Citations by year
2026: 5 · 2025: 31 · 2024: 52 · 2023: 48 · 2022: 44 · 2021: 62 · 2020: 61 · 2019: 65 · 2018: 35 · 2017: 10 · 2016: 1 · 2015: 1
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
  • Networked Systems Design and Implementation
Funders
  • National Science Foundation (SN10040 DE-SC0012463, 1139158)
  • U.S. Department of Energy
  • Amazon Web Services
  • VMware
  • Ant Financial Services Group
  • Directorate for Computer and Information Science and Engineering (CCF-1139158)
Topics
  • Physical Sciences › Computer Science › Information Systems › Cloud Computing and Resource Management
  • Physical Sciences › Computer Science › Artificial Intelligence › Data Stream Mining Techniques
  • Physical Sciences › Computer Science › Computer Networks and Communications › Software System Performance and Reliability
Keywords
online prediction, low-latency inference, model deployment, caching, request batching, adaptive model selection, model composition, online learning, inference latency

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review