The DOI matches its DataCite record.
TensorFlow-Serving: Flexible, High-Performance ML Serving
Open accessAbstract
We describe TensorFlow-Serving, a system to serve machine learning models inside Google which is also available in the cloud and via open-source. It is extremely flexible in terms of the types of ML platforms it supports, and ways to integrate with systems that convey new models and updated versions from training to serving. At the same time, the core code paths around model lookup and inference have been carefully optimized to avoid performance pitfalls observed in naive implementations. Google uses it in many production deployments, including a multi-tenant model hosting service called TFS^2.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Existing LLM serving systems store the KV cache of a request in contiguous memory space, which causes inefficient memory management.
“This is mainly because they store the KV cache of a request in contiguous memory space, as most deep learning frameworks [33,39] require tensors to be stored in contiguous memory.”
From Efficient Memory Management for Large Language Model Serving with PagedAttention (2023) · cites this paper as [33]
Section 2.2.1 (p. 5) describes concatenating input tensors for batching, but does not say that deep learning frameworks require tensors to occupy contiguous memory. The passages also do not address LLM KV-cache storage or resulting memory-management inefficiency (searched: KV cache, contiguous memory, tensor storage). (LLM, conf=0.73)
Experimental — accuracy not yet measuredExisting LLM serving systems store the KV cache of a request in contiguous memory space, which causes inefficient memory management.
“This is mainly because they store the KV cache of a request in contiguous memory space, as most deep learning frameworks [33,39] require tensors to be stored in contiguous memory.”
From Efficient Memory Management for Large Language Model Serving with PagedAttention (2023) · cites this paper as [31]
Section 2.2.1 (p. 5) describes concatenating input tensors to batch inference requests, but does not say tensors must occupy contiguous memory. Searches for KV caches and contiguous tensor storage found no passage addressing the claimed memory-management problem. (LLM, conf=0.76)
Experimental — accuracy not yet measuredFasterTransformer does not utilize a fine-grained scheduling mechanism and inefficiently manages memory like Orca (Max).
“as FasterTransformer does not utilize a fine-grained scheduling mechanism and inefficiently manages the memory like Orca (Max).”
From Efficient Memory Management for Large Language Model Serving with PagedAttention (2023) · cites this paper as [31]
The Introduction (p. 1) and §2.2.1 (p. 5) discuss scheduling and memory considerations for TensorFlow-Serving, not FasterTransformer or Orca (Max). Searches for FasterTransformer, Orca (Max), and fine-grained scheduling found no passage evaluating their scheduling or memory management. (LLM, conf=0.78)
Experimental — accuracy not yet measuredExisting systems statically allocate a chunk of memory for a request based on the request's maximum possible sequence length.
“Due to the unpredictable output lengths from the LLM, they statically allocate a chunk of memory for a request based on the request's maximum possible sequence length, irrespective of the actual input or eventual output length of the request.”
From Efficient Memory Management for Large Language Model Serving with PagedAttention (2023) · cites this paper as [31]
Section 2.2.1 (p. 5) discusses batching inference requests, while the Introduction (p. 1) discusses managing memory for loaded models. Neither addresses static memory allocation per request based on maximum sequence length or unpredictable LLM output length. Searches for those terms and their paraphrases found no supporting passage in our copy. (LLM, conf=0.75)
Experimental — accuracy not yet measuredExisting LLM serving systems suffer from internal and external memory fragmentation.
“First, the existing systems [31,60] suffer from internal and external memory fragmentation.”
From Efficient Memory Management for Large Language Model Serving with PagedAttention (2023) · cites this paper as [31]
The Introduction (p. 1) discusses managing server RAM as models grow, and §2.1.2 (p. 4) describes unloading models to free memory. Neither passage addresses internal or external memory fragmentation, and searches for those terms found no passage that does. The copy also does not discuss LLM serving systems. (LLM, conf=0.72)
Experimental — accuracy not yet measured
vLLM smooths out the rapid growth curve of KV cache memory seen in existing systems, leading to a notable boost in serving throughput.
“vLLM smooths out the rapid growth curve of KV cache memory seen in existing systems[31,60], leading to a notable boost in serving throughput.”
From Efficient Memory Management for Large Language Model Serving with PagedAttention (2023) · cites this paper as [31]
The Introduction (p. 1) discusses server RAM and model-loading costs, while Inter-Request Batching (p. 5) discusses throughput gains from batching. Searches for vLLM and KV-cache memory growth found no passage addressing the claimed memory behavior or its effect on throughput. (LLM, conf=0.76)
Experimental — accuracy not yet measuredGeneral model serving systems fail to take into account the autoregressive property and token state of LLM inference, resulting in missed opportunities for optimization.
“However, these general systems fail to take into account the autoregressive property and token state of LLM inference, resulting in missed opportunities for optimization.”
From Efficient Memory Management for Large Language Model Serving with PagedAttention (2023) · cites this paper as [33]
The Inference section (p. 4) discusses planned APIs for sequence models, and Inter-Request Batching (p. 5) describes batching within a sequence model’s loop. Neither passage addresses LLM token state or concludes that general serving systems miss optimizations because they ignore it (searched: autoregressive token state, sequence-model decoding and batching). (LLM, conf=0.76)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
No references could be resolved yet.
Research works printed but not linked: 11
- [2][2] E. Breck, S. Cai, E. Nielsen, M. Salib, and D. Sculley. What's your ML test score? A rubric for ML production systems. In NIPS Workshop on Reliable Machine Learning in the Wild , 2016. -
- [7][7] D. Crankshaw, X. Wang, G. Zhou, M. J. Franklin, J. E. Gonzalez, and I. Stoica. Clipper: A Low-Latency Online Prediction Serving System. In 14th USENIX Symposium on Networked Systems Design and Implementation (NSDI) , 2017. -
- [8][8] D. Agarwal, B. Long, J. Traupman, D. Xin, and L. Zhang. LASER: A Scalable Response Prediction Platform for Online Advertising. In 7th ACM International Conference on Web Search and Data Mining , 2014. -
- [9][9] D. Crankshaw, P. Bailis, J. E. Gonzalez, H. Li, Z. Zhang, M. J. Franklin, A. Ghodsi, and M. I. Jordan. The Missing Piece in Complex Analytics: Low Latency, Scalable Model Management and Serving with Velox. In 7th Biennial Conference on Innovative Data Systems Research , 2015. -
- [11][11] V. S. Pai, P. Druschel, and W. Zwaenepoel. Flash: An efficient and portable Web server. In USENIX Annual Technical Conference , 1999. -
- [12][12] D. C. Schmidt. Pattern languages of program design , chapter Reactor: An Object Behavioral Pattern for Concurrent Event Demultiplexing and Event Handler Dispatching, pages 529-545. 1995. -
- [13][13] M. Welsh, D. E. Culler, and E. A. Brewer. SEDA: An Architecture for Well-Conditioned, Scalable Internet Services. In 18th Symposium on Operating Systems Principles , 2001. -
- [14][14] N. P. Jouppi et al. In-Datacenter Performance Analysis of a Tensor Processing Unit. In 44th International Symposium on Computer Architecture (ISCA) , 2017. -
- [15][15] D. Baylor, E. Breck, H.-T. Cheng, N. Fiedel, C. Y. Foo, Z. Haque, S. Haykal, M. Ispir, V. Jain, L. Koc, C. Y. Koo, L. Lew, C. Mewald, A. Modi, N. Polyzotis, S. Ramesh, S. Roy, S. Whang, M. Wicke, J. Wilkiewicz, X. Zhang, and M. Zinkevich. The Anatomy of a Production-Scale Continuously-Training Machine Learning Platform. In 23rd SIGKDD Conference on Knowledge Discovery and Data Mining , 2017. -
- [19][19] G. Neubig, Y. Goldberg, and C. Dyer. On-the-fly Operation Batching in Dynamic Computation Graphs. https://arxiv.org/pdf/1705.07860.pdf. - · arXiv
- [20][20] J. C. Corbett, J. Dean, M. Epstein, A. Fikes, C. Frost, J. Furman, S. Ghemawat, A. Gubarev, C. Heiser, P. Hochschild, W. Hsieh, S. Kanthak, E. Kogan, H. Li, A. Lloyd, S. Melnik, D. Mwaura, D. Nagle, S. Quinlan, R. Rao, L. Rolig, Y. Saito, M. Szymaniak, C. Taylor, R. Wang, and D. Woodford. Spanner: Google's Globally-Distributed Database. In 10th USENIX Symposium on Operating Systems Design and Implementation , 2012. -
Cited by
Sorted by citations
Efficient Memory Management for Large Language Model Serving with PagedAttention
Kwon, W., Li, Z., Zhuang, S., Sheng, Y., Zheng, L., Yu, C. H., Gonzalez, J., Zhang, H., & Stoica, I. (2023) · Proceedings of the 29th Symposium on Operating Systems Principles · doi:10.1145/3600006.3613165 · Cited by 1,817
Low latency RNN inference with cellular batching
Gao, P., Yu, L., Wu, Y., & Li, J. (2018) · Proceedings of the Thirteenth EuroSys Conference · doi:10.1145/3190508.3190541 · Cited by 66
On the Cost of Model-Serving Frameworks: An Experimental Evaluation
De Rosa, P., Bromberg, Y.-D., Felber, P., Mvondo, D., & Schiavoni, V. (2024) · 2024 IEEE International Conference on Cloud Engineering (IC2E) · doi:10.1109/ic2e61754.2024.00032 · Cited by 6
“This paper aims to fill this existing gap by benchmarking five state-of-the-art open-source model serving frameworks, namely TensorFlow-Serving [26], [66], TorchServe [31], MLServer [20], MLflow [19], [82] and BentoML [4].”
Metadata
- Authors
- Olston, C., Fiedel, N., Gorovoy, K., Harmsen, J., Lao, L., Li, F., Rajashekhar, V., Ramesh, S., & Soyke, J.
- ORCID
- Li, Fangwei
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2017
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 1712.06139
- OpenAlex
- W2779888504
- Cited by
- 171
- References
- 0
- Citations by year
- 2025: 10 · 2024: 22 · 2023: 23 · 2022: 23 · 2021: 25 · 2020: 39 · 2019: 22 · 2018: 6 · 2017: 1
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Information Systems › Cloud Computing and Resource Management
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Computer Science › Computer Networks and Communications › Software System Performance and Reliability
- Keywords
- model serving, model inference, production deployment, model versioning, cloud machine learning
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
