CoChat
Sign Up
Cited by 23Parallel Computing and Optimization TechniquesAdvanced Neural Network ApplicationsTopic Modeling

$S^3$: Increasing GPU Utilization during Generative Inference for Higher Throughput

Open access
Jin, Yunho, Wu, Chun-Feng, Brooks, David, Wei, Gu-Yeon·2023·Advances in Neural Information Processing Systems 36
DOI 10.52202/075280-0791
Open this paper in CoChat — free accountOpen PDF

Abstract

Generating texts with a large language model (LLM) consumes massive amounts of memory. Apart from the already-large model parameters, the key/value (KV) cache that holds information about previous tokens in a sequence can grow to be even larger than the model itself. This problem is exacerbated in one of the current LLM serving frameworks which reserves the maximum sequence length of memory for the KV cache to guarantee generating a complete sequence as they do not know the output sequence length. This restricts us to use a smaller batch size leading to lower GPU utilization and above all, lower throughput. We argue that designing a system with a priori knowledge of the output sequence can mitigate this problem. To this end, we propose S$^{3}$, which predicts the output sequence length, schedules generation queries based on the prediction to increase device resource utilization and throughput, and handle mispredictions. Our proposed method achieves 6.49$\times$ throughput over those systems that assume the worst case for the output sequence length.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.52202/075280-0791

Cites

0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • The bucketing strategy transforms precise token-level prediction into a more tractable bucket-level approximation.

    “The bucketing strategy, a technique also employed in previous works [25; 57; 14], transforms the challenging task of precise token-level prediction into a more tractable bucket-level approximation.”

    From OmniRouter: Budget and Performance Controllable Multi-LLM Routing (2025) · cites this paper as [25]

  • Subsequent research has explored two main directions of prediction models: encoder-only models for classification and decoder-only models for generative prediction.

    “Subsequent research has explored two main directions of prediction models: encoder-only models for classification (DynamoLLM [42], S3 [25], TerriInfer [20], SSJF [37], and µ 3 [36]) and decoder-only models for generative prediction like Perception-only (PO) [57].”

    From OmniRouter: Budget and Performance Controllable Multi-LLM Routing (2025) · cites this paper as [25]

  • Predicting LLM generation length is crucial for optimizing computational resources.

    “Predicting LLM generation length is crucial for optimizing computational resources.”

    From OmniRouter: Budget and Performance Controllable Multi-LLM Routing (2025) · cites this paper as [25]

  • Existing routing frameworks typically model routing as a sequence of independent, greedy per-query decisions, overlooking global budget constraints and failing to achieve Pareto-efficient resource allocation.

    “Existing routing frameworks typically model this as a locally optimal decision-making problem, selecting the presumed best-fit LLM for each query individually, which overlooks global budget constraints, resulting in ineffective resource allocation.”

    From OmniRouter: Budget and Performance Controllable Multi-LLM Routing (2025) · cites this paper as [25]

    The Abstract and §3 (Design; page unavailable) describe predicting output length and batching generation requests within GPU memory capacity, not choosing a best-fit LLM for each query. A search for model routing, per-query decisions, and global budget constraints found no passage addressing the attributed claim or Pareto-efficient allocation. (LLM, conf=0.78)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

The publisher hasn’t deposited this paper’s reference list.

Research works printed but not linked: 35

  1. D. Crankshaw, X. Wang, G. Zhou, M. J. Franklin, J. E. Gonzalez, and I. Stoica, "Clipper: A Low-Latency online prediction serving system," in 14th USENIX Symposium on Networked Systems Design and Implementation (NSDI 17). Boston, MA: USENIX Association, Mar. 2017, pp. 613-627. [Online]. Available: https://www.usenix.org/conference/nsdi17/ technical-sessions/presentation/crankshaw
  2. R. Chard, Z. Li, K. Chard, L. Ward, Y. Babuji, A. Woodard, S. Tuecke, B. Blaiszik, M. J. Franklin, and I. Foster, "Dlhub: Model and data serving for science," in 2019 IEEE Interna- tional Parallel and Distributed Processing Symposium (IPDPS), 2019, pp. 283-292.
  3. Y. Sheng, L. Zheng, B. Yuan, Z. Li, M. Ryabinin, D. Y. Fu, Z. Xie, B. Chen, C. Barrett, J. E. Gonzalez, P. Liang, C. Ré, I. Stoica, and C. Zhang, "High-throughput Generative Inference of Large Language Models with a Single GPU," arXiv e-prints, p. arXiv:2303.06865, Mar. 2023. · arXiv
  4. P. Liang, R. Bommasani, T. Lee, D. Tsipras, D. Soylu, M. Yasunaga, Y. Zhang, D. Narayanan, Y. Wu, A. Kumar, B. Newman, B. Yuan, B. Yan, C. Zhang, C. Cosgrove, C. D. Manning, C. Ré, D. Acosta-Navas, D. A. Hudson, E. Zelikman, E. Durmus, F. Ladhak, F. Rong, H. Ren, H. Yao, J. Wang, K. Santhanam, L. Orr, L. Zheng, M. Yuksekgonul, M. Suzgun, N. Kim, N. Guha, N. Chatterji, O. Khattab, P. Henderson, Q. Huang, R. Chi, S. M. Xie, S. Santurkar, S. Ganguli, T. Hashimoto, T. Icard, T. Zhang, V. Chaudhary, W. Wang, X. Li, Y. Mai, Y. Zhang, and Y. Koreeda, "Holistic Evaluation of Language Models," arXiv e-prints, p. arXiv:2211.09110, Nov. 2022. · arXiv
  5. A. Narayan, I. Chami, L. Orr, S. Arora, and C. Ré, "Can Foundation Models Wrangle Your Data?" arXiv e-prints, p. arXiv:2205.09911, May 2022. · arXiv
  6. M. Brysbaert, "How many words do we read per minute? a review and meta-analysis of reading rate," Journal of Memory and Language, vol. 109, p. 104047, 2019. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S0749596X19300786
  7. S. Black, S. Biderman, E. Hallahan, Q. Anthony, L. Gao, L. Golding, H. He, C. Leahy, K. McDonell, J. Phang, M. Pieler, U. S. Prashanth, S. Purohit, L. Reynolds, J. Tow, B. Wang, and S. Weinbach, "GPT-NeoX-20B: An open-source autoregressive language model," in Proceedings of BigScience Episode #5 -Workshop on Challenges & Perspectives in Creating Large Language Models. virtual+Dublin: Association for Computational Linguistics, May 2022, pp. 95-136. [Online]. Available: https://aclanthology.org/2022.bigscience-1.9
  8. G.-I. Yu, J. S. Jeong, G.-W. Kim, S. Kim, and B.-G. Chun, "Orca: A distributed serving system for Transformer-Based generative models," in 16th USENIX Symposium on Operating Systems Design and Implementation (OSDI 22). Carlsbad, CA: USENIX Association, Jul. 2022, pp. 521-538. [Online]. Available: https://www.usenix.org/conference/osdi22/presentation/yu
  9. V. Sanh, L. Debut, J. Chaumond, and T. Wolf, "DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter," arXiv e-prints, p. arXiv:1910.01108, Oct. 2019. · arXiv
  10. L. Gao, S. Biderman, S. Black, L. Golding, T. Hoppe, C. Foster, J. Phang, H. He, A. Thite, N. Nabeshima, S. Presser, and C. Leahy, "The Pile: An 800GB Dataset of Diverse Text for Language Modeling," p. arXiv:2101.00027, Dec. 2020. · arXiv
  11. D. S. Johnson, "Near-optimal bin packing algorithms," Ph.D. dissertation, Massachusetts In- stitute of Technology, 1973.
  12. A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, A. Desmaison, A. Kopf, E. Yang, Z. De- Vito, M. Raison, A. Tejani, S. Chilamkurthy, B. Steiner, L. Fang, J. Bai, and S. Chintala, "Pytorch: An imperative style, high-performance deep learning library," in Advances in Neural Information Processing Systems 32.
  13. M. Abadi, P. Barham, J. Chen, Z. Chen, A. Davis, J. Dean, M. Devin, S. Ghemawat, G. Irving, M. Isard et al., "Tensorflow: A system for large-scale machine learning," in 12th {USENIX} Symposium on Operating Systems Design and Implementation ({OSDI} 16), 2016, pp. 265- 283.
  14. F. Romero, Q. Li, N. J. Yadwadkar, and C. Kozyrakis, "INFaaS: Automated model- less inference serving," in 2021 USENIX Annual Technical Conference (USENIX ATC 21). USENIX Association, Jul. 2021, pp. 397-411. [Online]. Available: https: //www.usenix.org/conference/atc21/presentation/romero
  15. A. Gujarati, R. Karimi, S. Alzayat, W. Hao, A. Kaufmann, Y. Vigfusson, and J. Mace, "Serving DNNs like clockwork: Performance predictability from the bottom up," in 14th USENIX Symposium on Operating Systems Design and Implementation (OSDI 20). USENIX Association, Nov. 2020, pp. 443-462. [Online]. Available: https://www.usenix.org/conference/osdi20/presentation/gujarati
  16. S. Choi, S. Lee, Y. Kim, J. Park, Y. Kwon, and J. Huh, "Serving heterogeneous machine learning models on Multi-GPU servers with Spatio-Temporal sharing," in 2022 USENIX Annual Technical Conference (USENIX ATC 22). Carlsbad, CA: USENIX Association, Jul. 2022, pp. 199-216. [Online]. Available: https://www.usenix.org/conference/atc22/ presentation/choi-seungbeom
  17. R. Yazdani Aminabadi, S. Rajbhandari, M. Zhang, A. A. Awan, C. Li, D. Li, E. Zheng, J. Rasley, S. Smith, O. Ruwase, and Y. He, "DeepSpeed Inference: Enabling Efficient In- ference of Transformer Models at Unprecedented Scale," arXiv e-prints, p. arXiv:2207.00032, Jun. 2022. · arXiv
  18. U. Gupta, S. Hsia, V. Saraph, X. Wang, B. Reagen, G.-Y. Wei, H.-H. S. Lee, D. Brooks, and C.- J. Wu, "DeepRecSys: A System for Optimizing End-To-End At-scale Neural Recommendation Inference," arXiv e-prints, p. arXiv:2001.02772, Jan. 2020. · arXiv
  19. S. Hsia, U. Gupta, B. Acun, N. Ardalani, P. Zhong, G.-Y. Wei, D. Brooks, and C.-J. Wu, "Mp-rec: Hardware-software co-design to enable multi-path recommendation," in Proceedings of the 28th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 3, ser. ASPLOS 2023. New York, NY, USA: Association for Computing Machinery, 2023, p. 449-465. [Online]. Available: https://doi.org/10.1145/3582016.3582068 · DOI
  20. J. Fang, Y. Yu, C. Zhao, and J. Zhou, "TurboTransformers: An Efficient GPU Serving System For Transformer Models," arXiv e-prints, p. arXiv:2010.05680, Oct. 2020. · arXiv
  21. C.-F. Wu, C.-J. Wu, G.-Y. Wei, and D. Brooks, "A joint management middleware to improve training performance of deep recommendation systems with ssds," in Proceedings of the 59th ACM/IEEE Design Automation Conference (DAC 22), 2022, pp. 157-162.
  22. C. Olston, N. Fiedel, K. Gorovoy, J. Harmsen, L. Lao, F. Li, V. Rajashekhar, S. Ramesh, and J. Soyke, "TensorFlow-Serving: Flexible, High-Performance ML Serving," arXiv e-prints, p. arXiv:1712.06139, Dec. 2017. · arXiv
  23. R. Pope, S. Douglas, A. Chowdhery, J. Devlin, J. Bradbury, A. Levskaya, J. Heek, K. Xiao, S. Agrawal, and J. Dean, "Efficiently scaling transformer inference," arXiv preprint arXiv:2211.05102, 2022. · arXiv
  24. X. Wang, Y. Xiong, Y. Wei, M. Wang, and L. Li, "LightSeq: A High Performance Inference Library for Transformers," arXiv e-prints, p. arXiv:2010.13887, Oct. 2020. · arXiv
  25. K. Choromanski, V. Likhosherstov, D. Dohan, X. Song, A. Gane, T. Sarlos, P. Hawkins, J. Davis, A. Mohiuddin, L. Kaiser, D. Belanger, L. Colwell, and A. Weller, "Rethinking At- tention with Performers," arXiv e-prints, p. arXiv:2009.14794, Sep. 2020. · arXiv
  26. V. Likhosherstov, K. Choromanski, J. Davis, X. Song, and A. Weller, "Sub-Linear Memory: How to Make Performers SLiM," arXiv e-prints, p. arXiv:2012.11346, Dec. 2020. · arXiv
  27. I. Beltagy, M. E. Peters, and A. Cohan, "Longformer: The Long-Document Transformer," arXiv e-prints, p. arXiv:2004.05150, Apr. 2020. · arXiv
  28. M. Zaheer, G. Guruganesh, A. Dubey, J. Ainslie, C. Alberti, S. Ontanon, P. Pham, A. Ravula, Q. Wang, L. Yang, and A. Ahmed, "Big Bird: Transformers for Longer Sequences," arXiv e-prints, p. arXiv:2007.14062, Jul. 2020. · arXiv
  29. H. Wang, Z. Zhang, and S. Han, "SpAtten: Efficient Sparse Attention Architecture with Cas- cade Token and Head Pruning," arXiv e-prints, p. arXiv:2012.09852, Dec. 2020. · arXiv
  30. N. Shazeer, "Fast Transformer Decoding: One Write-Head is All You Need," arXiv e-prints, p. arXiv:1911.02150, Nov. 2019. · arXiv
  31. J. W. Rae, A. Potapenko, S. M. Jayakumar, and T. P. Lillicrap, "Compressive Transformers for Long-Range Sequence Modelling," arXiv e-prints, p. arXiv:1911.05507, Nov. 2019. · arXiv
  32. S. Kim, A. Gholami, Z. Yao, M. W. Mahoney, and K. Keutzer, "I-BERT: Integer-only BERT Quantization," arXiv e-prints, p. arXiv:2101.01321, Jan. 2021. · arXiv
  33. Z. Liu, Y. Wang, K. Han, W. Zhang, S. Ma, and W. Gao, "Post-training quantization for vision transformer," in Advances in Neural Information Processing Systems, M. Ranzato, A. Beygelzimer, Y. Dauphin, P. Liang, and J. W. Vaughan, Eds., vol. 34. Curran Associates, Inc., 2021, pp. 28 092-28 103. [Online]. Available: https://proceedings.neurips.cc/paper files/ paper/2021/file/ec8956637a99787bd197eacd77acce5e-Paper.pdf
  34. Z. Yang, Y. Gao, W. Wang, and H. Ney, "Predicting and using target length in neural machine translation," in Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th International Joint Conference on Natural Language Processing. Suzhou, China: Association for Computational Linguistics, Dec. 2020, pp. 389-395. [Online]. Available: https://aclanthology.org/2020.aacl-main.41
  35. V. Janapa Reddi, C. Cheng, D. Kanter, P. Mattson, G. Schmuelling, C.-J. Wu, B. Anderson, M. Breughe, M. Charlebois, W. Chou, R. Chukka, C. Coleman, S. Davis, P. Deng, G. Diamos, J. Duke, D. Fick, J. S. Gardner, I. Hubara, S. Idgunji, T. B. Jablin, J. Jiao, T. St. John, P. Kan- war, D. Lee, J. Liao, A. Lokhmotov, F. Massa, P. Meng, P. Micikevicius, C. Osborne, G. Pekhi- menko, A. Tejusve Raghunath Rajan, D. Sequeira, A. Sirasao, F. Sun, H. Tang, M. Thomson, F. Wei, E. Wu, L. Xu, K. Yamada, B. Yu, G. Yuan, A. Zhong, P. Zhang, and Y. Zhou, "MLPerf Inference Benchmark," arXiv e-prints, p. arXiv:1911.02549, Nov. 2019. · arXiv

Cited by

Sorted by citations

Related

Authors
Jin, Y., Wu, C.-F., Brooks, D., & Wei, G.-Y.
ORCID
Wei, Gu-Yeon
Venue
Advances in Neural Information Processing Systems 36
Publisher
Neural Information Processing Systems Foundation, Inc. (NeurIPS)
Published
2023
Pages
18015-18027
ISSN
1049-5258 (ISSN-L)
DOI
10.52202/075280-0791
OpenAlex
W7133238857
Cited by
20
References
0
Citations by year
2026: 13 · 2025: 4 · 2024: 3
FWCI
3.58
Type
conference paper
Language
en
Open access
Licence
cc-by
Locations
  • neural information processing systems (publishedVersion)
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • arXiv (Cornell University) (cc-by)
Topics
  • Physical Sciences › Computer Science › Hardware and Architecture › Parallel Computing and Optimization Techniques
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
Keywords
GPU utilization, generative inference, inference throughput, large language models, GPU scheduling

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review