CoChat
Sign Up
Cited by 15Natural Language Processing TechniquesTopic ModelingParallel Computing and Optimization Techniques

SPIN: Accelerating Large Language Model Inference with Heterogeneous Speculative Models

Open access
Chen, Fahao, Li, Peng, Luan, Tom H., Su, Zhou, Deng, Jing·2025·IEEE INFOCOM 2025 - IEEE Conference on Computer Communications
DOI 10.1109/infocom55648.2025.11044522
Open this paper in CoChat — free accountOpen PDF

Abstract

Speculative decoding has been shown as an effective way to accelerate Large Language Model (LLM) inference by using a Small Speculative Model (SSM) to generate candidate tokens in a so-called speculation phase, which are subsequently verified by the LLM in a verification phase. However, current state-of-the-art speculative decoding approaches have three key limitations: handling requests with varying difficulty using homogeneous SSMs, lack of robust support for batch processing, and insufficient holistic optimization for both speculation and verification phases. In this paper, we introduce Spin, an efficient LLM inference serving system based on speculative decoding, designed to address these challenges through three main innovations. First, SPIN improves token speculation by using multiple heterogeneous SSMs, with a learning-based algorithm for SSM selection that operates without prior knowledge of request difficulty. Second, SPIN employs a request decomposition method to minimize batching overhead during LLM verification. Finally, SPIN orchestrates speculation and verification phases by pipelining their executions on GPUs to achieve further acceleration. Experimental results demonstrate that SPIN significantly outperforms state-of-the-art methods, achieving a performance increase of approximately 2.28×.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1109/infocom55648.2025.11044522

Cites

0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

The publisher hasn’t deposited this paper’s reference list.

Research works printed but not linked: 12

  1. [1][1] T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell et al. , 'Language models are few-shot learners,' Advances in neural information processing systems , vol. 33, pp. 1877-1901, 2020. -
  2. [6][6] Y. Wang, K. Chen, H. Tan, and K. Guo, 'Tabi: An efficient multilevel inference system for large language models,' in Proceedings of the Eighteenth European Conference on Computer Systems , 2023, pp. 233-248. -
  3. [7][7] Z. Liu, J. Wang, T. Dao, T. Zhou, B. Yuan, Z. Song, A. Shrivastava, C. Zhang, Y. Tian, C. Re et al. , 'Deja vu: Contextual sparsity for efficient llms at inference time,' in International Conference on Machine Learning . PMLR, 2023, pp. 22 137-22 176. -
  4. [8][8] A. Agrawal, N. Kedia, A. Panwar, J. Mohan, N. Kwatra, B. Gulavani, A. Tumanov, and R. Ramjee, 'Taming throughput-latency tradeoff in llm inference with sarathi-serve,' in 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24) . USENIX Association, 2024, pp. 117-134. -
  5. [9][9] Y. Zhong, S. Liu, J. Chen, J. Hu, Y. Zhu, X. Liu, X. Jin, and H. Zhang, 'Distserve: Disaggregating prefill and decoding for goodputoptimized large language model serving,' in 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24) . USENIX Association, 2024, pp. 193-210. -
  6. [10][10] B. Sun, Z. Huang, H. Zhao, W. Xiao, X. Zhang, Y. Li, and W. Lin, 'Llumnix: Dynamic scheduling for large language model serving,' in 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24) . USENIX Association, 2024, pp. 173-191. -
  7. [11][11] W. Lee, J. Lee, J. Seo, and J. Sim, 'Infinigen: Efficient generative inference of large language models with dynamic KV cache management,' in 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24) . USENIX Association, 2024, pp. 155-172. -
  8. [14][14] Y. Leviathan, M. Kalman, and Y. Matias, 'Fast inference from transformers via speculative decoding,' in International Conference on Machine Learning . PMLR, 2023, pp. 19 274-19 286. -
  9. [22][22] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, 'Attention is all you need,' Advances in neural information processing systems , vol. 30, 2017. -
  10. [24][24] G.-I. Yu, J. S. Jeong, G.-W. Kim, S. Kim, and B.-G. Chun, 'Orca: A distributed serving system for transformer-based generative models,' in 16th USENIX Symposium on Operating Systems Design and Implementation (OSDI 22) , 2022, pp. 521-538. -
  11. [25][25] B. Sun, Z. Huang, H. Zhao, W. Xiao, X. Zhang, Y. Li, and W. Lin, 'Llumnix: Dynamic scheduling for large language model serving,' in 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24) . USENIX Association, 2024, pp. 173-191. -
  12. [30][30] B. Gao, Z. He, P. Sharma, Q. Kang, D. Jevdjic, J. Deng, X. Yang, Z. Yu, and P. Zuo, 'Cost-Efficient large language model serving for multi-turn conversations with CachedAttention,' in 2024 USENIX Annual Technical Conference (USENIX ATC 24) , 2024, pp. 111-126. -

Cited by

Sorted by citations

Related

Authors
Chen, F., Li, P., Luan, T. H., Su, Z., & Deng, J.
Affiliations
  • University of Aizu (JP)
  • Xi'an Jiaotong University (CN)
  • University of North Carolina at Greensboro (US)
ORCID
Venue
IEEE INFOCOM 2025 - IEEE Conference on Computer Communications
Publisher
IEEE
Published
2025-05-19
Pages
1-10
DOI
10.1109/infocom55648.2025.11044522
OpenAlex
W7084082117
Cited by
11
References
0
Citations by year
2026: 11
FWCI
10.49
Type
conference paper
Language
en
Open access
Licence
Locations
  • IEEE Conference on Computer Communications (INFOCOM) (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University) (cc-by)
Funders
  • Japan Society for the Promotion of Science (23KJ1786)
  • Japan Science and Technology Agency (JST) PRESTO (23828673)
  • National Natural Science Foundation of China (62471383,U23A20276,U22A2029)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Hardware and Architecture › Parallel Computing and Optimization Techniques
Keywords
speculative decoding, batch processing

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review