The DOI matches its Crossref record; 23 of the 24 references it cites were found in a registry or an index.
SpecInfer: Accelerating Large Language Model Serving with Tree-based Speculative Inference and Verification
Open accessAbstract
This paper introduces SpecInfer, a system that accelerates generative large language model (LLM) serving with tree-based speculative inference and verification. The key idea behind SpecInfer is leveraging small speculative models to predict the LLM's outputs; the predictions are organized as a token tree, whose nodes each represent a candidate token sequence. The correctness of all candidate token sequences represented by a token tree is verified against the LLM in parallel using a novel tree-based parallel decoding mechanism. SpecInfer uses an LLM as a token tree verifier instead of an incremental decoder, which significantly reduces the end-to-end latency and computational requirement for serving generative LLMs while provably preserving model quality. Our evaluation shows that SpecInfer outperforms existing LLM serving systems by 1.5-2.8× for distributed LLM inference and by 2.6-3.5× for offloading-based LLM inference, while preserving the same generative performance. SpecInfer is publicly available at https://github.com/flexflow/FlexFlow/
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Existing tree-based works (Luo et al. 2025; Miao et al. 2024; Chen et al. 2024) employ static verification trees that lack adaptability by static structures, which cannot dynamically adjust to the changing context of the generated text, leading to suboptimal draft acceptance rates and limited speedup.”
From DIAA: A Decoding-Efficient Inference Acceleration Approach for On-Device Large Language Models (2026) · cites this paper as Miao et al. 2024
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“However, static token trees, most existing work adopted (Miao et al. 2024; Luo et al. 2025; Chen et al. 2024), are constructed once using fixed Top-K sampling and remain unchanged.”
From DIAA: A Decoding-Efficient Inference Acceleration Approach for On-Device Large Language Models (2026) · cites this paper as Miao et al. 2024
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
Computer Architecture: A Quantitative Approach
Hennessy, J. L., & Patterson, D. A. (1989) · UCLA - Biblioteca de Ciencias y Tecnologia Felix Morales Bueno · Cited by 9,434
LLaMA: Open and Efficient Foundation Language Models
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. (2023) · arXiv · doi:10.48550/arxiv.2302.13971 · Cited by 4,602
Computer architecture: a quantitative approach
Shute, M. (1993) · Microelectronics Journal · doi:10.1016/0026-2692(93)90111-q · Cited by 3,349
A Short Introduction to Boosting
Freund, Y., & Schapire, R. E. (1999) · Cited by 2,991
Ensemble deep learning: A review
Ganaie, M. A., Hu, M., Malik, A. K., Tanveer, M., & Suganthan, P. N. (2022) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2022.105151 · Cited by 2,446
Semantic Parsing on Freebase from Question-Answer Pairs
Berant, J., Chou, A., Frostig, R., & Liang, P. (2013) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d13-1160 · Cited by 1,593
TVM: an automated end-to-end optimizing compiler for deep learning
(2018) · arXiv (Cornell University) · doi:10.5555/3291168.3291211 · Cited by 1,170
Flagged first, then by citations
1 of the 24 references couldn’t be matched to a known paper.
Research works printed but not linked: 29
- Charlie Chen, Sebastian Borgeaud, Geoffrey Irving, Jean-Baptiste Lespiau, Laurent Sifre, and John Jumper. Accelerating large language model decoding with speculative sampling. arXiv preprint arXiv:2302.01318, 2023. · arXiv
- Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311, 2022. · arXiv
- Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. Gpt3. int8 (): 8-bit matrix multiplication for transformers at scale. Advances in Neural Information Processing Systems, 35:30318--30332, 2022.
- Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. Gptq: Accurate quantization for generative pre-trained transformers. In International Conference on Learning Representations, 2023.
- Freddy Gabbay and Avi Mendelson. Speculative execution based on value prediction. Citeseer, 1996.
- Joao Gante. Assisted generation: a new direction toward low-latency text generation, 2023.
- Sehoon Kim, Karttikeya Mangalam, Suhong Moon, John Canny, Jitendra Malik, Michael W. Mahoney, Amir Gholami, and Kurt Keutzer. Big little transformer decoder, 2023.
- Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Yu, Joseph E Gonzalez, Hao Zhang, and Ion Stoica. vllm: Easy, fast, and cheap llm serving with pagedattention. See https://vllm.ai/ (accessed 9 August 2023), 2023.
- Yaniv Leviathan, Matan Kalman, and Yossi Matias. Fast inference from transformers via speculative decoding. arXiv preprint arXiv:2211.17192, 2022. · arXiv
- Jiachang Liu, Dinghan Shen, Yizhe Zhang, Bill Dolan, Lawrence Carin, and Weizhu Chen. What makes good in-context examples for gpt-3? arXiv preprint arXiv:2101.06804, 2021. · arXiv
- Xupeng Miao, Gabriele Oliaro, Zhihao Zhang, Xinhao Cheng, Hongyi Jin, Tianqi Chen, and Zhihao Jia. Towards efficient generative large language model serving: A survey from algorithms to systems. arXiv preprint arXiv:2312.15234, 2023. · arXiv
- Xupeng Miao, Gabriele Oliaro, Zhihao Zhang, Xinhao Cheng, Zeyu Wang, Rae Ying Yee Wong, Zhuoming Chen, Daiyaan Arfeen, Reyna Abhyankar, and Zhihao Jia. Specinfer: Accelerating generative llm serving with speculative inference and token tree verification. arXiv preprint arXiv:2305.09781, 2023. · arXiv
- Xupeng Miao, Chunan Shi, Jiangfei Duan, Xiaoli Xi, Dahua Lin, Bin Cui, and Zhihao Jia. Spotserve: Serving generative large language models on preemptible instances. arXiv preprint arXiv:2311.15566, 2023. · arXiv
- OpenAI. Gpt-4 technical report, 2023.
- Gunho Park, Baeseong Park, Se Jung Kwon, Byeongwook Kim, Youngjoo Lee, and Dongsoo Lee. nuqmm: Quantized matmul for efficient inference of large-scale generative language models. arXiv preprint arXiv:2206.09557, 2022. · arXiv
- Baolin Peng, Chunyuan Li, Pengcheng He, Michel Galley, and Jianfeng Gao. Instruction tuning with gpt-4. arXiv preprint arXiv:2304.03277, 2023. · arXiv
- Jack W Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, et al. Scaling language models: Methods, analysis & insights from training gopher. arXiv preprint arXiv:2112.11446, 2021. · arXiv
- Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilić, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, et al. Bloom: A 176b-parameter open-access multilingual language model. arXiv preprint arXiv:2211.05100, 2022. · arXiv
- Ying Sheng, Lianmin Zheng, Binhang Yuan, Zhuohan Li, Max Ryabinin, Daniel Y. Fu, Zhiqiang Xie, Beidi Chen, Clark Barrett, Joseph E. Gonzalez, Percy Liang, Christopher Ré, Ion Stoica, and Ce Zhang. High-throughput generative inference of large language models with a single gpu, 2023.
- James E Smith. A study of branch prediction strategies. In 25 years of the international symposia on Computer architecture (selected papers), pages 202--215, 1998.
- Shaden Smith, Mostofa Patwary, Brandon Norick, Patrick LeGresley, Samyam Rajbhandari, Jared Casper, Zhun Liu, Shrimai Prabhumoye, George Zerveas, Vijay Korthikanti, et al. Using deepspeed and mega-tron to train megatron-turing nlg 530b, a large-scale generative language model. arXiv preprint arXiv:2201.11990, 2022. · arXiv
- Colin Unger, Zhihao Jia, Wei Wu, Sina Lin, Mandeep Baines, Carlos Efrain Quintero Narvaez, Vinay Ramakrishnaiah, Nirmal Prajapati, Pat McCormick, Jamaludin Mohd-Yusof, Xi Luo, Dheevatsa Mudigere, Jongsoo Park, Misha Smelyanskiy, and Alex Aiken. Unity: Accelerating DNN training through joint optimization of algebraic transformations and parallelization. In 16th USENIX Symposium on Operating Systems Design and Implementation (OSDI 22), pages 267--284, Carlsbad, CA, July 2022. USENIX Association.
- Guangxuan Xiao, Ji Lin, Mickael Seznec, Julien Demouth, and Song Han. Smoothquant: Accurate and efficient post-training quantization for large language models. arXiv preprint arXiv:2211.10438, 2022. · arXiv
- Nan Yang, Tao Ge, Liang Wang, Binxing Jiao, Daxin Jiang, Linjun Yang, Rangan Majumder, and Furu Wei. Inference with reference: Lossless acceleration of large language models. arXiv preprint arXiv:2304.04487, 2023. · arXiv
- Zhewei Yao, Reza Yazdani Aminabadi, Minjia Zhang, Xiaoxia Wu, Conglong Li, and Yuxiong He. Zeroquant: Efficient and affordable post-training quantization for large-scale transformers. Advances in Neural Information Processing Systems, 35:27168--27183, 2022.
- Gyeong-In Yu, Joo Seong Jeong, Geon-Woo Kim, Soojeong Kim, and Byung-Gon Chun. Orca: A distributed serving system for Transformer-Based generative models. In 16th USENIX Symposium on Operating Systems Design and Implementation (OSDI 22), pages 521--538, Carlsbad, CA, July 2022. USENIX Association.
- Haoyu Zhang, Jianjun Xu, and Ji Wang. Pretraining-based natural language generation for text summarization. arXiv preprint arXiv:1902.09243, 2019. · arXiv
- Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068, 2022. · arXiv
- Lianmin Zheng Chengfan Jia Minmin Sun Zhao Wu Cody Hao Yu Ameer Haj-Ali Yida Wang Jun Yang Danyang Zhuo Koushik Sen et al. Ansor: Generating high-performance tensor programs for deep learning. In 14th {USENIX} Symposium on Operating Systems Design and Implementation ({OSDI} 20) pages 863--879 2020.
Cited by
Sorted by citations
Items 101–121 of 121
Huang, Z., Jia, W., Li, Y., Lin, X., Fan, S., Tang, C., Deng, S., & Liu, Y. (2026) · Proceedings - International Symposium on Computer Architecture/Proceedings · doi:10.1109/isca66397.2026.00054
He, L., Zhou, F., Peng, C., Dong, S., Zhang, Z., Yang, H., Liu, Y., Zhang, G., & Jia, H. (2026) · Proceedings - International Symposium on Computer Architecture/Proceedings · doi:10.1109/isca66397.2026.00058
Self-citationA Survey on Optimization Techniques for On-Device Large Language Model Inference
Jo, M., & Park, S. (2026) · International Conference on Ubiquitous and Future Networks · doi:10.1109/icufn69619.2026.11628507
OrionInfer: Low-Overhead Parallelism Switching and Live Migration for Efficient LLM Serving
Feng, J., Yang, G., Huang, Y., Liang, S., Huang, C., Yan, M., & Wu, J. (2026) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3770855.3817626
Adeseye, A., Isoaho, J., & Tahir, M. (2026) · ACM Transactions on Privacy and Security · doi:10.1145/3839358
Efficient VLM Inference System With LoRA Adapters at the Edge
Wang, W., Mi, L., Chen, J., Yu, T., Cao, T., Li, Y., Li, M., Dai, H., Chen, G., & Liu, Y. (2026) · IEEE Transactions on Mobile Computing · doi:10.1109/tmc.2026.3723365
Asynchronous Speculative Sentence Decoding with Step-Level Alignment Classification
Hu, X., Li, Y., Ye, K., & Gao, X. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-36493-7_9
ParaCascade: A Parallel Cascading Framework Supporting Early Routing
Wei, H., Yin, L., Chen, C., Sun, J., Zhang, M., & Peng, Y. (2026) · IEEE International Conference on Joint Cloud Computing (JCC) · doi:10.1109/jcc72984.2026.00031
A Comparative Study of Fine-Tuned Small Models for Inference
Phanichraksaphong, V., Sento, A., & Kitjaidure, Y. (2026) · IEEE International Conference on Consumer Electronics - Taiwan (ICCE-Taiwan) · doi:10.1109/icce-taiwan71481.2026.11652842
Ygg: Tree-Based Collaborative Speculative Decoding with Token-Only Transmission
Li, S., Liang, Y., Li, R., Li, J., & Wu, J. (2026) · International Workshop on Quality of Service · doi:10.1109/iwqos70441.2026.11661225
WaSMa: Workflow-Aware Scheduling for Multi-Agent LLM Systems
Rao, U., Sadiq, A. Z., Shen, H., Yang, Y., Hodges, E.-A. R., & Muntean, M. (2026) · International Conference on Computer Communications and Networks (ICCCN) · doi:10.1109/icccn69946.2026.11662938
REKV: Cross-Precision Model Switching with Persistent KV Cache Reuse for LLM Serving
Wu, J., Xu, M., Wang, Y., Ye, K., Dai, H., & Xu, C. (2026) · IEEE International Conference on Web Services (ICWS) · doi:10.1109/icws72778.2026.00120
Cross-Layer Performance Analysis of Single-GPU Large Language Model Inference
Zhao, Z., Li, X., Meng, Z., Fu, Z., Wang, Q., Zhang, Y., & Li, Y. (2026) · Proceedings of the International Conference on Parallel Processing · doi:10.1145/3832810.3832894
wang, H., Jiao, L., Zhu, K., & Zhang, L. (2026) · Proceedings of the International Conference on Parallel Processing · doi:10.1145/3832810.3832825
MeshRT: Compile-Time Governed Wafer-Scale Runtime for Low-Latency High-Throughput Inference
He, C., Xu, L., Lu, Z., Huang, Y., Xiao, H., Deng, C., Ma, L., Miao, Z., Yang, F., & Mai, L. (2026) · ACM SIGOPS Symposium on Operating Systems Principles (SOSP) · doi:10.1145/3830418.3843856
Huang, C., Chen, Z., Guo, Z., & Zhang, W. (2026) · Research Square · doi:10.21203/rs.3.rs-11354781/v1
Self-citationZhao, K., Liu, Z., Jiang, C., Chen, P., Wang, X., Tian, K., Wang, N., Wang, Y., Shen, Y., Yan, J., et al. (2026) · IEEE Access · doi:10.1109/access.2026.3739598
ALPS: An Application-Aware, Length-Predictive Scheduler for Efficient LLM Inference at Edge
Chen, C., Jiao, L., & Mortier, R. (2026) · IEEE Transactions on Parallel and Distributed Systems · doi:10.1109/tpds.2026.3741399
Qiu, Y., Wang, S., Qin, Y., Geng, M., & Dong, D. (2026) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/3832783.3837454
Sparsity-Aware Scheduling for Efficient MoE-Based LLM Serving on Edge Servers
Zhuang, Y., Gong, C., Zheng, Z., Lei, J., Huang, X., Wu, J., Wu, F., & Chen, G. (2026) · ACM Annual International Conference on Mobile Computing and Networking (MobiCom) · doi:10.1145/3795866.3844460
Adaptive hybrid speculative decoding for accelerating large language model inference
Yong, Y., Yang, T. T., Gao, S. S., & Yin, Y. H. (2026) · Neurocomputing · doi:10.1016/j.neucom.2026.133760
Metadata
- Authors
- Miao, X., Oliaro, G., Zhang, Z., Cheng, X., Wang, Z., Zhang, Z., Wong, R. Y. Y., Zhu, A., Yang, L., Shi, X., Shi, C., Chen, Z., Arfeen, D., Abhyankar, R., & Jia, Z.
- Affiliations
- Shanghai Jiao Tong University (CN)
- ORCID
- Venue
- Proceedings of the 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 3
- Publisher
- ACM
- Published
- 2024-04-27
- Pages
- 932-949
- OpenAlex
- W4395112660
- Cited by
- 114
- References
- 24
- Citations by year
- 2026: 54 · 2025: 50 · 2024: 9
- FWCI
- 33.35
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS) (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Funders
- NSF (CNS-2147909, CNS-2211882, CNS-2239351)
- Amazon Research Award
- Cisco Research Award
- Google Faculty Research Award
- Meta Research Award
- Oracle Research Award
- Qualcomm Innovation Fellowship
- Samsung GRO Research Award
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Hardware and Architecture › Parallel Computing and Optimization Techniques
- Keywords
- large language model serving, parallel decoding, inference latency reduction, distributed inference, inference offloading
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
