The DOI matches its Crossref record; 23 of the 24 references it cites were found in a registry or an index.
SpecInfer: Accelerating Large Language Model Serving with Tree-based Speculative Inference and Verification
Open accessAbstract
This paper introduces SpecInfer, a system that accelerates generative large language model (LLM) serving with tree-based speculative inference and verification. The key idea behind SpecInfer is leveraging small speculative models to predict the LLM's outputs; the predictions are organized as a token tree, whose nodes each represent a candidate token sequence. The correctness of all candidate token sequences represented by a token tree is verified against the LLM in parallel using a novel tree-based parallel decoding mechanism. SpecInfer uses an LLM as a token tree verifier instead of an incremental decoder, which significantly reduces the end-to-end latency and computational requirement for serving generative LLMs while provably preserving model quality. Our evaluation shows that SpecInfer outperforms existing LLM serving systems by 1.5-2.8× for distributed LLM inference and by 2.6-3.5× for offloading-based LLM inference, while preserving the same generative performance. SpecInfer is publicly available at https://github.com/flexflow/FlexFlow/
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Existing tree-based works (Luo et al. 2025; Miao et al. 2024; Chen et al. 2024) employ static verification trees that lack adaptability by static structures, which cannot dynamically adjust to the changing context of the generated text, leading to suboptimal draft acceptance rates and limited speedup.”
From DIAA: A Decoding-Efficient Inference Acceleration Approach for On-Device Large Language Models (2026) · cites this paper as Miao et al. 2024
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“However, static token trees, most existing work adopted (Miao et al. 2024; Luo et al. 2025; Chen et al. 2024), are constructed once using fixed Top-K sampling and remain unchanged.”
From DIAA: A Decoding-Efficient Inference Acceleration Approach for On-Device Large Language Models (2026) · cites this paper as Miao et al. 2024
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
Computer Architecture: A Quantitative Approach
Hennessy, J. L., & Patterson, D. A. (1989) · UCLA - Biblioteca de Ciencias y Tecnologia Felix Morales Bueno · Cited by 9,434
LLaMA: Open and Efficient Foundation Language Models
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. (2023) · arXiv · doi:10.48550/arxiv.2302.13971 · Cited by 4,602
Computer architecture: a quantitative approach
Shute, M. (1993) · Microelectronics Journal · doi:10.1016/0026-2692(93)90111-q · Cited by 3,349
A Short Introduction to Boosting
Freund, Y., & Schapire, R. E. (1999) · Cited by 2,991
Ensemble deep learning: A review
Ganaie, M. A., Hu, M., Malik, A. K., Tanveer, M., & Suganthan, P. N. (2022) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2022.105151 · Cited by 2,446
Semantic Parsing on Freebase from Question-Answer Pairs
Berant, J., Chou, A., Frostig, R., & Liang, P. (2013) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d13-1160 · Cited by 1,593
TVM: an automated end-to-end optimizing compiler for deep learning
(2018) · arXiv (Cornell University) · doi:10.5555/3291168.3291211 · Cited by 1,170
Research works printed but not linked: 29
- Charlie Chen, Sebastian Borgeaud, Geoffrey Irving, Jean-Baptiste Lespiau, Laurent Sifre, and John Jumper. Accelerating large language model decoding with speculative sampling. arXiv preprint arXiv:2302.01318, 2023. · arXiv
- Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311, 2022. · arXiv
- Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. Gpt3. int8 (): 8-bit matrix multiplication for transformers at scale. Advances in Neural Information Processing Systems, 35:30318--30332, 2022.
- Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. Gptq: Accurate quantization for generative pre-trained transformers. In International Conference on Learning Representations, 2023.
- Freddy Gabbay and Avi Mendelson. Speculative execution based on value prediction. Citeseer, 1996.
- Joao Gante. Assisted generation: a new direction toward low-latency text generation, 2023.
- Sehoon Kim, Karttikeya Mangalam, Suhong Moon, John Canny, Jitendra Malik, Michael W. Mahoney, Amir Gholami, and Kurt Keutzer. Big little transformer decoder, 2023.
- Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Yu, Joseph E Gonzalez, Hao Zhang, and Ion Stoica. vllm: Easy, fast, and cheap llm serving with pagedattention. See https://vllm.ai/ (accessed 9 August 2023), 2023.
- Yaniv Leviathan, Matan Kalman, and Yossi Matias. Fast inference from transformers via speculative decoding. arXiv preprint arXiv:2211.17192, 2022. · arXiv
- Jiachang Liu, Dinghan Shen, Yizhe Zhang, Bill Dolan, Lawrence Carin, and Weizhu Chen. What makes good in-context examples for gpt-3? arXiv preprint arXiv:2101.06804, 2021. · arXiv
- Xupeng Miao, Gabriele Oliaro, Zhihao Zhang, Xinhao Cheng, Hongyi Jin, Tianqi Chen, and Zhihao Jia. Towards efficient generative large language model serving: A survey from algorithms to systems. arXiv preprint arXiv:2312.15234, 2023. · arXiv
- Xupeng Miao, Gabriele Oliaro, Zhihao Zhang, Xinhao Cheng, Zeyu Wang, Rae Ying Yee Wong, Zhuoming Chen, Daiyaan Arfeen, Reyna Abhyankar, and Zhihao Jia. Specinfer: Accelerating generative llm serving with speculative inference and token tree verification. arXiv preprint arXiv:2305.09781, 2023. · arXiv
- Xupeng Miao, Chunan Shi, Jiangfei Duan, Xiaoli Xi, Dahua Lin, Bin Cui, and Zhihao Jia. Spotserve: Serving generative large language models on preemptible instances. arXiv preprint arXiv:2311.15566, 2023. · arXiv
- OpenAI. Gpt-4 technical report, 2023.
- Gunho Park, Baeseong Park, Se Jung Kwon, Byeongwook Kim, Youngjoo Lee, and Dongsoo Lee. nuqmm: Quantized matmul for efficient inference of large-scale generative language models. arXiv preprint arXiv:2206.09557, 2022. · arXiv
- Baolin Peng, Chunyuan Li, Pengcheng He, Michel Galley, and Jianfeng Gao. Instruction tuning with gpt-4. arXiv preprint arXiv:2304.03277, 2023. · arXiv
- Jack W Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, et al. Scaling language models: Methods, analysis & insights from training gopher. arXiv preprint arXiv:2112.11446, 2021. · arXiv
- Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilić, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, et al. Bloom: A 176b-parameter open-access multilingual language model. arXiv preprint arXiv:2211.05100, 2022. · arXiv
- Ying Sheng, Lianmin Zheng, Binhang Yuan, Zhuohan Li, Max Ryabinin, Daniel Y. Fu, Zhiqiang Xie, Beidi Chen, Clark Barrett, Joseph E. Gonzalez, Percy Liang, Christopher Ré, Ion Stoica, and Ce Zhang. High-throughput generative inference of large language models with a single gpu, 2023.
- James E Smith. A study of branch prediction strategies. In 25 years of the international symposia on Computer architecture (selected papers), pages 202--215, 1998.
- Shaden Smith, Mostofa Patwary, Brandon Norick, Patrick LeGresley, Samyam Rajbhandari, Jared Casper, Zhun Liu, Shrimai Prabhumoye, George Zerveas, Vijay Korthikanti, et al. Using deepspeed and mega-tron to train megatron-turing nlg 530b, a large-scale generative language model. arXiv preprint arXiv:2201.11990, 2022. · arXiv
- Colin Unger, Zhihao Jia, Wei Wu, Sina Lin, Mandeep Baines, Carlos Efrain Quintero Narvaez, Vinay Ramakrishnaiah, Nirmal Prajapati, Pat McCormick, Jamaludin Mohd-Yusof, Xi Luo, Dheevatsa Mudigere, Jongsoo Park, Misha Smelyanskiy, and Alex Aiken. Unity: Accelerating DNN training through joint optimization of algebraic transformations and parallelization. In 16th USENIX Symposium on Operating Systems Design and Implementation (OSDI 22), pages 267--284, Carlsbad, CA, July 2022. USENIX Association.
- Guangxuan Xiao, Ji Lin, Mickael Seznec, Julien Demouth, and Song Han. Smoothquant: Accurate and efficient post-training quantization for large language models. arXiv preprint arXiv:2211.10438, 2022. · arXiv
- Nan Yang, Tao Ge, Liang Wang, Binxing Jiao, Daxin Jiang, Linjun Yang, Rangan Majumder, and Furu Wei. Inference with reference: Lossless acceleration of large language models. arXiv preprint arXiv:2304.04487, 2023. · arXiv
- Zhewei Yao, Reza Yazdani Aminabadi, Minjia Zhang, Xiaoxia Wu, Conglong Li, and Yuxiong He. Zeroquant: Efficient and affordable post-training quantization for large-scale transformers. Advances in Neural Information Processing Systems, 35:27168--27183, 2022.
- Gyeong-In Yu, Joo Seong Jeong, Geon-Woo Kim, Soojeong Kim, and Byung-Gon Chun. Orca: A distributed serving system for Transformer-Based generative models. In 16th USENIX Symposium on Operating Systems Design and Implementation (OSDI 22), pages 521--538, Carlsbad, CA, July 2022. USENIX Association.
- Haoyu Zhang, Jianjun Xu, and Ji Wang. Pretraining-based natural language generation for text summarization. arXiv preprint arXiv:1902.09243, 2019. · arXiv
- Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068, 2022. · arXiv
- Lianmin Zheng Chengfan Jia Minmin Sun Zhao Wu Cody Hao Yu Ameer Haj-Ali Yida Wang Jun Yang Danyang Zhuo Koushik Sen et al. Ansor: Generating high-performance tensor programs for deep learning. In 14th {USENIX} Symposium on Operating Systems Design and Implementation ({OSDI} 20) pages 863--879 2020.
Cited by
Sorted by citations
Towards Efficient Generative Large Language Model Serving: A Survey from Algorithms to Systems
Miao, X., Oliaro, G., Zhang, Z., Cheng, X., Jin, H., Chen, T., & Jia, Z. (2025) · ACM Computing Surveys · doi:10.1145/3754448 · Cited by 176
Self-citationBurstGPT: A Real-World Workload Dataset to Optimize LLM Serving Systems
Wang, Y., Chen, Y., Li, Z., Kang, X., Fang, Y., Zhou, Y., Zheng, Y., Tang, Z., He, X. J., Guo, R., et al. (2025) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3711896.3737413 · Cited by 172
Wang, R., Gao, Z., Zhang, L., Yue, S., & Gao, Z. (2025) · Computer Science Review · doi:10.1016/j.cosrev.2025.100755 · Cited by 61
Accelerating Retrieval-Augmented Generation
Quinn, D., Nouri, M., Patel, N., Salihu, J., Salemi, A., Lee, S., Zamani, H., & Alian, M. (2025) · Proceedings of the 30th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 1 · doi:10.1145/3669940.3707264 · Cited by 48
LLM Inference Serving: Survey of Recent Advances and Opportunities
Li, B., Jiang, Y., Gadepally, V. N., & Tiwari, D. (2024) · IEEE High Performance Extreme Computing Conference · doi:10.1109/hpec62836.2024.10938426 · Cited by 108
Speculative Decoding: Exploiting Speculative Execution for Accelerating Seq2seq Generation
Xia, H., Ge, T., Wang, P., Chen, S.-Q., Wei, F., & Sui, Z. (2023) · Findings of the Association for Computational Linguistics: EMNLP 2023 · doi:10.18653/v1/2023.findings-emnlp.257 · Cited by 40
Helix: Serving Large Language Models over Heterogeneous GPUs and Network via Max-Flow
Mei, Y., Zhuang, Y., Miao, X., Yang, J., Jia, Z., & Vinayak, R. (2025) · Proceedings of the 30th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 1 · doi:10.1145/3669940.3707215 · Cited by 35
Self-citationA Survey: Collaborative Hardware and Software Design in the Era of Large Language Models
Guo, C., Cheng, F., Du, Z., Kiessling, J., Ku, J., Li, S., Li, Z., Ma, M., Molom-Ochir, T., Morris, B., et al. (2025) · IEEE Circuits and Systems Magazine · doi:10.1109/mcas.2024.3476008 · Cited by 32
InSS: An Intelligent Scheduling Orchestrator for Multi-GPU Inference With Spatio-Temporal Sharing
Han, Z., Zhou, R., Xu, C., Zeng, Y., & Zhang, R. (2024) · IEEE Transactions on Parallel and Distributed Systems · doi:10.1109/tpds.2024.3430063 · Cited by 29
Efficient LLMs Training and Inference: An Introduction
Li, R., Fu, D., Shi, C., Huang, Z., & Lu, G. (2024) · IEEE Access · doi:10.1109/access.2024.3501358 · Cited by 36
Self-citationEfficient Training and Inference: Techniques for Large Language Models Using LlamaPreprint
Cunningham, S. R., Archambault, D., & Kung, A. (2024) · doi:10.36227/techrxiv.171651876.65094225/v1 · Cited by 22
Efficient Inference for Edge Large Language Models: A Survey
Cai, G., Tian, R., Yang, L., Jia, Y., Li, L., & Wang, J. (2025) · Tsinghua Science & Technology · doi:10.26599/tst.2025.9010166 · Cited by 19
Self-citationthrottLL’eM: Predictive GPU Throttling for Energy Efficient LLM Inference Serving
Kakolyris, A. K., Masouros, D., Vavaroutsos, P., Xydis, S., & Soudris, D. (2025) · IEEE International Symposium on High-Performance Computer Architecture (HPCA) · doi:10.1109/hpca61900.2025.00103 · Cited by 44
WindServe: Efficient Phase-Disaggregated LLM Serving with Stream-based Dynamic Scheduling
Feng, J., Huang, Y., Zhang, R., Liang, S., Yan, M., & Wu, J. (2025) · Proceedings of the 52nd Annual International Symposium on Computer Architecture · doi:10.1145/3695053.3730999 · Cited by 16
Towards End-to-End Optimization of LLM-based Applications with Ayo
Tan, X., Jiang, Y., Yang, Y., & Xu, H. (2025) · Proceedings of the 30th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2 · doi:10.1145/3676641.3716278 · Cited by 13
Sun, W., Gao, M., Li, Z., Zhang, A., Chou, I. Y., Zhu, J., Wei, S., & Liu, L. (2025) · IEEE International Symposium on High-Performance Computer Architecture (HPCA) · doi:10.1109/hpca61900.2025.00128 · Cited by 16
SPIN: Accelerating Large Language Model Inference with Heterogeneous Speculative Models
Chen, F., Li, P., Luan, T. H., Su, Z., & Deng, J. (2025) · IEEE INFOCOM 2025 - IEEE Conference on Computer Communications · doi:10.1109/infocom55648.2025.11044522 · Cited by 11
Zhao, Y., Xie, Z., Liang, C., Zhuang, C., & Gu, J. (2024) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3637528.3671614 · Cited by 26
ResDecode: Accelerating Large Language Models Inference via Residual Decoding Heads
Zeng, Z., Yu, J., Pang, Q., Wang, Z., Zhuang, H., Fan, Y., Shao, H., & Zou, X. (2025) · Big Data Mining and Analytics · doi:10.26599/bdma.2024.9020074 · Cited by 10
Self-citationSCOOT: SLO-Oriented Performance Tuning for LLM Inference Engines
Cheng, K., Wang, Z., Hu, W., Yang, T., Li, J., & Zhang, S. (2025) · International World Wide Web Conference (WWW) · doi:10.1145/3696410.3714930 · Cited by 12
Self-citationLess is More: Optimizing Function Calling for LLM Execution on Edge Devices
Paramanayakam, V., Καρατζάς, Α., Anagnostopoulos, I., & Stamoulis, D. S. (2025) · Design, Automation, and Test in Europe · doi:10.23919/date64628.2025.10992798 · Cited by 11
MinCache: A hybrid cache system for efficient chatbots with hierarchical embedding matching and LLM
Haqiq, K., Jahan, M. V., Farimani, S. A., & Masoom, S. M. F. (2025) · Future Generation Computer Systems · doi:10.1016/j.future.2025.107822 · Cited by 11
Deploying Foundation Model Powered Agent Services: A Survey
Xu, W., Chen, J., Zheng, P., Yi, X., Tian, T., Zhu, W., Wan, Q., Wang, H., Fan, Y., Su, Q., et al. (2025) · IEEE Communications Surveys & Tutorials · doi:10.1109/comst.2025.3580745 · Cited by 20
Function Calling in Large Language Models: Industrial Practices, Challenges, and Future Directions
Wang, M., Zhang, Y., Yu, B., Hao, B., Peng, C., Chen, Y.-C., Zhou, W., Gu, J., Zhuang, C., Guo, R., et al. (2026) · ACM Computing Surveys · doi:10.1145/3788284 · Cited by 12
On the Cost of Model-Serving Frameworks: An Experimental Evaluation
De Rosa, P., Bromberg, Y.-D., Felber, P., Mvondo, D., & Schiavoni, V. (2024) · 2024 IEEE International Conference on Cloud Engineering (IC2E) · doi:10.1109/ic2e61754.2024.00032 · Cited by 6
“For example, [62] introduced SpecInfer, a system that accelerates generative LLM serving with tree-based speculative inference and verification.”
LAD: Efficient Accelerator for Generative Inference of LLM with Locality Aware Decoding
Wang, H., Li, Y., Xu, H., Wang, Y., Liu, L., Yang, J., & Han, Y. (2025) · IEEE International Symposium on High-Performance Computer Architecture (HPCA) · doi:10.1109/hpca61900.2025.00111 · Cited by 6
He, S., Zhu, Z., He, Y., & Jia, T. (2025) · doi:10.1109/iccad66269.2025.11240889 · Cited by 6
Xi, Y., Wang, H., Chen, B., Lin, J., Zhu, M., Liu, W., Tang, R., Wei, Z., Zhang, W., & Yu, Y. H. (2025) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3726302.3729961 · Cited by 21
Yin, W., Yi, R., Xu, D., Huang, G., Xu, M., & Liu, X. (2025) · ACM Annual International Conference on Mobile Computing and Networking (MobiCom) · doi:10.1145/3680207.3765259 · Cited by 18
FlexSpec: Frozen Drafts Meet Evolving Targets in Edge-Cloud Collaborative LLM Speculative Decoding
Li, Y., Kong, R., Lyu, Z., Li, Q., Chen, X., Cai, H., Yan, L., Wang, S., Zhao, J., Zhu, G., et al. (2026) · IEEE Transactions on Mobile Computing · doi:10.1109/tmc.2026.3683574 · Cited by 8
DySpec: Faster speculative decoding with dynamic token tree structure
Xiong, Y., Zhang, R., Li, Y., & Zou, L. (2025) · World Wide Web · doi:10.1007/s11280-025-01344-0 · Cited by 4
Proceedings of the Workshop on Edge and Mobile Foundation Models
(2024) · doi:10.1145/3662006 · Cited by 4
Zhou, T., Yang, Y., Yang, C., Xiao, L., Liu, X., & Hu, M. (2025) · IEEE International Conference on Big Data Security on Cloud, IEEE International Conference on High Performance and Smart Computing, and IEEE International Conference on Intelligent Data and Security (BigDataSecurity/HPSC/IDS) · doi:10.1109/hpsc66065.2025.00016 · Cited by 3
SpecMamba: Accelerating Mamba Inference on FPGA with Speculative Decoding
Zhong, L., Xu, S., Wen, H., Xie, T., Guo, Q., Wang, Y., & Li, M. (2025) · doi:10.1109/iccad66269.2025.11240945 · Cited by 3
AdaSpec: Adaptive Speculative Decoding for Fast, SLO-Aware Large Language Model Serving
Huang, K., Wu, H., Shi, Z., Zou, H., Yu, M., & Shi, Q. (2025) · ACM Symposium on Cloud Computing (SoCC) · doi:10.1145/3772052.3772239 · Cited by 9
Empower Vision Applications with LoRA LMM
Mi, L., Wang, W., Tu, W., He, Q., Kong, R., Fang, X., Dong, Y., Zhang, Y., Li, Y., Li, M., et al. (2025) · European Conference on Computer Systems · doi:10.1145/3689031.3717472 · Cited by 9
When DeepSeek-R1 meets financial applications: benchmarking, opportunities, and limitations
Liu, S., Chen, L., Yan, J., Jiang, Y., Wang, X., Li, X., & Yang, Q. (2025) · Frontiers of Information Technology & Electronic Engineering · doi:10.1631/fitee.2500227 · Cited by 2
Big Loop and Atomization: A Holistic Review on the Expansion Capabilities of Large Language Models
Hu, Z., Huang, Y., Feng, J., & Deng, C. (2025) · Applied Sciences · doi:10.3390/app15179466 · Cited by 3
Chameleon: Adaptive Caching and Scheduling for Many-Adapter LLM Inference Environments
Iliakopoulou, N., Stojkovic, J., Alverti, C., Xu, T., Franke, H., & Torrellas, J. (2025) · IEEE/ACM International Symposium on Microarchitecture (MICRO) · doi:10.1145/3725843.3756083 · Cited by 15
Towards Efficient LLM Inference via Collective and Adaptive Speculative Decoding
Wang, S., Yang, H., Wang, X., Liu, T., Wang, P., Xu, Y., Liang, X., Ma, K., Feng, T., You, X., et al. (2025) · International Conference for High Performance Computing, Networking, Storage and Analysis (SC) · doi:10.1145/3712285.3759834 · Cited by 2
Liakopoulos, D., Sinha, P., Hu, T., Lee, M., & Yadwadkar, N. J. (2025) · International Conference for High Performance Computing, Networking, Storage and Analysis (SC) · doi:10.1145/3712285.3759867 · Cited by 2
Huang, H., Zhan, W., Duan, H., Peng, K., Min, G., Zhao, Z., Zhao, Z., & Ye, Y. (2026) · IEEE Transactions on Mobile Computing · doi:10.1109/tmc.2026.3667591 · Cited by 2
Adaptive Draft Sequence Length: Enhancing Speculative Decoding Throughput on PIM-Enabled Systems
Wang, R., Wang, Q., Liu, H., Zheng, L., Liao, X., Jin, H., & Xue, J. (2026) · IEEE International Symposium on High-Performance Computer Architecture (HPCA) · doi:10.1109/hpca68181.2026.11408598 · Cited by 2
History Doesn't Repeat Itself but Rollouts Rhyme: Accelerating Reinforcement Learning with RhymeRL
He, J., Li, T., Feng, E., Du, D., Liu, Q., Liu, T., Xia, Y., & Chen, H. (2026) · International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS) · doi:10.1145/3779212.3790172 · Cited by 7
Whisper in Medusa’s Ear: Multi-head Efficient Decoding for Transformer-based ASR
Segal, Y., Shamsian, A., Navon, A., Hetz, G., & Keshet, J. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10888140 · Cited by 2
Preble: Efficient Distributed Prompt Scheduling for LLM Serving
Srivatsa, V., He, Z., Abhyankar, R., Li, D., & Zhang, Y. (2024) · arXiv · doi:10.48550/arxiv.2407.00023 · Cited by 1
Self-citationDynamicAttention: Dynamic KV Cache for Disaggregate LLM Inference
Ding, Z., & Yang, T. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10890367 · Cited by 1
Gao, X., Xie, W., Xiang, Y., & Ji, F. Q. (2025) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v39i22.34566 · Cited by 17
Speculative Decoding for Verilog: Speed and Quality, All in One
Xu, C., Liu, Y., Zhou, Y., Huang, S., Xu, N., & Xu, Q. (2025) · ACM/IEEE Design Automation Conference (DAC) · doi:10.1109/dac63849.2025.11133030 · Cited by 1
GPMassSimulator: A Graphormer-Based Method for Glycopeptide MS/MS Spectra Prediction
Ren, Y., Bu, D., Duan, B., Zhang, C., Cong, B., & Sun, S. (2025) · Analytical Chemistry · doi:10.1021/acs.analchem.5c02375 · Cited by 1
Ghidorah: Fast LLM Inference on Edge with Speculative Decoding and Hetero-Core Parallelism
Wei, J., Huang, Y., Zhou, Y., Jiang, J., & Du, J. (2025) · IEEE International Conference on Computer Design: VLSI in Computers and Processors (ICCD) · doi:10.1109/iccd65941.2025.00043 · Cited by 1
FastCoder: Accelerating Repository-level Code Generation via Efficient Retrieval and Verification
Zhao, Q., Zhang, L., Liu, F., Lian, X., Meng, Q., Jiao, Z., Zhou, Z., Li, J., & Shi, L. (2025) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase63991.2025.00190 · Cited by 1
Chen, K., Zhou, Z., Chen, Y., & Han, J. H. (2026) · IEEE Transactions on Very Large Scale Integration (VLSI) Systems · doi:10.1109/tvlsi.2026.3659893 · Cited by 1
Unlocking One-for-All Generative AI at Edge
Miriyala, S. S., Rajendiran, V. N., & Allur, S. K. (2026) · Springer eBooks · doi:10.1007/978-3-032-06418-9_2 · Cited by 1
Taming the Long-Tail: Efficient Reasoning RL Training with Adaptive Drafter
Hu, Q., Yang, S. W., Guo, J., Yao, X., Lin, Y., Gu, Y., Cai, H., Gan, C., Klimovic, A., & Han, S. (2026) · International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS) · doi:10.1145/3779212.3790231 · Cited by 17
Device-Server Collaborative Speculative Decoding for Real-Time LLM Streaming
Biswas, B. R., Guan, Y., Yin, M., Han, T., & Hou, X. (2025) · Globecom · doi:10.1109/globecom59602.2025.11432593 · Cited by 1
SMART: When is It Actually Worth Expanding a Speculative Tree?
Wang, L., & Zhou, P. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-37627-5_13 · Cited by 3
Efficient LLM Inference Over Heterogeneous Edge Networks With Speculative Decoding
Zhu, B., Chen, Z., Zhao, L., Shin, H., & Nallanathan, A. (2026) · IEEE Transactions on Communications · doi:10.1109/tcomm.2026.3712530 · Cited by 3
Self-citationArlo: Serving Transformer-based Language Models with Dynamic Input Lengths
Tan, X. H., Li, J., Yang, Y., Li, J., & Xu, H. L. (2024) · Proceedings of the International Conference on Parallel Processing · doi:10.1145/3673038.3673124 · Cited by 1
Enhancing Stateful Processing in Programmable Data Planes: Model and Improved Architecture
Feng, Y., Zhou, H., Liu, S., Chen, Z., Song, H., & Liu, B. (2024) · IEEE Transactions on Networking · doi:10.1109/tnet.2024.3510638 · Cited by 1
Self-citationAMUSD: Asynchronous Multi-Device Speculative Decoding for LLM Acceleration
McDanel, B. (2025) · IEEE International Symposium on Circuits and Systems proceedings · doi:10.1109/iscas56072.2025.11043575 · Cited by 1
Gao, X., Xie, W., Lixin, Fang, X., Hang, C., Li, C., Lin, Y., & Xu, X. (2026) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v40i25.39266
DIAA: A Decoding-Efficient Inference Acceleration Approach for On-Device Large Language Models
Tian, H., Lu, S., Tian, F., Cui, G., Li, Z., Zhang, X., Sheng, Q. Z., & Dou, W. (2026) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v40i31.39789
Confidence-Modulated Speculative Decoding for Large Language Models
Sen, J., Dasgupta, S., & Waghela, H. (2025) · 2025 IEEE 6th India Council International Subsections Conference (INDISCON) · doi:10.1109/indiscon66021.2025.11254640
WWW.Serve: Interconnecting Global LLM Services through Decentralization
Wang, H., Xia, Z., Chen, Z., & Chen, B. (2026) · arXiv · doi:10.48550/arxiv.2603.20661
Self-citationWen, Z., Xu, L., & Wang, M. (2025) · Integrated Circuits and Systems · doi:10.23919/ics.2025.3575371
Improving Speculative Decoding with Dynamic Adjustment and Probability Smoothing
Han, Y. (2025) · International Conference on Advanced Algorithms and Control Engineering (ICAACE) · doi:10.1109/icaace65325.2025.11019425
Masker: Speculative Decoding for Long-Text QA with CopyHead
Zhou, Y., Pan, H., Liu, R., & Zhang, L. (2025) · Communications in computer and information science · doi:10.1007/978-981-96-6954-7_3
TOPO-X: Co-optimize Flow Scheduling, Topology, and ML Training Parallelism
Hu, Y., Han, T., Zhao, Y., Wu, F., & Li, X. (2025) · International Conference on Computer Communications and Networks (ICCCN) · doi:10.1109/icccn65249.2025.11133974
MTP: A Meaning-Typed Language Abstraction for AI-Integrated Programming
Dantanarayana, J. L., Kang, Y., Sivasothynathan, K., Clarke, C., Li, B., Kashmira, S., Flautner, K., Tang, L., & Mars, J. (2025) · Proceedings of the ACM on Programming Languages · doi:10.1145/3763092
SuperSpec: Enhanced Verification and Sampling for End-to-End LLM Speculative Decoding
Shen, C., Guo, R., Yang, C., Lin, Y., Zhao, Z., Hu, Y., Chen, S., Liu, X., & Li, K. (2025) · IEEE International Conference on High Performance Computing and Communications; IEEE International Conference on Smart City; IEEE International Conference on Data Science and Systems (HPCC/SmartCity/DSS) · doi:10.1109/hpcc67675.2025.00064
XY-Serve: End-to-End Versatile Production Serving for Dynamic LLM Workloads
Song, M., Tang, X., Hou, F., Li, J., Wei, W., Ma, Y., Xiao, R., Si, H., Jiang, D., Yin, S., et al. (2025) · International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS) · doi:10.1145/3760250.3762228
Efficient Multilingual Translation of Native Indian Languages in GenAI based Product Offer Platform
Awasthi, S. A., Kumar, D., Jhunjhunwala, A., Padmanabhan, K. M., Moorthy, V., Sengodan, G., Srinivasan, S., George, P., Pandey, A., & Pushpalatha, M. (2025) · IEEE International Conference on Electronics, Computing and Communication Technologies (CONECCT) · doi:10.1109/conecct65861.2025.11306851
CoMPI: Coordinated Model Merging and Parallel Inference at Edge
Zeng, S., Zhang, H., & Yan, Z. (2025) · ACM Symposium on Cloud Computing (SoCC) · doi:10.1145/3772052.3772249
KVSpec: Efficient KV-Cache Reuse for Speculative Decoding
Xu, C. L., & Xue, J. (2025) · International Conference on Cloud Computing, Big Data Application and Software Engineering (CBASE) · doi:10.1109/cbase67452.2025.11335478
JanusQuant: Accurate and Efficient 2-bit KV Cache Quantization for Long-Context Inference
Sun, C., Xia, Y., Wang, H., Yang, D., Zhou, X., & Cheng, D. (2026) · ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming · doi:10.1145/3774934.3786428 · Cited by 2
SEC : Enabling MLLMs for Low-Latency IoT Video Analysis via Semantic-Aware Edge–Cloud Collaboration
Yang, M., Tian, Y., Cong, P., Zhang, L., Xi, G., Wang, S., & Wang, W. (2026) · IEEE Internet of Things Journal · doi:10.1109/jiot.2026.3669144
Wang, Y., Wang, H., Yang, J., Su, Y., Guo, R., Yue, Z., Gu, J., Wei, S., Hu, Y., & Yin, S. (2026) · Digest of technical papers/Digest of technical papers - IEEE International Solid-State Circuits Conference · doi:10.1109/isscc49663.2026.11408953
VAR-Turbo: Unlocking the Potential of Visual Autoregressive Models Through Dual Redundancy
Xiang, X., & Tu, F. (2026) · IEEE International Symposium on High-Performance Computer Architecture (HPCA) · doi:10.1109/hpca68181.2026.11408607 · Cited by 1
Towards Resource-Efficient Serverless LLM Inference with SLINFER
Xu, C., Zijun, L. I., Chen, Q., Zhao, H., Tang, X., & Guo, M. (2026) · IEEE International Symposium on High-Performance Computer Architecture (HPCA) · doi:10.1109/hpca68181.2026.11408548 · Cited by 4
SwiftSpec: Disaggregated Speculative Decoding and Fused Kernels for Low-Latency LLM Inference
Zhang, Z., Jiang, Z., Jiang, C., Yu, M., Zheng, S., Lin, H., Liu, X., & Hoffmann, H. (2026) · International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS) · doi:10.1145/3779212.3790246 · Cited by 3
Self-citationASDT: Adaptive Speculative Decoding Tree Based on Token Acceptance Feedback
Song, Y., Xia, H., & Duan, F. (2025) · International Symposium on Artificial Intelligence and Big Data (AIBDF) · doi:10.1109/aibdf67964.2025.11440682
Zheng, L., Hu, Y., Shen, C., Zhao, Z., Shi, B., Zhang, Y., Zhao, L., Li, W., & Li, K. (2026) · IEEE Transactions on Parallel and Distributed Systems · doi:10.1109/tpds.2026.3683867
Experience-Driven Dynamic Exits for LLMs with Reinforcement Learning
Zhu, Y., Pao, H., Hu, N., Guo, W., Zhan, S., Lai, B., Wang, Z., Zeng, Y., & Zheng, H. (2026) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp55912.2026.11464968
Self-citationTOKCOINFER: Token-Level Multi-Model Collaboration for Energy-Efficient LLM Inference
Zhang, Y., Zhang, D., Wang, B., & Zhang, S. (2026) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp55912.2026.11464816
A survey on edge multimodal large models: Compression, inference acceleration, and applications
Chen, S., Shu, Y., Wang, C., & Chen, J. (2026) · National Science Open · doi:10.1360/nso/20260016
Efficient LLM Serving for Agentic Workflows: A Data Systems Perspective
Wadlom, N., Shen, J., & Lu, Y. (2026) · Proceedings of the ACM on Management of Data · doi:10.1145/3802046
CoDec: Prefix-Shared Decoding Kernel for LLMs
Wang, Z., Ning, R., Fang, C., Zhang, Z. J., Lin, X., Ma, S., Zhou, M., Li, X., Wang, Z., Huan, C., et al. (2026) · Proceedings of the ACM on Management of Data · doi:10.1145/3802028 · Cited by 7
Self-citationRAG Under Load: Analytical and Empirical Modeling of Retrieval–Generation Performance at Scale
GUDIPATI, S. K., AKKISETTI, S. R., & Mohammed, A. R. (2026) · Lecture notes in networks and systems · doi:10.1007/978-3-032-26211-0_19
Boosting the Performance of Tree-Based Speculative Decoding of LLMs on FPGAs
Liu, T., Li, G., Mo, Z., Zhu, Z. G., Pei, M., & Cheng, J. (2026) · Design, Automation, and Test in Europe · doi:10.23919/date69613.2026.11539704
Reliable LLM-Based Edge-Cloud-Expert Cascades for Telecom Knowledge Systems
Hou, Q., Park, S., Zecchin, M., Cai, Y., Yu, G., Simeone, O., & Melodia, T. (2026) · IEEE Transactions on Communications · doi:10.1109/tcomm.2026.3700473
Yu, J., Taneja, A., Lin, J., & Zhang, M. (2026) · ISC High Performance (ISC) · doi:10.23919/isc.2026.11520495
Ahmed, A. M., Sadiq, A. T., Farhan, A. K., Ahmad, M., & Shakeel, M. R. (2026) · Archives of Computational Methods in Engineering · doi:10.1007/s11831-026-10656-x · Cited by 1
MemSpec: Memory-Aware Runtime for Adaptive Draft Scheduling in Speculative Decoding on Edge Devices
Kim, E., Jeon, Y. J., & Han, M. (2026) · ACM SIGPLAN/SIGBED Conference on Languages, Compilers, and Tools for Embedded Systems (LCTES) · doi:10.1145/3814943.3816174
Pacer: Blockwise Pre-Verification for Speculative Decoding With Adaptive Length
Zhang, X., Zhang, Y., Zhu, Z. G., Wang, H., Ma, D., Zhang, D., Chen, L., & Yu, K. (2026) · IEEE Transactions on Audio Speech and Language Processing · doi:10.1109/taslpro.2026.3705693
EdgeSpec: Distributed Speculative Decoding for Large Language Models at Edge
Chen, Y., Tian, M., Qiu, C., Wang, X. L., Yao, H., He, Y., & Liu, Y. (2026) · IEEE Conference on Computer Communications (INFOCOM) · doi:10.1109/infocom59046.2026.11571680
CoSine: Enhancing LLM Serving via Collaborative and Decoupled Speculative Inference
Gao, L., Liu, J. J., Zhang, X., Gao, G., & Liao, Y. (2026) · IEEE Conference on Computer Communications (INFOCOM) · doi:10.1109/infocom59046.2026.11571388
DWI: Efficient On-Device LLM Decoding Through Dynamic Width Selection on Nested Models
Chen, W. P., Deng, Y., Wang, X., Fu, Z., Li, X. F., Cao, T., Liu, Y., & Ren, J. (2026) · IEEE Transactions on Mobile Computing · doi:10.1109/tmc.2026.3711495
Collaborative Edge Inference for Large Language Models with Speculative Decoding
Zhu, B., Chen, Z., Zhao, L., Shin, H., & Nallanathan, A. (2026) · International Conference on Communications · doi:10.1109/icc59461.2026.11587792
Self-citationLARK: Layer-Aware Routing with K-Ary Dynamic Pruning Tree for Self-speculative Decoding
Xu, B., Zhang, Z., Huang, Z., & Zhang, L. (2026) · Lecture notes in computer science · doi:10.1007/978-981-92-2759-4_26
Self-citation
Metadata
- Authors
- Miao, X., Oliaro, G., Zhang, Z., Cheng, X., Wang, Z., Zhang, Z., Wong, R. Y. Y., Zhu, A., Yang, L., Shi, X., Shi, C., Chen, Z., Arfeen, D., Abhyankar, R., & Jia, Z.
- Affiliations
- Shanghai Jiao Tong University (CN)
- ORCID
- Venue
- Proceedings of the 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 3
- Publisher
- ACM
- Published
- 2024-04-27
- Pages
- 932-949
- OpenAlex
- W4395112660
- Cited by
- 114
- References
- 24
- Citations by year
- 2026: 54 · 2025: 50 · 2024: 9
- FWCI
- 33.35
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS) (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Funders
- NSF (CNS-2147909, CNS-2211882, CNS-2239351)
- Amazon Research Award
- Cisco Research Award
- Google Faculty Research Award
- Meta Research Award
- Oracle Research Award
- Qualcomm Innovation Fellowship
- Samsung GRO Research Award
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Hardware and Architecture › Parallel Computing and Optimization Techniques
- Keywords
- large language model serving, parallel decoding, inference latency reduction, distributed inference, inference offloading
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
