The DOI matches its Crossref record; the references it cites are not checked yet.
Turning Trash into Treasure: Accelerating Inference of Large Language Models with Token Recycling
Open accessAbstract
Xianzhen Luo, Yixuan Wang, Qingfu Zhu, Zhiming Zhang, Xuanyu Zhang, Qing Yang, Dongliang Xu. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2025.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Static token trees are constructed once using fixed Top-K sampling and remain unchanged.
“However, static token trees, most existing work adopted (Miao et al. 2024; Luo et al. 2025; Chen et al. 2024), are constructed once using fixed Top-K sampling and remain unchanged.”
From DIAA: A Decoding-Efficient Inference Acceleration Approach for On-Device Large Language Models (2026) · cites this paper as Luo et al. 2025
Section 3.2 says the tree structure is static: each node has the same number of children across decoding steps. But Appendix A.2 says that structure is selected through iterative validation, not constructed once by fixed Top-K sampling, and that a draft tree is built for the current content during inference. Thus the unchanged structure is supported, but the claim conflates it with the changing token-filled draft trees. (LLM, conf=0.88)
Experimental — accuracy not yet measuredExisting tree-based works employ static verification trees that lack adaptability by static structures, which cannot dynamically adjust to the changing context of the generated text, leading to suboptimal draft acceptance rates and limited speedup.
“Existing tree-based works (Luo et al. 2025; Miao et al. 2024; Chen et al. 2024) employ static verification trees that lack adaptability by static structures, which cannot dynamically adjust to the changing context of the generated text, leading to suboptimal draft acceptance rates and limited speedup.”
From DIAA: A Decoding-Efficient Inference Acceleration Approach for On-Device Large Language Models (2026) · cites this paper as Luo et al. 2025
Split into 2 atoms — 0 confirmed, 2 partially supported, 0 disputed, 0 misattributed, 0 unverifiable. Expand the claim to see per-atom verdicts.
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The publisher hasn’t deposited this paper’s reference list.
Research works printed but not linked: 37
- Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. 2021. Program synthesis with large language models. arXiv preprint arXiv:2108.07732. · arXiv
- Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In Ad- vances in Neural Information Processing Systems 33: Annual Conference on Neural Information Process- ing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual.
- Tianle Cai, Yuhong Li, Zhengyang Geng, Hongwu Peng, Jason D. Lee, Deming Chen, and Tri Dao. 2024. Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads. Preprint, arXiv:2401.10774. · arXiv
- Charlie Chen, Sebastian Borgeaud, Geoffrey Irving, Jean-Baptiste Lespiau, Laurent Sifre, and John Jumper. 2023. Accelerating Large Language Model Decoding with Speculative Sampling. Preprint, arXiv:2302.01318. · arXiv
- Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Ka- plan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sas- try, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cum- mings, Matthias Plappert, Fotios Chantzis, Eliza- beth Barnes, Ariel Herbert-Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba. 2021. Evaluating Large Language Models Trained on Code. Preprint, arXiv:2107.03374. · arXiv
- Tri Dao. 2024. FlashAttention-2: Faster attention with better parallelism and work partitioning. In Inter- national Conference on Learning Representations (ICLR).
- DeepSeek-AI. 2024. Deepseek-v2: A strong, economi- cal, and efficient mixture-of-experts language model. Preprint, arXiv:2405.04434. · arXiv
- Elias Frantar and Dan Alistarh. 2023. SparseGPT: Mas- sive language models can be accurately pruned in one-shot. In Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pages 10323-10337.
- PMLR.
- Yichao Fu, Peter Bailis, Ion Stoica, and Hao Zhang. 2024. Break the Sequential Dependency of LLM Inference Using Lookahead Decoding. Preprint, arXiv:2402.02057. · arXiv
- Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. 2023. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805. · arXiv
- Zhuocheng Gong, Jiahao Liu, Ziyue Wang, Pengfei Wu, Jingang Wang, Xunliang Cai, Dongyan Zhao, and Rui Yan. 2024. Graph-structured speculative decoding. In Findings of the Association for Compu- tational Linguistics ACL 2024, pages 11404-11415.
- Zhenyu He, Zexuan Zhong, Tianle Cai, Jason D Lee, and Di He. 2023. Rest: Retrieval-based speculative decoding. arXiv preprint arXiv:2311.08252. · arXiv
- Sehoon Kim, Coleman Hooper, Amir Gholami, Zhen Dong, Xiuyu Li, Sheng Shen, Michael W. Mahoney, and Kurt Keutzer. 2024. SqueezeLLM: Dense-and- Sparse Quantization. Preprint, arXiv:2306.07629. · arXiv
- Siqi Kou, Lanxiang Hu, Zhezhi He, Zhijie Deng, and Hao Zhang. 2024. CLLMs: Consistency Large Lan- guage Models. Preprint, arXiv:2403.00835. · arXiv
- Yaniv Leviathan, Matan Kalman, and Yossi Matias. 2023. Fast Inference from Transformers via Spec- ulative Decoding. In Proceedings of the 40th Inter- national Conference on Machine Learning, pages 19274-19286.
- PMLR.
- Yuhui Li, Fangyun Wei, Chao Zhang, and Hongyang Zhang. 2024a. Eagle-2: Faster inference of language models with dynamic draft trees. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 7421-7432.
- Yuhui Li, Fangyun Wei, Chao Zhang, and Hongyang Zhang. 2024b. Eagle: Speculative sampling requires rethinking feature uncertainty. In Forty-first Interna- tional Conference on Machine Learning.
- Feng Lin, Hanling Yi, Hongbin Li, Yifan Yang, Xi- aotian Yu, Guangming Lu, and Rong Xiao. 2024. BiTA: Bi-Directional Tuning for Lossless Accel- eration in Large Language Models. Preprint, arXiv:2401.12522. · arXiv
- Fangcheng Liu, Yehui Tang, Zhenhua Liu, Yunsheng Ni, Kai Han, and Yunhe Wang. 2024a. Kangaroo: Lossless Self-Speculative Decoding via Double Early Exiting. Preprint, arXiv:2404.18911. · arXiv
- Xiaoxuan Liu, Lanxiang Hu, Peter Bailis, Alvin Che- ung, Zhijie Deng, Ion Stoica, and Hao Zhang. 2024b. Online speculative decoding. In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pages 31131-31146. PMLR.
- Xupeng Miao, Gabriele Oliaro, Zhihao Zhang, Xinhao Cheng, Zeyu Wang, Zhengxin Zhang, Rae Ying Yee Wong, Alan Zhu, Lijie Yang, Xiaoxiang Shi, Chunan Shi, Zhuoming Chen, Daiyaan Arfeen, Reyna Ab- hyankar, and Zhihao Jia. 2024. SpecInfer: Accelerat- ing Large Language Model Serving with Tree-based Speculative Inference and Verification. In Proceed- ings of the 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 3, volume 3 of ASP- LOS '24, pages 932-949. Association for Computing Machinery.
- Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, Jérémy Rapin, et al. 2023. Code llama: Open foundation models for code. arXiv preprint arXiv:2308.12950. · arXiv
- Noam Shazeer. 2019. Fast Transformer Decoding: One Write-Head is All You Need. Preprint, arXiv:1911.02150. · arXiv
- Benjamin Frederick Spector and Christopher Re. Accel- erating llm inference with staged speculative decod- ing. In Workshop on Efficient Systems for Foundation Models@ ICML2023.
- Ziteng Sun, Ananda Theertha Suresh, Jae Hun Ro, Ah- mad Beirami, Himanshu Jain, and Felix Yu. 2023. Spectr: Fast speculative decoding via optimal trans- port. In Thirty-seventh Conference on Neural Infor- mation Processing Systems.
- Yixuan Wang, Xianzhen Luo, Fuxuan Wei, Yijun Liu, Qingfu Zhu, Xuanyu Zhang, Qing Yang, Dongliang Xu, and Wanxiang Che. 2024b. Make some noise: Unlocking language model parallel inference ca- pability through noisy training. arXiv preprint arXiv:2406.17404. · arXiv
- Heming Xia, Zhe Yang, Qingxiu Dong, Peiyi Wang, Yongqi Li, Tao Ge, Tianyu Liu, Wenjie Li, and Zhi- fang Sui. 2024. Unlocking Efficiency in Large Lan- guage Model Inference: A Comprehensive Survey of Speculative Decoding. Preprint, arXiv:2401.07851. · arXiv
- Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. 2024. Efficient streaming lan- guage models with attention sinks. In The Twelfth International Conference on Learning Representa- tions.
- Yuzhuang Xu, Xu Han, Zonghan Yang, Shuo Wang, Qingfu Zhu, Zhiyuan Liu, Weidong Liu, and Wanxiang Che. 2024. OneBit: Towards Ex- tremely Low-bit Large Language Models. Preprint, arXiv:2402.11295. · arXiv
- Nan Yang, Tao Ge, Liang Wang, Binxing Jiao, Daxin Jiang, Linjun Yang, Rangan Majumder, and Furu Wei. 2023. Inference with reference: Lossless acceleration of large language models. Preprint, arXiv:2304.04487. · arXiv
- Hongyi Yuan, Keming Lu, Fei Huang, Zheng Yuan, and Chang Zhou. 2024. Speculative contrastive decoding. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), Bangkok, Thailand. Association for Computational Linguistics.
- Weilin Zhao, Yuxiang Huang, Xu Han, Chaojun Xiao, Zhiyuan Liu, and Maosong Sun. 2024. Ouroboros: Speculative decoding with large model enhanced drafting. arXiv preprint arXiv:2402.13720. · arXiv
- Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023. Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. Preprint, arXiv:2306.05685. · arXiv
- Yongchao Zhou, Kaifeng Lyu, Ankit Singh Rawat, Aditya Krishna Menon, Afshin Rostamizadeh, Sanjiv Kumar, Jean-François Kagy, and Rishabh Agarwal. 2024a. Distillspec: Improving speculative decoding via knowledge distillation. In The Twelfth Interna- tional Conference on Learning Representations.
- Zixuan Zhou, Xuefei Ning, Ke Hong, Tianyu Fu, Ji- aming Xu, Shiyao Li, Yuming Lou, Luning Wang, Zhihang Yuan, Xiuhong Li, et al. 2024b. A survey on efficient inference for large language models. arXiv preprint arXiv:2404.14294. · arXiv
Cited by
Sorted by citations
SAM Decoding: Speculative Decoding via Suffix Automaton
Hu, Y., Wang, K., Zhang, X., Zhang, F., Li, C., Chen, H., & Zhang, J. (2025) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2025.acl-long.595 · Cited by 5
Self-citationCho, S., Choi, S., Hwang, T., Seo, J., Jeong, S., Lee, H., Song, H., Park, J. C., & Kwon, Y. (2025) · Findings of the Association for Computational Linguistics: NAACL · doi:10.18653/v1/2025.findings-naacl.216 · Cited by 2
DReSD: Dense Retrieval for Speculative Decoding
Gritta, M., Xue, H., & Λάμπουρας, Γ. (2025) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2025.findings-acl.1017 · Cited by 1
Faster In-Context Learning for LLMs via N-Gram Trie Speculative Decoding
Chen, J., Li, Q., Li, Z., Qi, B., Guoming, L., Ai, H., Zhao, H., & Wang, P. (2025) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2025.emnlp-main.911 · Cited by 2
Teng, Y., Jiang, Z., Shi, H., Liu, X., Ning, X., DaiMember, G., Wang, Y., Li, Z., & Liu, X. (2026) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2026.3700227 · Cited by 3
Self-citationDIAA: A Decoding-Efficient Inference Acceleration Approach for On-Device Large Language Models
Tian, H., Lu, S., Tian, F., Cui, G., Li, Z., Zhang, X., Sheng, Q. Z., & Dou, W. (2026) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v40i31.39789
Self-citationSuperSpec: Enhanced Verification and Sampling for End-to-End LLM Speculative Decoding
Shen, C., Guo, R., Yang, C., Lin, Y., Zhao, Z., Hu, Y., Chen, S., Liu, X., & Li, K. (2025) · IEEE International Conference on High Performance Computing and Communications; IEEE International Conference on Smart City; IEEE International Conference on Data Science and Systems (HPCC/SmartCity/DSS) · doi:10.1109/hpcc67675.2025.00064
Towards Optimal Speculative Decoding: A Comprehensive Survey on Optimizations and Future Directions
Ji, Y., Xu, W., Qian, X., Zhang, L., & Wan, J. (2026) · ACM Computing Surveys · doi:10.1145/3846171
Metadata
- Authors
- Luo, X., Wang, Y., Zhu, Q., Zhang, Z., Zhang, X., Yang, Q., & Xu, D.
- Affiliations
- Harbin Institute of Technology (CN)
- Venue
- Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)
- Published
- 2025
- Pages
- 6816-6831
- ISSN
- 0736-587X (ISSN-L)
- OpenAlex
- W4412945522
- Cited by
- 2
- References
- 0
- Citations by year
- 2026: 1 · 2025: 1
- FWCI
- 1.91
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Proceedings of the conference - Association for Computational Linguistics. Meeting (publishedVersion, cc-by)
- Funders
- National Natural Science Foundation of China (62236004, 62476073)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Keywords
- large language models, inference acceleration
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
