Accelerating Large Language Model Decoding with Speculative Sampling
Open accessAbstract
We present speculative sampling, an algorithm for accelerating transformer decoding by enabling the generation of multiple tokens from each transformer call. Our algorithm relies on the observation that the latency of parallel scoring of short continuations, generated by a faster but less powerful draft model, is comparable to that of sampling a single token from the larger target model. This is combined with a novel modified rejection sampling scheme which preserves the distribution of the target model within hardware numerics. We benchmark speculative sampling with Chinchilla, a 70 billion parameter language model, achieving a 2-2.5x decoding speedup in a distributed setup, without compromising the sample quality or making modifications to the model itself.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Chen et al.'s speculative sampling uses hard-threshold top-k matching in verification and does not account for probabilistic confidence.
“It demonstrates high speedups in text generation, but the verification step still relies on hardthreshold topk matching. It does not account for probabilistic confidence, making it prone to overfitting or underutilizing valid drafts.”
From Confidence-Modulated Speculative Decoding for Large Language Models (2025) · cites this paper as [6]
The Modified Rejection Sampling section says draft tokens are accepted with probability min(1, q(x)/p(x)), using the target and draft models’ token probabilities—not hard-threshold top-k matching. It describes top-k as an optional sampling method whose probabilities can be adjusted before rejection sampling. The paper reports decoding speedups, but its account of verification contradicts the claim that verification ignores probabilistic confidence. (LLM, conf=0.90)
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredMost existing speculative decoding approaches are limited by static drafting policies, rigid verification strategies, or architectural dependencies that do not generalize well across tasks or model configurations.
“While recent advances in speculative decoding and non-autoregressive generation have made substantial strides toward accelerating sequence generation, most existing approaches are limited by static drafting policies, rigid verification strategies, or architectural dependencies that do not generalize well across tasks or model configurations.”
From Confidence-Modulated Speculative Decoding for Large Language Models (2025) · cites this paper as [6]
The Related Work section says some earlier methods are limited to greedy sampling, bias results, focus on other modalities, or have not been scaled to distributed setups. Choice of Draft Models says some drafting methods require changes to the target model or substantial training data. These passages support specific limitations, but not the broader claim that most existing approaches use static drafting policies or rigid verification strategies, or generally fail across tasks and model configurations. Page numbers are unavailable in our copy. (LLM, conf=0.84)
Experimental — accuracy not yet measuredExtending to temperature-based sampling would require the rejection-sampling correction of Leviathan et al. [12] and Chen et al. [3].
“Extending to temperature-based sampling would require the rejection-sampling correction of Leviathan et al. [12] and Chen et al. [3]”
From Pre-Compiled Pipeline Shards for Distributed LLM Inference on Intel AI PC Fleets (2026) · cites this paper as Chen et al. [3]
Prior works that utilized the attention matrix to compress multiple trajectories adjusted the position ids so each trajectory follows consecutive increasing position ids.
“For these works, on top of the attention matrix, they adjusted the position ids as well, so each trajectory follows a consecutive increasing position ids.”
From MEM1: Learning to Synergize Memory and Reasoning for Efficient Long-Horizon Agents (2025) · cites this paper as [10]
The Speculative Sampling Conditional Scoring section discusses parallel scoring and attention-cache costs, but does not describe compressing multiple trajectories or adjusting position IDs. Searches for position IDs, attention masks, and multiple trajectories found no passage in our copy supporting the attributed procedure; page numbers were unavailable. (LLM, conf=0.75)
Experimental — accuracy not yet measuredPrior works that utilized the attention matrix to compress multiple trajectories mainly targeted tree-exploration.
“Specifically, prior works [28, 10, 8] that utilized the attention matrix to compress multiple trajectories mainly targeted tree-exploration, i.e., generating multiple sequences with the same prefix.”
From MEM1: Learning to Synergize Memory and Reasoning for Efficient Long-Horizon Agents (2025) · cites this paper as [10]
The Speculative Sampling Conditional Scoring section describes scoring a short continuation of tokens in parallel. Our copy does not say that the method uses an attention matrix to compress multiple trajectories or targets tree exploration; searches for those concepts found no supporting passage. (LLM, conf=0.74)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
The publisher hasn’t deposited this paper’s reference list.
Research works printed but not linked: 20
- A. Arnab, M. Dehghani, G. Heigold, C. Sun, M. Lucic, and C. Schmid. Vivit: A video vision transformer. In 2021 IEEE/CVF International Conference on Computer Vision (ICCV), pages 6816-6826. IEEE Computer Society, 2021.
- T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
- M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. de Oliveira Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman, A. Ray, R. Puri, G. Krueger, M. Petrov, H. Khlaaf, G. Sastry, P. Mishkin, B. Chan, S. Gray, N. Ryder, M. Pavlov, A. Power, L. Kaiser, M. Bavarian, C. Winter, P. Tillet, F. P. Such, D. Cummings, M. Plappert, F. Chantzis, E. Barnes, A. Herbert-Voss, W. H. Guss, A. Nichol, A. Paino, N. Tezak, J. Tang, I. Babuschkin, S. Balaji, S. Jain, W. Saunders, C. Hesse, A. N. Carr, J. Leike, J. Achiam, V. Misra, E. Morikawa, A. Radford, M. Knight, M. Brundage, M. Murati, K. Mayer, P. Welinder, B. McGrew, D. Amodei, S. McCandlish, I. Sutskever, and W. Zaremba. Evaluating large language models trained on code. CoRR, abs/2107.03374, 2021. URL https://arxiv.org/abs/2107.03374. · arXiv
- A. Chowdhery, S. Narang, J. Devlin, M. Bosma, G. Mishra, A. Roberts, P. Barham, H. W. Chung, C. Sutton, S. Gehrmann, et al. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311, 2022. · arXiv
- T. Dettmers, M. Lewis, Y. Belkada, and L. Zettlemoyer. Llm. int8 (): 8-bit matrix multiplication for transformers at scale. arXiv preprint arXiv:2208.07339, 2022. · arXiv
- A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Min- derer, G. Heigold, S. Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929, 2020. · arXiv
- T. Ge, H. Xia, X. Sun, S. Chen, and F. Wei. Lossless acceleration for seq2seq generation with aggressive decoding. ArXiv, abs/2205.10350, 2022. · arXiv
- J. Hoffmann, S. Borgeaud, A. Mensch, E. Buchatskaya, T. Cai, E. Rutherford, D. d. L. Casas, L. A. Hendricks, J. Welbl, A. Clark, et al. Training compute-optimal large language models. arXiv preprint arXiv:2203.15556, 2022. · arXiv
- Y. Kim and A. M. Rush. Sequence-level knowledge distillation. CoRR, abs/1606.07947, 2016. URL http://arxiv.org/abs/1606.07947. · arXiv
- Y. Leviathan, M. Kalman, and Y. Matias. Fast inference from transformers via speculative decoding. ArXiv, abs/2211.17192, 2022. · arXiv
- Y. Levine, N. Wies, O. Sharir, H. Bata, and A. Shashua. The depth-to-width interplay in self-attention. arXiv preprint arXiv:2006.12467, 2020. · arXiv
- R. Pope, S. Douglas, A. Chowdhery, J. Devlin, J. Bradbury, A. Levskaya, J. Heek, K. Xiao, S. Agrawal, and J. Dean. Efficiently scaling transformer inference. arXiv preprint arXiv:2211.05102, 2022. · arXiv
- J. W. Rae, S. Borgeaud, T. Cai, K. Millican, J. Hoffmann, F. Song, J. Aslanides, S. Henderson, R. Ring, S. Young, et al. Scaling language models: Methods, analysis & insights from training gopher. arXiv preprint arXiv:2112.11446, 2021. · arXiv
- V. Sanh, L. Debut, J. Chaumond, and T. Wolf. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. arXiv preprint arXiv:1910.01108, 2019. · arXiv
- N. Shazeer. Fast transformer decoding: One write-head is all you need. CoRR, abs/1911.02150, 2019. URL http://arxiv.org/abs/1911.02150. · arXiv
- M. Shoeybi, M. Patwary, R. Puri, P. LeGresley, J. Casper, and B. Catanzaro. Megatron-lm: Training multi-billion parameter language models using model parallelism. arXiv preprint arXiv:1909.08053, 2019. · arXiv
- Y. Song, C. Meng, R. Liao, and S. Ermon. Accelerating feedforward computation via parallel nonlinear equation solving. In M. Meila and T. Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 9791-9800. PMLR, 18-24 Jul 2021. URL https://proceedings.mlr.press/v139/song21a.html.
- M. Stern, N. Shazeer, and J. Uszkoreit. Blockwise parallel decoding for deep autoregressive models. CoRR, abs/1811.03115, 2018. URL http://arxiv.org/abs/1811.03115. · arXiv
- A. Wiggers and E. Hoogeboom. Predictive sampling with forecasting autoregressive models. In H. D. III and A. Singh, editors, Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 10260-10269. PMLR, 13-18 Jul 2020. URL https://proceedings.mlr.press/v119/wiggers20a.html.
- Z. Yao, R. Y. Aminabadi, M. Zhang, X. Wu, C. Li, and Y. He. Zeroquant: Efficient and affordable post-training quantization for large-scale transformers. arXiv preprint arXiv:2206.01861, 2022. · arXiv
Cited by
Sorted by citations
A Survey on RAG Meeting LLMs: Towards Retrieval-Augmented Large Language Models
Fan, W., Ding, Y., Ning, L., Wang, S., Li, H., Yin, D., Chua, T.-S., & Li, Q. (2024) · Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining · doi:10.1145/3637528.3671470 · Cited by 827 · see citing works
GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints
Ainslie, J., Lee-Thorp, J., de Jong, M., Zemlyanskiy, Y., Lebrón, F., & Sanghai, S. (2023) · arXiv · doi:10.48550/arxiv.2305.13245 · Cited by 447 · see citing works
“Speculative sampling (Chen et al., 2023; Leviathan et al., 2022; Kim et al., 2023) ameliorates the memory bandwidth bottleneck by proposing multiple tokens with a smaller model which are then scored in parallel by a larger model.”
BackgroundGQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints
Ainslie, J., Lee-Thorp, J., de Jong, M., Zemlyanskiy, Y., Lebron, F., & Sanghai, S. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.298 · Cited by 454 · see citing works
Scalable watermarking for identifying large language model outputs
Dathathri, S., See, A., Ghaisas, S., Huang, P.-S., McAdam, R., Welbl, J., Bachani, V., Kaskasoli, A., Stanforth, R., Matejovicova, T., et al. (2024) · Nature · doi:10.1038/s41586-024-08025-4 · Cited by 132 · see citing works
“Furthermore, we provide an algorithm to combine generative watermarking with speculative sampling 5 —a frequently used technique to increase LLM text generation speed—allowing for the integration of SynthID-Text into large-scale production systems with negligible additional computational overhead.”
MethodInfluentialUnderstanding the Potential of FPGA-based Spatial Acceleration for Large Language Model Inference
Chen, H., Zhang, J., Du, Y., Xiang, S., Yue, Z., Zhang, N., Cai, Y., & Zhang, Z. (2024) · ACM Transactions on Reconfigurable Technology and Systems · doi:10.1145/3656177 · Cited by 86 · see citing works
Chen, B., Cheng, X., Li, P., Geng, Y., Gong, J., Li, S., Bei, Z., Tan, X., Wang, B., Zeng, X., et al. (2025) · Nature Methods · doi:10.1038/s41592-025-02636-z · Cited by 174 · see citing works
FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance
Chen, L., Zaharia, M., & Zou, J. (2023) · arXiv · doi:10.48550/arxiv.2305.05176 · Cited by 51 · see citing works
“Speculative decoding has recently emerged for LLM inference acceleration without retraining or model architecture modification Leviathan et al. (2023); Chen et al. (2023); Sun et al. (2023).”
EdgeLLM: Fast On-Device LLM Inference With Speculative Decoding
Xu, D., Yin, W., Zhang, H., Jin, X., Zhang, Y., Wei, S., Xu, M., & Liu, X. (2024) · IEEE Transactions on Mobile Computing · doi:10.1109/tmc.2024.3513457 · Cited by 52 · see citing works
Wang, R., Gao, Z., Zhang, L., Yue, S., & Gao, Z. (2025) · Computer Science Review · doi:10.1016/j.cosrev.2025.100755 · Cited by 51 · see citing works
Resource-efficient Algorithms and Systems of Foundation Models: A Survey
Xu, M., Cai, D., Yin, W., Wang, S., Jin, X., & Liu, X. (2024) · ACM Computing Surveys · doi:10.1145/3706418 · Cited by 153 · see citing works
Speculative Decoding: Exploiting Speculative Execution for Accelerating Seq2seq Generation
Xia, H., Ge, T., Wang, P., Chen, S.-Q., Wei, F., & Sui, Z. (2023) · Findings of the Association for Computational Linguistics: EMNLP 2023 · doi:10.18653/v1/2023.findings-emnlp.257 · Cited by 40 · see citing works
“Chen et al. (2023) is similar to Leviathan et al. (2023) but it was the first to validate this methodology to accelerate a large language model (i.e., 70B Chinchilla) with a 4B drafter model, thus receiving the most attention.”
MethodEfficient Diffusion Models: A Comprehensive Survey From Principles to Practices
Ma, Z., Zhang, Y., Jia, G., Zhao, L., Ma, Y., Ma, M., Liu, G., Zhang, K., Ding, N., Li, J., et al. (2025) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2025.3569700 · Cited by 46 · see citing works
MELTing Point: Mobile Evaluation of Language Transformers
Laskaridis, S., Katevas, K., Minto, L., & Haddadi, H. (2024) · Proceedings of the 30th Annual International Conference on Mobile Computing and Networking · doi:10.1145/3636534.3690668 · Cited by 37 · see citing works
Accelerating Transformer Inference for Translation via Parallel Decoding
Santilli, A., Severino, S., Postolache, E., Maiorca, V., Mancusi, M., Marin, R., & Rodola, E. (2023) · Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2023.acl-long.689 · Cited by 29 · see citing works
Large language models as tool makers
Cai, T., Wang, X., Ma, T., Chen, X., & Zhou, D. (2023) · arXiv · doi:10.48550/arxiv.2305.17126 · Cited by 28 · see citing works
Spotting LLMs With Binoculars: Zero-Shot Detection of Machine-Generated Text
Hans, A., Schwarzschild, A., Cherepanova, V., Kazemi, H., Saha, A., Goldblum, M., Geiping, J., & Goldstein, T. (2024) · arXiv · doi:10.48550/arxiv.2401.12070 · Cited by 24 · see citing works
Benchmarking Uncertainty Quantification Methods for Large Language Models with LM-Polygraph
Vashurin, R., Fadeeva, E., Vazhentsev, A., Rvanova, L., Vasilev, D., Tsvigun, A., Petrakov, S., Xing, R., Sadallah, A., Grishchenkov, K., et al. (2025) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00737 · Cited by 21 · see citing works
Generative AI: Overview, Economic Impact, and Applications in Asset ManagementPreprint
Luk, M. (2023) · SSRN Electronic Journal · doi:10.2139/ssrn.4574814 · Cited by 22 · see citing works
Bae, S., Ko, J., Song, H., & Yun, S.-Y. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.362 · Cited by 28 · see citing works
MoE-L ightning : High-Throughput MoE Inference on Memory-constrained GPUs
Cao, S., Liu, S., Griggs, T., Schafhalter, P., Liu, X., Sheng, Y., Gonzalez, J. E., Zaharia, M., & Stoica, I. G. (2025) · International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS) · doi:10.1145/3669940.3707267 · Cited by 22 · see citing works
throttLL’eM: Predictive GPU Throttling for Energy Efficient LLM Inference Serving
Kakolyris, A. K., Masouros, D., Vavaroutsos, P., Xydis, S., & Soudris, D. (2025) · IEEE International Symposium on High-Performance Computer Architecture (HPCA) · doi:10.1109/hpca61900.2025.00103 · Cited by 20 · see citing works
SumGPT: A Multimodal Framework for Radiology Report Summarization to Improve Clinical Performance
Sultan, T., Rony, M. A. T., Islam, M. S., AlShathri, S. I., & El‐Shafai, W. (2025) · IEEE Access · doi:10.1109/access.2025.3528335 · Cited by 16 · see citing works
OPT-Tree: Speculative Decoding with Adaptive Draft Tree Structure
Wang, J., Su, Y., Li, J., Xia, Q., Ye, Z., Duan, X., Wang, Z., & Zhang, M. (2025) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00735 · Cited by 11 · see citing works
SPIN: Accelerating Large Language Model Inference with Heterogeneous Speculative Models
Chen, F., Li, P., Luan, T. H., Su, Z., & Deng, J. (2025) · IEEE INFOCOM 2025 - IEEE Conference on Computer Communications · doi:10.1109/infocom55648.2025.11044522 · Cited by 11 · see citing works
Zhao, Y., Xie, Z., Liang, C., Zhuang, C., & Gu, J. (2024) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3637528.3671614 · Cited by 13 · see citing works
Liu, C., Li, Y., Chen, C., Kuang, H., Ma, X., Zou, X., Liu, J., Lu, Z., Zhang, Z., & Liu, X. (2025) · IEEE Internet of Things Journal · doi:10.1109/jiot.2025.3550841 · Cited by 8 · see citing works
Self-citationZhang, C., You, Y., Wang, N., Park, J., & Zhang, L. (2025) · IEEE Journal on Emerging and Selected Topics in Circuits and Systems · doi:10.1109/jetcas.2025.3575272 · Cited by 11 · see citing works
Efficient Benchmarking (of Language Models)
Perlitz, Y., Bandel, E., Gera, A., Arviv, O., Ein-Dor, L., Shnarch, E., Slonim, N., Shmueli-Scheuer, M., & Choshen, L. (2024) · Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers) · doi:10.18653/v1/2024.naacl-long.139 · Cited by 7 · see citing works
SnapKV: LLM Knows What You are Looking for Before Generation
Li, Y., Huang, Y., Yang, B., Venkitesh, B., Locatelli, A., Ye, H., Cai, T., Lewis, P., & Chen, D. (2024) · arXiv · doi:10.48550/arxiv.2404.14469 · Cited by 74 · see citing works
Debiasing Watermarks for Large Language Models via Maximal Coupling
Xie, Y., Li, X., Mallick, T., Su, W., & Zhang, R. (2025) · Journal of the American Statistical Association · doi:10.1080/01621459.2025.2520455 · Cited by 6 · see citing works
Dynamic Quality-Latency Aware Routing for LLM Inference in Wireless Edge-Device Networks
Bao, R., Xue, N., Sun, Y., & Chen, Z. (2025) · IEEE/CIC International Conference on Communications in China (ICCC) · doi:10.1109/icccworkshops67136.2025.11147210 · Cited by 4 · see citing works
Accelerating Codec-based Speech Synthesis with Multi-Token Prediction and Speculative Decoding
Nguyen, T. D., Kim, J., Choi, J., Choi, S., Park, J., Lee, Y., & Chung, J. S. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10887855 · Cited by 4 · see citing works
Proceedings of the Workshop on Edge and Mobile Foundation Models
(2024) · doi:10.1145/3662006 · Cited by 4 · see citing works
BiTA: Bi-directional tuning for lossless acceleration in large language models
Lin, F., Yi, H., Yang, Y., Li, H., Yu, X., Lu, G., & Xiao, R. (2025) · Expert Systems with Applications · doi:10.1016/j.eswa.2025.127305 · Cited by 3 · see citing works
Faster Speech-LLaMA Inference with Multi-token Prediction
Raj, D., Keren, G., Jia, J., Mahadeokar, J., & Kalinli, O. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10890328 · Cited by 2 · see citing works
Whisper in Medusa’s Ear: Multi-head Efficient Decoding for Transformer-based ASR
Segal, Y., Shamsian, A., Navon, A., Hetz, G., & Keshet, J. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10888140 · Cited by 2 · see citing works
Predicting Compact Phrasal Rewrites with Large Language Models for ASR Post Editing
Zhang, H., Stahlberg, F., & Kumar, S. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10890428 · Cited by 1 · see citing works
Enhancing Stateful Processing in Programmable Data Planes: Model and Improved Architecture
Feng, Y., Zhou, H., Liu, S., Chen, Z., Song, H., & Liu, B. (2024) · IEEE Transactions on Networking · doi:10.1109/tnet.2024.3510638 · Cited by 1 · see citing works
MEM1: Learning to Synergize Memory and Reasoning for Efficient Long-Horizon Agents
Zhou, Z., Qu, A., Wu, Z., Kim, S., Prakash, A., Rus, D., Zhao, J., Low, B. K. H., & Liang, P. P. (2025) · arXiv · doi:10.48550/arxiv.2506.15841
Gao, X., Xie, W., Lixin, Fang, X., Hang, C., Li, C., Lin, Y., & Xu, X. (2026) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v40i25.39266
DIAA: A Decoding-Efficient Inference Acceleration Approach for On-Device Large Language Models
Tian, H., Lu, S., Tian, F., Cui, G., Li, Z., Zhang, X., Sheng, Q. Z., & Dou, W. (2026) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v40i31.39789
Adaptive hybrid speculative decoding for accelerating large language model inference
Yong, Y., Yang, T. T., Gao, S. S., & Yin, Y. H. (2026) · Neurocomputing · doi:10.1016/j.neucom.2026.133760
Confidence-Modulated Speculative Decoding for Large Language Models
Sen, J., Dasgupta, S., & Waghela, H. (2025) · 2025 IEEE 6th India Council International Subsections Conference (INDISCON) · doi:10.1109/indiscon66021.2025.11254640
The Energy Cost of Reasoning: Analyzing Energy Usage in LLMs with Test-time Compute
Jin, Y., Wei, G.-Y., & Brooks, D. (2025) · arXiv · doi:10.48550/arxiv.2505.14733
WWW.Serve: Interconnecting Global LLM Services through Decentralization
Wang, H., Xia, Z., Chen, Z., & Chen, B. (2026) · arXiv · doi:10.48550/arxiv.2603.20661
Pre-Compiled Pipeline Shards for Distributed LLM Inference on Intel AI PC Fleets
Berenbaum, T., & Venkatachalam, M. (2026) · arXiv · doi:10.48550/arxiv.2608.19147
Draft Model Knows When to Stop: A Self-Verification Length Policy for Speculative DecodingPreprint
Zhang, Z., Xu, J., Liang, T., Chen, X., He, Z., Wang, R., & Tu, Z. (2024) · Qeios · doi:10.32388/i3vw8j
Efficient pipeline for retrieval-augmented generation system under big data
Yu, R., Yang, Y., Zhou, J., & Wu, F. (2025) · Scientia Sinica Informationis · doi:10.1360/ssi-2024-0331
Wen, Z., Xu, L., & Wang, M. (2025) · Integrated Circuits and Systems · doi:10.23919/ics.2025.3575371
KOALA: Enhancing Speculative Decoding for LLM via Multi-Layer Draft Heads with Adversarial Learning
Zhang, K., Zhao, J., & Chen, R. (2025) · International Conference on Computer Supported Cooperative Work in Design (CSCWD) · doi:10.1109/cscwd64889.2025.11033265
Masker: Speculative Decoding for Long-Text QA with CopyHead
Zhou, Y., Pan, H., Liu, R., & Zhang, L. (2025) · Communications in computer and information science · doi:10.1007/978-981-96-6954-7_3
Grumeza, T.-R., Staniloiu, G.-L., & Fortis, A. (2026) · Lecture notes on data engineering and communications technologies · doi:10.1007/978-3-032-23335-6_24
GenFacet: End-to-End Generative Faceted Search via Multi-Task Preference Alignment in E-Commerce
Zhai, Z., Yang, M., & Li, J. (2026) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3805712.3808451 · Cited by 1 · see citing works
Schema-Conditioned Next-Tool Prediction for Coding Agents under Interface Drift
Nyberg, S., & Karlsson, I. (2026) · International Workshop on Agentic AI for Next-Generation Software Development · doi:10.1145/3843282.3844430
Qiu, Y., Wang, S., Qin, Y., Geng, M., & Dong, D. (2026) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/3832783.3837454
- Authors
- Chen, C., Borgeaud, S., Irving, G., Lespiau, J.-B., Sifre, L., & Jumper, J.
- ORCID
- Jumper, John
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2023
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2302.01318
- OpenAlex
- W4319166707
- Cited by
- 40
- References
- 0
- Citations by year
- 2026: 4 · 2025: 21 · 2024: 10 · 2023: 5
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Speech Recognition and Synthesis
- Keywords
- draft model, rejection sampling, distributed decoding, chinchilla
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
