CoChat
Sign Up
PreprintCited by 71Natural Language Processing TechniquesTopic ModelingAdvanced Neural Network Applications

Fast Inference from Transformers via Speculative Decoding

Open access
Leviathan, Yaniv, Kalman, Matan, Matias, Yossi·2022·arXiv
DOI 10.48550/arxiv.2211.17192arXiv 2211.17192
Open this paper in CoChat — free accountOpen PDF

Abstract

Inference from large autoregressive models like Transformers is slow - decoding K tokens takes K serial runs of the model. In this work we introduce speculative decoding - an algorithm to sample from autoregressive models faster without any changes to the outputs, by computing several tokens in parallel. At the heart of our approach lie the observations that (1) hard language-modeling tasks often include easier subtasks that can be approximated well by more efficient models, and (2) using speculative execution and a novel sampling method, we can make exact decoding from the large models faster, by running them in parallel on the outputs of the approximation models, potentially generating several tokens concurrently, and without changing the distribution. Our method can accelerate existing off-the-shelf models without retraining or architecture changes. We demonstrate it on T5-XXL and show a 2X-3X acceleration compared to the standard T5X implementation, with identical outputs.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2211.17192

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Prior works that utilized the attention matrix to compress multiple trajectories mainly targeted tree-exploration.

    “Specifically, prior works [28, 10, 8] that utilized the attention matrix to compress multiple trajectories mainly targeted tree-exploration, i.e., generating multiple sequences with the same prefix.”

    From MEM1: Learning to Synergize Memory and Reasoning for Efficient Long-Horizon Agents (2025) · cites this paper as [28]

    Section 2.1 (p. 2) describes the paper’s main method as generating candidate tokens with an approximation model and checking them in parallel, not exploring multiple sequences that share a prefix. Appendix A.4 (p. 12) treats beam search as a possible application whose analysis is left for future work, contrary to the claim that this paper mainly targeted tree exploration. Our copy also does not describe compressing trajectories with an attention matrix. (LLM, conf=0.80)

    Checked by a reasoning model — may be wrong.

    Experimental — accuracy not yet measured

  • Speculative decoding [38], offloaded inference [43, 44], and weight quantization [45, 46] reduce per-request work within a pool, complementary to Janus's routing.

    “Speculative decoding [38], offloaded inference [43, 44], and weight quantization [45, 46] reduce per-request work within a pool, complementary to Janus 's routing.”

    From Janus: Joint Prefill/Decode Disaggregation with KV-Cache-Aware Multi-Cloud Routing for Edge-Adjacent LLM Serving (2026) · cites this paper as [38]

    The Discussion (p. 8) says speculative decoding improves latency through increased concurrency at the cost of more arithmetic operations. It therefore contradicts the attributed claim that speculative decoding reduces per-request work, although it does report inference speedups. (LLM, conf=0.88)

    Checked by a reasoning model — may be wrong.

    Experimental — accuracy not yet measured

  • Most existing speculative decoding approaches are limited by static drafting policies, rigid verification strategies, or architectural dependencies that do not generalize well across tasks or model configurations.

    “While recent advances in speculative decoding and non-autoregressive generation have made substantial strides toward accelerating sequence generation, most existing approaches are limited by static drafting policies, rigid verification strategies, or architectural dependencies that do not generalize well across tasks or model configurations.”

    From Confidence-Modulated Speculative Decoding for Large Language Models (2025) · cites this paper as [5]

    The Discussion (p. 8) says this paper fixed its approximation model and number of guesses during inference, and Related work (p. 7) describes restrictions of two earlier methods. Those passages support examples of limited drafting and decoding choices, but not the broader assertion that most existing approaches have rigid verification strategies or fail to generalize across tasks and model configurations. The paper describes its own method as usable without architecture changes. (LLM, conf=0.84)

    Experimental — accuracy not yet measured

  • Leviathan et al.'s speculative decoding for GPT-style decoders relies on a fixed number of drafted tokens per iteration and a strict binary verification scheme.

    “This approach enables efficient decoding for LLMs but continues to rely on a fixed number of drafted tokens per iteration and a strict binary verification scheme.”

    From Confidence-Modulated Speculative Decoding for Large Language Models (2025) · cites this paper as [5]

  • Prior works that utilized the attention matrix to compress multiple trajectories adjusted the position ids so each trajectory follows consecutive increasing position ids.

    “For these works, on top of the attention matrix, they adjusted the position ids as well, so each trajectory follows a consecutive increasing position ids.”

    From MEM1: Learning to Synergize Memory and Reasoning for Efficient Long-Horizon Agents (2025) · cites this paper as [28]

    The Overview (p. 2) describes evaluating guessed tokens in parallel, and Related work (p. 7) discusses earlier parallel-decoding methods. Neither passage addresses using an attention matrix to combine multiple trajectories or adjusting position IDs to increase consecutively within each trajectory. Searches for position IDs, positional encodings, attention matrices, and trajectories found no passage establishing the claim. (LLM, conf=0.75)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

The citation lists haven’t been fetched yet

Cited by 71

Research works printed but not linked: 28

  1. Bapna, A., Arivazhagan, N., and Firat, O. Controlling computation versus quality for neural sequence models. ArXiv , abs/2002.07106, 2020. · arXiv
  2. Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D. M., Wu, J., Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S., Chess, B., Clark, J., Berner, C., McCandlish, S., Radford, A., Sutskever, I., and Amodei, D. Language models are few-shot learners. In Proceedings of the 34th International Conference on Neural Information Processing Systems , NIPS'20, Red Hook, NY, USA, 2020. Curran Associates Inc. ISBN 9781713829546.
  3. Burton, F. W. Speculative computation, parallelism, and functional programming. IEEE Transactions on Computers , C-34(12):1190-1193, 1985. doi: 10.1109/TC.1985. 6312218. · DOI
  4. Chelba, C., Mikolov, T., Schuster, M., Ge, Q., Brants, T., Koehn, P. T., and Robinson, T. One billion word bench- mark for measuring progress in statistical language modeling. In Interspeech , 2013.
  5. Chen, C., Borgeaud, S., Irving, G., Lespiau, J.-B., Sifre, L., and Jumper, J. M. Accelerating large language model decoding with speculative sampling. ArXiv , abs/2302.01318, 2023. · arXiv
  6. Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H. W., Sutton, C., Gehrmann, S., Schuh, P., Shi, K., Tsvyashchenko, S., Maynez, J., Rao, A., Barnes, P., Tay, Y., Shazeer, N. M., Prabhakaran, V., Reif, E., Du, N., Hutchinson, B. C., Pope, R., Bradbury, J., Austin, J., Isard, M., Gur-Ari, G., Yin, P., Duke, T., Levskaya, A., Ghemawat, S., Dev, S., Michalewski, H., Garc´ ıa, X., Misra, V., Robinson, K., Fedus, L., Zhou, D., Ippolito, D., Luan, D., Lim, H., Zoph, B., Spiridonov, A., Sepassi, R., Dohan, D., Agrawal, S., Omernick, M., Dai, A. M., Pillai, T. S., Pellat, M., Lewkowycz, A., Moreira, E., Child, R., Polozov, O., Lee, K., Zhou, Z., Wang, X., Saeta, B., D´ ıaz, M., Firat, O., Catasta, M., Wei, J., Meier-Hellstern, K. S., Eck, D., Dean, J., Petrov, S., and Fiedel, N. Palm: Scaling language modeling with pathways. ArXiv , abs/2204.02311, 2022. · arXiv
  7. Dehghani, M., Arnab, A., Beyer, L., Vaswani, A., and Tay, Y. The efficiency misnomer. ArXiv , abs/2110.12894, 2021. · arXiv
  8. Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. Bert: Pre-training of deep bidirectional transformers for language understanding. ArXiv , abs/1810.04805, 2019. · arXiv
  9. Elbayad, M., Gu, J., Grave, E., and Auli, M. Depth-adaptive transformer. ArXiv , abs/1910.10073, 2019. · arXiv
  10. Han, Y., Huang, G., Song, S., Yang, L., Wang, H., and Wang, Y. Dynamic neural networks: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence , 44: 7436-7456, 2021.
  11. Hendrycks, D. and Gimpel, K. Bridging nonlinearities and stochastic regularizers with gaussian error linear units. ArXiv , abs/1606.08415, 2016. · arXiv
  12. Hennessy, J. L. and Patterson, D. A. Computer Architecture: A Quantitative Approach . Morgan Kaufmann, Amsterdam, 5 edition, 2012. ISBN 978-0-12-383872-8.
  13. Hinton, G. E., Vinyals, O., and Dean, J. Distilling the knowledge in a neural network. ArXiv , abs/1503.02531, 2015. · arXiv
  14. Hubara, I., Courbariaux, M., Soudry, D., El-Yaniv, R., and Bengio, Y. Quantized neural networks: Training neural networks with low precision weights and activations. ArXiv , abs/1609.07061, 2016. · arXiv
  15. Jaszczur, S., Chowdhery, A., Mohiuddin, A., Kaiser, L., Gajewski, W., Michalewski, H., and Kanerva, J. Sparse is enough in scaling transformers. In Neural Information Processing Systems , 2021.
  16. Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., and Liu, P. J. Exploring the limits of transfer learning with a unified text-to-text transformer. The Journal of Machine Learning Research , 21(1):5485-5551, 2020.
  17. Roberts, A., Chung, H. W., Levskaya, A., Mishra, G., Bradbury, J., Andor, D., Narang, S., Lester, B., Gaffney, C., Mohiuddin, A., Hawthorne, C., Lewkowycz, A., Salcianu, A., van Zee, M., Austin, J., Goodman, S., Soares, L. B., Hu, H., Tsvyashchenko, S., Chowdhery, A., Bastings, J., Bulian, J., Garc´ ıa, X., Ni, J., Chen, A., Kenealy, K., Clark, J., Lee, S., Garrette, D. H., Lee-Thorp, J., Raffel, C., Shazeer, N. M., Ritter, M., Bosma, M., Passos, A., Maitin-Shepard, J. B., Fiedel, N., Omernick, M., Saeta, B., Sepassi, R., Spiridonov, A., Newlan, J., and Gesmundo, A. Scaling up models and data with t5x and seqio. ArXiv , abs/2203.17189, 2022. · arXiv
  18. Scardapane, S., Scarpiniti, M., Baccarelli, E., and Uncini, A. Why should we add early exits to neural networks? Cognitive Computation , 12(5):954-966, 2020.
  19. Schuster, T., Fisch, A., Jaakkola, T., and Barzilay, R. Consistent accelerated inference via confident adaptive transformers. In Conference on Empirical Methods in Natural Language Processing , 2021.
  20. Schwartz, R., Stanovsky, G., Swayamdipta, S., Dodge, J., and Smith, N. A. The right tool for the job: Matching model and instance complexities. In Annual Meeting of the Association for Computational Linguistics , 2020.
  21. Shazeer, N. M. Fast transformer decoding: One write-head is all you need. ArXiv , abs/1911.02150, 2019. · arXiv
  22. So, D. R., Ma'nke, W., Liu, H., Dai, Z., Shazeer, N. M., and Le, Q. V. Primer: Searching for efficient transformers for language modeling. ArXiv , abs/2109.08668, 2021. · arXiv
  23. Stern, M., Shazeer, N., and Uszkoreit, J. Blockwise parallel decoding for deep autoregressive models. Advances in Neural Information Processing Systems , 31, 2018.
  24. Sukhbaatar, S., Grave, E., Bojanowski, P., and Joulin, A. Adaptive attention span in transformers. In Annual Meeting of the Association for Computational Linguistics , 2019.
  25. Sun, X., Ge, T., Wei, F., and Wang, H. Instantaneous grammatical error correction with shallow aggressive decoding. ArXiv , abs/2106.04970, 2021. · arXiv
  26. Thoppilan, R., Freitas, D. D., Hall, J., Shazeer, N. M., Kulshreshtha, A., Cheng, H.-T., Jin, A., Bos, T., Baker, L., Du, Y., Li, Y., Lee, H., Zheng, H., Ghafouri, A., Menegali, M., Huang, Y., Krikun, M., Lepikhin, D., Qin, J., Chen, D., Xu, Y., Chen, Z., Roberts, A., Bosma, M., Zhou, Y., Chang, C.-C., Krivokon, I. A., Rusch, W. J., Pickett, M., Meier-Hellstern, K. S., Morris, M. R., Doshi, T., Santos, R. D., Duke, T., Søraker, J. H., Zevenbergen, B., Prabhakaran, V., D´ ıaz, M., Hutchinson, B., Olson, K., Molina, A., Hoffman-John, E., Lee, J., Aroyo, L., Rajakumar, R., Butryna, A., Lamm, M., Kuzmina, V. O., Fenton, J., Cohen, A., Bernstein, R., Kurzweil, R., Aguera-Arcas, B., Cui, C., Croak, M., hsin Chi, E. H., and Le, Q. Lamda: Language models for dialog applications. ArXiv , abs/2201.08239, 2022. · arXiv
  27. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., and Polosukhin, I. Attention is all you need. Advances in neural information processing systems , 30, 2017.
  28. Yu, J., Xu, Y., Koh, J. Y., Luong, T., Baid, G., Wang, Z., Vasudevan, V., Ku, A., Yang, Y., Ayan, B. K., Hutchinson, B. C., Han, W., Parekh, Z., Li, X., Zhang, H., Baldridge, J., and Wu, Y. Scaling autoregressive models for contentrich text-to-image generation. ArXiv , abs/2206.10789, 2022. · arXiv

Authors
Leviathan, Y., Kalman, M., & Matias, Y.
Affiliations
Google (United States) (US)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2022
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2211.17192
arXiv
2211.17192
OpenAlex
W4310561894
Cited by
71
References
0
Citations by year
2026: 2 · 2025: 35 · 2024: 27 · 2023: 7
Type
posted content
Language
en
Open access
Licence
creativecommons.org/licenses/by/4.0/legalcode
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University) (cc-by)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
Keywords
speculative decoding, inference acceleration, autoregressive language models, transformer models, parallel decoding, approximate models

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review