Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat

The DOI matches its Crossref record; 42 of the 42 references it cites were found in a registry or an index; 32 of the 39 sources its claims cite were checked for support.

Confidence-Modulated Speculative Decoding for Large Language Models

Open access
Sen, Jaydip, Dasgupta, Subhasis, Waghela, Hetvi · 2025 · 2025 IEEE 6th India Council International Subsections Conference (INDISCON)

Abstract

Speculative decoding has proven to be a powerful method for speeding up autoregressive inference by allowing tokens to be generated in parallel using a draft-then-verify approach. However, the current techniques tend to rely on fixed lengths for drafting and strict criteria for verification, which limits their flexibility when faced with different levels of model uncertainty or varying input complexities. In this paper, we propose a new framework for speculative decoding, built on the idea of adjusting drafting based on confidence levels. By using entropy and margin-based uncertainty measures derived from the drafter’s output distribution, the approach can dynamically change the number of tokens it generates at each step. This adaptability not only reduces the need for rollbacks but also makes better use of available resources, all while ensuring that the output remains accurate. The process for verifying these drafts is also adjusted based on confidence signals, which makes it easier to accept tokens without sacrificing the quality of the generation. When tested on machine translation and summarization tasks, this method delivers impressive speedups over standard speculative decoding and even improves BLEU and ROUGE scores. This approach provides a flexible, easy-toimplement solution for more efficient and reliable decoding in large language models, especially when faced with uncertain conditions.

Overview

CoChat Summary

TL;DR

The paper proposes CM-ASD, a confidence-modulated adaptive speculative decoding framework that uses entropy, logit-margin, and softmax-margin confidence to dynamically set drafting length and verification threshold. It reports up to 5.0x translation speedup and 4.0x summarization speedup while maintaining BLEU/ROUGE near baselines, though summarization ROUGE and speed remain below fixed-window SpecDec(k=25).

Key findings

  • On WMT translation, CM-ASD speedups vs Transformer-base (beam=5) are EN-DE 4.7x, DE-EN 5.0x, EN-RO 4.2x, RO-EN 4.4x, with BLEU 28.91, 32.58, 35.38, 34.94. SpecDec(k=25) reports higher speeds of 5.1x, 5.5x, 4.6x, 4.8x and BLEU 28.93, 32.55, 35.45, 35.03; the text states CM-ASD delivers “over 4.4x” on all four tasks, though Table I lists 4.2x for EN-RO.
  • Ablation on WMT14 EN-DE: full CM-ASD achieves 4.27 accepted tokens per iteration, BLEU 28.91, drafting time 3.81 ms, and 4.7x speed. Removing confidence modulation drops to 2.91 tokens, BLEU 28.73, and 3.2x; entropy-only, margin-only, and softmax-only confidence variants reach 4.2x, 4.1x, and 4.1x respectively.
  • On CNN/DailyMail summarization with BART-base, CM-ASD achieves 4.0x speed with ROUGE-1 41.90, ROUGE-2 18.30, ROUGE-L 38.10. SpecDec(k=25) reaches 5.1x with ROUGE-1 43.11, ROUGE-2 20.43, ROUGE-L 40.19; AR BART-base (b=5) is 1.0x with 43.08, 20.41, 40.15. CM-ASD outperforms non-autoregressive models in all ROUGE metrics.
  • Relative BLEU against Transformer-base greedy output is 87.34 for CM-ASD, 86.52 for SpecDec(k=25), and 59.10–65.42 for non-autoregressive baselines; the paper states CM-ASD maintains over 87% output alignment.

Methods

  • Design: CM-ASD augments speculative decoding with confidence estimation from the drafter’s output distribution using entropy, logit margin, and softmax margin, combined into a weighted ensemble \(C_t = \lambda_1 \tilde{C}_t^{(ent)} + \lambda_2 \tilde{C}_t^{(margin)} + \lambda_3 \tilde{C}_t^{(soft)}\), with \(\sum \lambda_i = 1\). Drafting length is dynamically set as \(k_j = \min(k_{\max}, \max(k_{\min}, \lfloor \alpha \cdot \bar{C}_{j:k} \cdot k_{\max} \rfloor))\), and verification threshold as \(\tau_t = \tau_{base} + \gamma \cdot (1 - C_t)\).
  • Data and models: WMT’14 English-German and WMT’16 English-Romanian translation tasks (tables also report DE-EN and RO-EN), and CNN/DailyMail abstractive summarization. Translation/summarization use a Transformer encoder-decoder with 6 encoder and 6 decoder layers; open-ended generation uses a decoder-only model with 12 layers, 768 hidden size, and 12 attention heads. Drafters are a 2-layer decoder for encoder-decoder and a 3-layer Transformer for decoder-only; summarization uses BART-base as verifier with drafter initialized from the BART encoder. Implementation is PyTorch/Fairseq.
  • Analysis: Baselines include autoregressive greedy, SpecDec with fixed \(k\) and \(\tau\), blockwise decoding with \(k=10\) and \(k=25\), and non-autoregressive models. Metrics include BLEU via SacreBLEU, ROUGE-1/2/L F1, average tokens verified per iteration, rollback frequency, and wall-clock decoding time. Experiments run on a single NVIDIA A100 GPU with batch size 16, three random seeds, and reported mean and standard deviation.

Limitations

  • The paper does not state a limitations section; it lists future work directions such as multimodal generation, richer uncertainty measures, model compression, and instruction-tuned/dialogue LLMs.
  • Evident from the results: on CNN/DailyMail summarization, CM-ASD’s ROUGE scores (41.90/18.30/38.10) and speed (4.0x) are below SpecDec(k=25) (43.11/20.43/40.19; 5.1x); the paper acknowledges its ROUGE scores “fall slightly below” SpecDec.
  • Evident from the results: on WMT translation, CM-ASD speed is below SpecDec(k=25) on all four tasks (e.g., 4.7x vs 5.1x EN-DE; 5.0x vs 5.5x DE-EN; 4.2x vs 4.6x EN-RO; 4.4x vs 4.8x RO-EN), and BLEU is slightly lower than SpecDec(k=25) on EN-DE, EN-RO, and RO-EN. No open-ended generation results are reported despite the setup mentioning a decoder-only model, and hyperparameters \(\alpha\), \(\tau_{base}\), \(\gamma\), and \(\lambda\) require tuning (the paper suggests grid search or meta-learning).

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1109/indiscon66021.2025.11254640

Cites

42 of 42 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Verification

Claims the paper makes

22 Valid · 7 Not valid · 3 Not checked · 77 Own results and interpretations

32 more claims · 5 with their cited sources checked

Create a free account to see all 109 claims and every source verdict.

See all claims — free account
  • It may state two separate claims as one. 54
  • Probably not a substantial claim. 28
  • Borderline: one of the tests for a substantial claim is unclear. 1

Create a free account to see these claims.

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

5 of 5

Citations

Cites

Show all 11 references

Research works printed but not linked: 3

  1. WMT 2014 English-German dataset.
  2. Lin, 2004, ROUGE: A package for automatic evaluation of summaries, Text Summarization Branches Out, 74
  3. Huang, Directed acyclic Transformer for non-autoregressive machine translation

Cited by

See all 2

Metadata

Authors
Sen, J., Dasgupta, S., & Waghela, H.
Affiliations
Praxis Business School (IN)
Venue
2025 IEEE 6th India Council International Subsections Conference (INDISCON)
Publisher
IEEE
Published
2025-08-21
Pages
1-6
OpenAlex
W4416924739
Cited by
0
References
11
FWCI
0.00
Type
conference paper
Language
en
Open access
Locations
  • IEEE India Council International Subsections Conference (INDISCON) (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Speech Recognition and Synthesis
Keywords
speculative decoding, large language models, uncertainty estimation, entropy-based uncertainty, autoregressive inference, inference acceleration, machine translation, text summarization

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (gpt-6-sol).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Confidence-Modulated Speculative Decoding for Large Language Models. https://app.cochat.ai/papers/10.1109/indiscon66021.2025.11254640
The paper
Sen, J., Dasgupta, S., & Waghela, H. (2025). Confidence-Modulated Speculative Decoding for Large Language Models. 2025 IEEE 6th India Council International Subsections Conference (INDISCON). https://doi.org/10.1109/indiscon66021.2025.11254640

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.