CoChat
Sign Up
Cited by 839Natural Language Processing TechniquesAdvanced Neural Network ApplicationsTopic Modeling

Sequence-Level Knowledge Distillation

Open access
Kim, Yoon, Rush, Alexander M.·2016·Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing
DOI 10.18653/v1/d16-1139
Open this paper in CoChat — free accountOpen PDF

Abstract

Neural machine translation (NMT) offers a novel alternative formulation of translation that is potentially simpler than statistical approaches.However to reach competitive performance, NMT models need to be exceedingly large.In this paper we consider applying knowledge distillation approaches (Bucila et al., 2006;Hinton et al., 2015) that have proven successful for reducing the size of neural models in other domains to the problem of NMT.We demonstrate that standard knowledge distillation applied to word-level prediction can be effective for NMT, and also introduce two novel sequence-level versions of knowledge distillation that further improve performance, and somewhat surprisingly, seem to eliminate the need for beam search (even when applied on the original teacher model).Our best student model runs 10 times faster than its state-of-the-art teacher with little loss in performance.It is also significantly better than a baseline model trained without knowledge distillation: by 4.2/1.7 BLEU with greedy decoding/beam search.Applying weight pruning on top of knowledge distillation results in a student model that has 13× fewer parameters than the original teacher model, with a decrease of 0.4 BLEU.

AI snapshot

Written from the paper’s full text.

TL;DR

The paper applies knowledge distillation to neural machine translation, showing word-level distillation helps and introducing sequence-level variants (Seq-KD and Seq-Inter) that train a smaller student on teacher beam-search outputs. The best student runs 10× faster than its teacher with little BLEU loss, and pruning on top yields 13× fewer parameters with a 0.4 BLEU drop.

Key findings

  • Word-level knowledge distillation improves over the baseline, but sequence-level knowledge distillation does better on English→German and similarly on Thai→English; combining them gives further gains for the 2×300 and 2×100 models, though not for the 2×500 model. For English→German 2×500, baseline BLEU is 14.7/17.6 (greedy/beam), Seq-KD is 18.9/19.0, and Seq-KD + Seq-Inter is 18.9/19.3, i.e. +4.2/+1.7 BLEU over baseline.
  • Sequence-level distillation reduces the need for beam search: greedy decoding with a fine-tuned teacher (Baseline + Seq-Inter) reaches 19.6 BLEU versus 19.5 for beam search with the original teacher. Seq-KD models also assign much more mass to the approximate mode: 16.9% for the 2×500 English→German Seq-KD model versus 0.9% for the baseline.
  • The student model can run 10× faster with greedy decoding than the teacher with beam search on GPU: 1051.3 vs 101.9 source words/sec.
  • Weight pruning on top of distillation is orthogonal: pruning 80% of weights in the 2×500 Seq-KD + Seq-Inter student gives 13× fewer parameters than the original teacher with only 0.4 BLEU decrease; pruning 90% gives 8m parameters (26× fewer) with a 1.0 BLEU decrease.

Methods

  • Uses an attentional encoder-decoder NMT architecture from Luong et al. (2015). Two settings: English→German WMT 2014 (4m sentences, top 50k vocabulary, teacher 4×1000 LSTM, students 2×300 and 2×500) and Thai→English IWSLT 2015 (90k sentences, 25k vocabulary, teacher 2×500, student 2×100). Evaluation is tokenized BLEU with multi-bleu.perl.
  • Three distillation variants: Word-KD (cross-entropy with teacher word distribution plus NLL; α tested in {0.5, 0.9}, with 0.5 better); Seq-KD (student trained on teacher beam-search output, beam K=5); and Seq-Inter (student trained on the teacher beam hypothesis with highest smoothed sentence BLEU to gold, beam K=35, fine-tuned from a pretrained model with learning rate 0.1; English→German Seq-Inter data generated on ~50% of the training set). Methods can be combined.
  • Weight pruning removes weights with lowest absolute values, then retrains on Seq-KD data with learning rate 0.2 and fine-tunes toward Seq-Inter data with learning rate 0.1. Speed is measured on GPU (GeForce GTX Titan X), CPU, and smartphone (Samsung Galaxy 6).

Limitations

  • Stated: the mode approximation for sequence-level distillation is poor because it approximates an exponentially sized distribution with a single sample; Seq-Inter’s two-sequence interpolation is nonideal because it doubles training data and requires training on both teacher-generated and true sequences conditioned on the same source, which are often different. The paper also calls its illustrative noise function “simple (unrealistic).”
  • Stated: experiments cover only English→German and Thai→English NMT; the 4×1000 model could not be opened on the smartphone; larger beam sizes than K=5 did not improve results; and English→German Seq-Inter data was generated on only ~50% of the training set for efficiency.
  • Evident from the text: no statistical significance tests are reported, hyperparameter searches are narrow (τ ∈ {1, 1.5, 2}; α ∈ {0.5, 0.9}), and compression is compared mainly against pruning rather than other compression families.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.18653/v1/d16-1139

Cites

56 of 56 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

4 of 9 cited sources checked · 3 supported · 0 flagged · accuracy not yet measured

  • Premisecitation

    NMT systems typically need very high capacity to perform well.

    “While both simple and surprisingly accurate, NMT systems typically need to have very high capacity in order to perform well: Sutskever et al. (2014) used a 4-layer LSTM with 1000 hidden units per layer (herein 4×1000) and Zhou et al. (2016) obtained state-of-the-art results on English → French with a 16-layer LSTM with 512 units per layer.”
    Quote in the paperSubstantial claim1 of 2 sources checked
    • cites Sutskever et al. (2014) source
    • cites Zhou et al. (2016) source
  • Findingnone given

    The best student model runs 10 times faster than its state-of-the-art teacher with little loss in performance.

    “Our best student model runs 10 times faster than its state-of-the-art teacher with little loss in performance.”
    Quote in the paperSubstantial claim
  • Findingown data

    Seq-Inter improves upon the original teacher model fine-tuned towards Seq-Inter data (Baseline + Seq-Inter).

    “Sequence-level interpolation (Seq-Inter), in addition to improving models trained via Word-KD and Seq-KD, also improves upon the original teacher model that was trained on the actual data but finetuned towards Seq-Inter data (Baseline + Seq-Inter).”
    Quote in the paperSubstantial claim
  • Findingown data

    Greedy decoding with the fine-tuned Baseline + Seq-Inter model has similar performance (19.6 BLEU) to beam search with the original model (19.5 BLEU).

    “In fact, greedy decoding with this fine-tuned model has similar performance (19.6) as beam search with the original model (19.5), allowing for faster decoding even with an identically-sized model.”
    Quote in the paperSubstantial claim
  • Findingown data

    Seq-KD models assign much higher probability mass to the approximate mode than baseline models trained on original data.

    “Our results suggest that this is indeed the case: the probability mass that Seq-KD models assign to the approximate mode is much higher than is the case for baseline models trained on original data (Table 1: p(t = ŷ)).”
    Quote in the paperSubstantial claim
  • May combine two claims. 15
  • Classified as probably not a substantial claim. 6

  • May combine two claims.

    Premisecitation

    Large networks seem necessary for training but learn redundant representations.

    “Researchers have particularly noted that large networks seem to be necessary for training, but learn redundant representations in the process (Denil et al., 2013).”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites Denil et al., 2013 source

      The Abstract (p. 1) reports significant redundancy in model parameters, and the Introduction (p. 2) explains that individual feature weights can be redundant. The paper does not say that large networks are necessary for training; its finding concerns redundant parameterization, not necessarily redundant representations. (LLM, conf=0.85)

      Experimental — accuracy not yet measured

  • Classified as probably not a substantial claim.

    Premisecitation

    Knowledge distillation approaches have proven successful for reducing the size of neural models in other domains.

    “In this paper we consider applying knowledge distillation approaches (Bucila et al., 2006;Hinton et al., 2015) that have proven successful for reducing the size of neural models in other domains to the problem of NMT.”
    Quote in the paperSubstantial claim1 of 2 sources checked
    • cites Bucila et al., 2006

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites Hinton et al., 2015 source

  • May combine two claims.

    Premisecitation

    Training on the teacher distribution q(y|x;θT) is attractive because it gives more information about other classes and has less variance in gradients.

    “Training on q(y | x; θ T ) is attractive since it gives more information about other classes for a given data point (e.g.similarity between classes) and has less variance in gradients (Hinton et al., 2015).”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites Hinton et al., 2015 source

  • May combine two claims.

    Findingnone given

    Applying weight pruning on top of knowledge distillation results in a student model with 13× fewer parameters than the original teacher model, with a decrease of 0.4 BLEU.

    “Applying weight pruning on top of knowledge distillation results in a student model that has 13× fewer parameters than the original teacher model, with a decrease of 0.4 BLEU.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Combining Seq-KD and Word-KD results in further gains for the 2×300 and 2×100 models, but not for the 2×500 model.

    “Combining them (Seq-KD + Word-KD) results in further gains for the 2 × 300 and 2 × 100 models (although not for the 2 × 500 model)”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingnone given

    A beam of size 5 captures 2.8% of the distribution for German→English and 3.8% for Thai→English.

    “A beam of size 5 captures 2.8% of the distribution for German → English, and 3.8% for Thai → English.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    Regressing the student network's top-most hidden layer to the teacher's top-most hidden layer as a pretraining step gave comparable results to standard knowledge distillation.

    “We found this to give comparable results to standard knowledge distillation and hence did not pursue this further.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    With beam of size 1, q(ŷ | s) accounts for 1.3% of the distribution for German→English and 2.3% for Thai→English on average.

    “In fact, in experiments we find that with beam of size 1, q(ŷ | s) (on average) accounts for 1.3% of the distribution for German → English, and 2.3% for Thai → English (Table 1: p(t = ŷ)).”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingnone given

    The baseline 2 × 500 English → German model has perplexity 8.2 while the corresponding Seq-KD model has perplexity 22.7.

    “The perplexity of the baseline 2 × 500 English → German model is 8.2 while the perplexity of the corresponding Seq-KD model is 22.7”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingown data

    For Seq-KD, beam size K=5 was used and larger beam did not show improvements.

    “We use beam size K = 5 (we did not see improvements with a larger beam).”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingcitation

    Student models can be run efficiently on a standard smartphone.

    “Our student models can even be run efficiently on a standard smartphone.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites 1 https://github.com/harvardnlp/nmt-android

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • May combine two claims.

    Findingown data

    On GPU, the student model with greedy decoding runs about 10 times faster than the teacher model with beam search (1051.3 vs 101.9 source words/sec), with similar performance.

    “We find that we can run the student model 10 times faster with greedy decoding than the teacher model with beam search on GPU (1051.3vs 101.9 words/sec), with similar performance.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingnone given

    On the 2 × 500 English → German model, Seq-KD does significantly better than baseline for both greedy (+4.2 BLEU) and beam search (+1.4 BLEU) decoding.

    “Seq-KD model does significantly better for both greedy (+4.2 BLEU) and beam search (+1.4 BLEU) decoding.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Pruning 90% of the weights results in a decrease of 1.0 BLEU and a model with 8m parameters, 26× fewer than the original teacher model.

    “While pruning 90% of the weights results in a more appreciable decrease of 1.0 BLEU, the model is drastically smaller with 8m parameters, which is 26× fewer than the original teacher model.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationown data

    Word-KD and Seq-KD provide orthogonal means of transferring knowledge from the teacher to the student: Word-KD transfers local word-level knowledge while Seq-KD transfers global sequence-level knowledge.

    “indicating that these methods provide orthogonal means of transferring knowledge from the teacher to the student: Word-KD is transferring knowledge at the the local (i.e.word) level while Seq-KD is transferring knowledge at the global (i.e.sequence) level.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Premisecitation

    Weight pruning for NMT was recently investigated by See et al. (2016), who found that up to 80-90% of the parameters in a large NMT model can be pruned with little loss in performance.

    “Weight pruning for NMT was recently investigated by See et al. (2016), who found that up to 80 -90% of the parameters in a large NMT model can be pruned with little loss in performance.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites See et al. (2016)

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • May combine two claims.

    Interpretationreasoning

    Sequence-level knowledge distillation allows for greedy decoding, which is significant for running NMT systems across various devices.

    “Therefore, the fact that sequencelevel knowledge distillation allows for greedy decoding is significant, with practical implications for running NMT systems across various devices.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: Run-time complexity for beam search grows linearly with beam size.

  • May combine two claims.

    Interpretationreasoning

    The above methods are complementary and can be combined with each other.

    “The above methods are complementary and can be combined with each other.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationreasoning

    The size of word embeddings has little impact on run-time because the word embedding layer is a simple lookup table that only affects the first layer of the model.

    “The size of word embeddings have little impact on run-time as the word embedding layer is a simple lookup table that only affects the first layer of the model.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable).

    Interpretationreasoning

    The sequence-level objective is worthwhile because it gives the teacher the chance to assign probabilities to complete sequences and transfer a broader range of knowledge.

    “Despite its intractability, we posit that this sequence-level objective is worthwhile.It gives the teacher the chance to assign probabilities to complete sequences and therefore transfer a broader range of knowledge.”
    Quote in the paperNot a substantial claim
20 of 20

  • The authors state this as an assumption.

    The teacher has previously been trained, and we are estimating parameters for the student.

    “We generally assume that the teacher has previously been trained, and that we are estimating parameters for the student.”

  • The authors state this as an assumption.

    The teacher distribution parameters remain fixed during student training.

    “where θ T parameterizes the teacher distribution and remains fixed.”

  • The paper relies on this without support.

    Run-time complexity for beam search grows linearly with beam size.

    “Run-time complexity for beam search grows linearly with beam size.”

  • The paper relies on this without support.

    Combining pruning and knowledge distillation has not been investigated before.

    “To our knowledge combining pruning and knowledge distillation has not been investigated before.”

  • The paper relies on this without support.

    The translation domain has generally required the largest capacity deep learning models.

    “We have chosen to focus on translation as this domain has generally required the largest capacity deep learning models”
5 of 5

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Extracting reasoning yields a substantially more effective form of capability stealing than output-only distillation because a reasoning trace reveals the intermediate solution trajectory and supplies a far denser supervision signal.

    “Extracting reasoning matters not because output-only distillation is ineffective, but because reasoning yields a substantially more effective form of capability stealing.”

    From Stealing Reasoning Traces from Proprietary LLM APIs (2026) · cites this paper as Kim and Rush, 2016

    The Results and Discussion section reports that word-level knowledge distillation improves on the baseline, supporting the sentence’s premise that distillation can be effective. The paper studies teacher-generated translations, however; it does not examine reasoning extraction or show that reasoning traces make capability stealing more effective than output-only distillation. (LLM, conf=0.83)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

  • Word-embedding parameter share does not match the stated counts

    Minor

    The stated 50m word-embedding parameters out of 84m total amount to about 59.5%, not approximately 63%.

    Reported 63 · recomputed 59.52381

  • Incorrect BLEU improvement for 2 × 500 Baseline + Seq-Inter

    Minor

    The English→German 2 × 500 Baseline + Seq-Inter greedy BLEU is 18.5, compared with 14.7 for the baseline. The improvement is 3.8 BLEU, not the reported 3.6.

    Reported 3.6 · recomputed 3.8

2 of 2

Cites

Flagged first, then by citations

Research works printed but not linked: 4

  1. Cristian Bucila, Rich Caruana, and Alexandru Niculescu-Mizil. 2006. Model Compres- sion. In Proceedings of KDD.
  2. and Blunsom2013] Nal Kalchbrenner and Phil Blunsom. 2013. Recurrent Continuous Transla- tion Models. In Proceedings of EMNLP.
  3. Jiwei Li, Michael Galley, Chris Brockett, Jianfeg Gao, and Bill Dolan. 2016. A Diversity- Promoting Objective Function for Neural Conversa- tional Models. In Proceedings of NAACL 2016.
  4. Mariet and Sra2016] Zelda Mariet and Suvrit Sra. 2016. Diversity Networks. In Proceedings of ICLR.

Cited by

Sorted by citations

Related

Authors
Kim, Y., & Rush, A. M.
Affiliations
Harvard University (US)
ORCID
Venue
Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing
Publisher
Association for Computational Linguistics
Published
2016
Pages
1317-1327
DOI
10.18653/v1/d16-1139
OpenAlex
W2963736842
Semantic Scholar
57a10537978600fd33dcdd48922c791609a4851a
Cited by
836
References
53
Citations by year
2026: 36 · 2025: 59 · 2024: 52 · 2023: 137 · 2022: 115 · 2021: 157 · 2020: 143 · 2019: 75 · 2018: 42 · 2017: 16 · 2016: 4
FWCI
24.59
Type
conference paper
Language
en
Open access
Licence
cc-by
Locations
  • Conference on Empirical Methods in Natural Language Processing (EMNLP) (publishedVersion, cc-by)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
Keywords
knowledge distillation, neural machine translation, model compression, weight pruning, teacher-student models, parameter reduction

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-23.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-23.

Cited sources are checked by gpt-6-sol.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review