The DOI matches its Crossref record; 57 of the 57 references it cites were found in a registry or an index; 4 of the 9 sources its claims cite were checked for support.
Sequence-Level Knowledge Distillation
Open accessAbstract
Neural machine translation (NMT) offers a novel alternative formulation of translation that is potentially simpler than statistical approaches.However to reach competitive performance, NMT models need to be exceedingly large.In this paper we consider applying knowledge distillation approaches (Bucila et al., 2006;Hinton et al., 2015) that have proven successful for reducing the size of neural models in other domains to the problem of NMT.We demonstrate that standard knowledge distillation applied to word-level prediction can be effective for NMT, and also introduce two novel sequence-level versions of knowledge distillation that further improve performance, and somewhat surprisingly, seem to eliminate the need for beam search (even when applied on the original teacher model).Our best student model runs 10 times faster than its state-of-the-art teacher with little loss in performance.It is also significantly better than a baseline model trained without knowledge distillation: by 4.2/1.7 BLEU with greedy decoding/beam search.Applying weight pruning on top of knowledge distillation results in a student model that has 13× fewer parameters than the original teacher model, with a decrease of 0.4 BLEU.
Overview
AI snapshot
Written from the paper’s full text.
TL;DR
The paper applies knowledge distillation to neural machine translation, showing word-level distillation helps and introducing sequence-level variants (Seq-KD and Seq-Inter) that train a smaller student on teacher beam-search outputs. The best student runs 10× faster than its teacher with little BLEU loss, and pruning on top yields 13× fewer parameters with a 0.4 BLEU drop.
Key findings
- Word-level knowledge distillation improves over the baseline, but sequence-level knowledge distillation does better on English→German and similarly on Thai→English; combining them gives further gains for the 2×300 and 2×100 models, though not for the 2×500 model. For English→German 2×500, baseline BLEU is 14.7/17.6 (greedy/beam), Seq-KD is 18.9/19.0, and Seq-KD + Seq-Inter is 18.9/19.3, i.e. +4.2/+1.7 BLEU over baseline.
- Sequence-level distillation reduces the need for beam search: greedy decoding with a fine-tuned teacher (Baseline + Seq-Inter) reaches 19.6 BLEU versus 19.5 for beam search with the original teacher. Seq-KD models also assign much more mass to the approximate mode: 16.9% for the 2×500 English→German Seq-KD model versus 0.9% for the baseline.
- The student model can run 10× faster with greedy decoding than the teacher with beam search on GPU: 1051.3 vs 101.9 source words/sec.
- Weight pruning on top of distillation is orthogonal: pruning 80% of weights in the 2×500 Seq-KD + Seq-Inter student gives 13× fewer parameters than the original teacher with only 0.4 BLEU decrease; pruning 90% gives 8m parameters (26× fewer) with a 1.0 BLEU decrease.
Methods
- Uses an attentional encoder-decoder NMT architecture from Luong et al. (2015). Two settings: English→German WMT 2014 (4m sentences, top 50k vocabulary, teacher 4×1000 LSTM, students 2×300 and 2×500) and Thai→English IWSLT 2015 (90k sentences, 25k vocabulary, teacher 2×500, student 2×100). Evaluation is tokenized BLEU with multi-bleu.perl.
- Three distillation variants: Word-KD (cross-entropy with teacher word distribution plus NLL; α tested in {0.5, 0.9}, with 0.5 better); Seq-KD (student trained on teacher beam-search output, beam K=5); and Seq-Inter (student trained on the teacher beam hypothesis with highest smoothed sentence BLEU to gold, beam K=35, fine-tuned from a pretrained model with learning rate 0.1; English→German Seq-Inter data generated on ~50% of the training set). Methods can be combined.
- Weight pruning removes weights with lowest absolute values, then retrains on Seq-KD data with learning rate 0.2 and fine-tunes toward Seq-Inter data with learning rate 0.1. Speed is measured on GPU (GeForce GTX Titan X), CPU, and smartphone (Samsung Galaxy 6).
Limitations
- Stated: the mode approximation for sequence-level distillation is poor because it approximates an exponentially sized distribution with a single sample; Seq-Inter’s two-sequence interpolation is nonideal because it doubles training data and requires training on both teacher-generated and true sequences conditioned on the same source, which are often different. The paper also calls its illustrative noise function “simple (unrealistic).”
- Stated: experiments cover only English→German and Thai→English NMT; the 4×1000 model could not be opened on the smartphone; larger beam sizes than K=5 did not improve results; and English→German Seq-Inter data was generated on only ~50% of the training set for efficiency.
- Evident from the text: no statistical significance tests are reported, hyperparameter searches are narrow (τ ∈ {1, 1.5, 2}; α ∈ {0.5, 0.9}), and compression is compared mainly against pruning rather than other compression families.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
26 Own result · 13 Own interpretation · 9 No basis given · 3 Sources not found · 2 Partly supported · 1 Supported · 1 Supported so far · accuracy not yet measured
The sources with a verdict support this claim; the rest weren’t checked yet, have no full text we could read, or their verdicts aren’t rated yet.
In the paper
“While both simple and surprisingly accurate, NMT systems typically need to have very high capacity in order to perform well: Sutskever et al. (2014) used a 4-layer LSTM with 1000 hidden units per layer (herein 4×1000) and Zhou et al. (2016) obtained state-of-the-art results on English → French with a 16-layer LSTM with 512 units per layer.”
§ Sequence-Level Knowledge Distillation > 1 Introduction
What the sources say
- Not checked — no full text of the sourcecites Sutskever et al. (2014) source
- Supportedcites Zhou et al. (2016) source
The Model settings section states that every LSTM layer has 512 memory cells. The Introduction and Conclusion report a depth-16 model that achieved state-of-the-art results on WMT’14 English-to-French, supporting the specific example attributed to Zhou et al. (LLM, conf=0.93)
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Our best student model runs 10 times faster than its state-of-the-art teacher with little loss in performance.”
§ Sequence-Level Knowledge Distillation > Abstract
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Sequence-level interpolation (Seq-Inter), in addition to improving models trained via Word-KD and Seq-KD, also improves upon the original teacher model that was trained on the actual data but finetuned towards Seq-Inter data (Baseline + Seq-Inter).”
§ Sequence-Level Knowledge Distillation > 5 Results and Discussion
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In fact, greedy decoding with this fine-tuned model has similar performance (19.6) as beam search with the original model (19.5), allowing for faster decoding even with an identically-sized model.”
§ Sequence-Level Knowledge Distillation > 5 Results and Discussion
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Our results suggest that this is indeed the case: the probability mass that Seq-KD models assign to the approximate mode is much higher than is the case for baseline models trained on original data (Table 1: p(t = ŷ)).”
§ Sequence-Level Knowledge Distillation > 5 Results and Discussion
hedgedcorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Sequence-level interpolation (Seq-Inter), in addition to improving models trained via Word-KD and Seq-KD, also improves upon the original teacher model that was trained on the actual data but finetuned towards Seq-Inter data (Baseline + Seq-Inter).”
§ Sequence-Level Knowledge Distillation > 5 Results and Discussion
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We find that while word-level knowledge distillation (Word-KD) does improve upon the baseline, sequence-level knowledge distillation (Seq-KD) does better on English → German and performs similarly on Thai → English.”
§ Sequence-Level Knowledge Distillation > 5 Results and Discussion
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We find that while word-level knowledge distillation (Word-KD) does improve upon the baseline, sequence-level knowledge distillation (Seq-KD) does better on English → German and performs similarly on Thai → English.”
§ Sequence-Level Knowledge Distillation > 5 Results and Discussion
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Seq-Inter offers a compromise between the two, with the greedily-decoded sequence accounting for 7.6% of the distribution.”
§ Sequence-Level Knowledge Distillation > 5 Results and Discussion
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We find that while word-level knowledge distillation (Word-KD) does improve upon the baseline, sequence-level knowledge distillation (Seq-KD) does better on English → German and performs similarly on Thai → English.”
§ Sequence-Level Knowledge Distillation > 5 Results and Discussion
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- It may state two separate claims as one. 15
- Probably not a substantial claim. 6
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Researchers have particularly noted that large networks seem to be necessary for training, but learn redundant representations in the process (Denil et al., 2013).”
§ Sequence-Level Knowledge Distillation > 1 Introduction
What the sources say
- Partly supportedcites Denil et al., 2013 sourceExperimental — accuracy not yet measured
The Abstract (p. 1) reports significant redundancy in model parameters, and the Introduction (p. 2) explains that individual feature weights can be redundant. The paper does not say that large networks are necessary for training; its finding concerns redundant parameterization, not necessarily redundant representations. (LLM, conf=0.85)
hedgeddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Partly supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“In this paper we consider applying knowledge distillation approaches (Bucila et al., 2006;Hinton et al., 2015) that have proven successful for reducing the size of neural models in other domains to the problem of NMT.”
§ Sequence-Level Knowledge Distillation > Abstract
What the sources say
- Not matchedcites Bucila et al., 2006
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Supportedcites Hinton et al., 2015 source
The Discussion (p. 8) says distillation worked well for transferring knowledge from an ensemble or large model into a smaller model, including in MNIST and acoustic-model experiments. These are successful applications outside NMT, as the citing sentence claims. (LLM, conf=0.94)
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
- Not matched
Every source the paper cites for this claim was checked and supports it.
In the paper
“Training on q(y | x; θ T ) is attractive since it gives more information about other classes for a given data point (e.g.similarity between classes) and has less variance in gradients (Hinton et al., 2015).”
§ Sequence-Level Knowledge Distillation > 2.2 Knowledge Distillation
What the sources say
- Supportedcites Hinton et al., 2015 source
The Introduction (p. 2) says that soft targets from a cumbersome model provide more information per training case and less variance in gradients when they have high entropy. It also explains that probabilities assigned to other classes convey a similarity structure. (LLM, conf=0.93)
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Supported
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Applying weight pruning on top of knowledge distillation results in a student model that has 13× fewer parameters than the original teacher model, with a decrease of 0.4 BLEU.”
§ Sequence-Level Knowledge Distillation > Abstract
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Combining them (Seq-KD + Word-KD) results in further gains for the 2 × 300 and 2 × 100 models (although not for the 2 × 500 model)”
§ Sequence-Level Knowledge Distillation > 5 Results and Discussion
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“A beam of size 5 captures 2.8% of the distribution for German → English, and 3.8% for Thai → English.”
§ Sequence-Level Knowledge Distillation > 3.2 Sequence-Level Knowledge Distillation
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We found this to give comparable results to standard knowledge distillation and hence did not pursue this further.”
§ Sequence-Level Knowledge Distillation > 5.3 Further Observations
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In fact, in experiments we find that with beam of size 1, q(ŷ | s) (on average) accounts for 1.3% of the distribution for German → English, and 2.3% for Thai → English (Table 1: p(t = ŷ)).”
§ Sequence-Level Knowledge Distillation > 3.2 Sequence-Level Knowledge Distillation
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“The perplexity of the baseline 2 × 500 English → German model is 8.2 while the perplexity of the corresponding Seq-KD model is 22.7”
§ Sequence-Level Knowledge Distillation > 7 Conclusion
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We use beam size K = 5 (we did not see improvements with a larger beam).”
§ Sequence-Level Knowledge Distillation > Sequence-Level Knowledge Distillation (Seq-KD)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
In the paper
“Our student models can even be run efficiently on a standard smartphone.”
§ Sequence-Level Knowledge Distillation > 1 Introduction
What the sources say
- Not matchedcites 1 https://github.com/harvardnlp/nmt-android
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Not matched
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We find that we can run the student model 10 times faster with greedy decoding than the teacher model with beam search on GPU (1051.3vs 101.9 words/sec), with similar performance.”
§ Sequence-Level Knowledge Distillation > 5.1 Decoding Speed
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Seq-KD model does significantly better for both greedy (+4.2 BLEU) and beam search (+1.4 BLEU) decoding.”
§ Sequence-Level Knowledge Distillation > 7 Conclusion
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“While pruning 90% of the weights results in a more appreciable decrease of 1.0 BLEU, the model is drastically smaller with 8m parameters, which is 26× fewer than the original teacher model.”
§ Sequence-Level Knowledge Distillation > 5.2 Weight Pruning
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“indicating that these methods provide orthogonal means of transferring knowledge from the teacher to the student: Word-KD is transferring knowledge at the the local (i.e.word) level while Seq-KD is transferring knowledge at the global (i.e.sequence) level.”
§ Sequence-Level Knowledge Distillation > 5 Results and Discussion
hedgeddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“Weight pruning for NMT was recently investigated by See et al. (2016), who found that up to 80 -90% of the parameters in a large NMT model can be pruned with little loss in performance.”
§ Sequence-Level Knowledge Distillation > 5.2 Weight Pruning
What the sources say
- Not matchedcites See et al. (2016)
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
- Not matched
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Therefore, the fact that sequencelevel knowledge distillation allows for greedy decoding is significant, with practical implications for running NMT systems across various devices.”
§ Sequence-Level Knowledge Distillation > 5.1 Decoding Speed
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Rests on an assumption: Run-time complexity for beam search grows linearly with beam size.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The above methods are complementary and can be combined with each other.”
§ Sequence-Level Knowledge Distillation > Sequence-Level Interpolation (Seq-Inter)
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The size of word embeddings have little impact on run-time as the word embedding layer is a simple lookup table that only affects the first layer of the model.”
§ Sequence-Level Knowledge Distillation > 5.2 Weight Pruning
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: unclear — It is unclear why the paper says it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Despite its intractability, we posit that this sequence-level objective is worthwhile.It gives the teacher the chance to assign probabilities to complete sequences and therefore transfer a broader range of knowledge.”
§ Sequence-Level Knowledge Distillation > 3.2 Sequence-Level Knowledge Distillation
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence.
The authors state this as an assumption.
The teacher has previously been trained, and we are estimating parameters for the student.
“We generally assume that the teacher has previously been trained, and that we are estimating parameters for the student.”
The authors state this as an assumption.
The teacher distribution parameters remain fixed during student training.
“where θ T parameterizes the teacher distribution and remains fixed.”
The paper relies on this without support.
Run-time complexity for beam search grows linearly with beam size.
“Run-time complexity for beam search grows linearly with beam size.”
The paper relies on this without support.
Combining pruning and knowledge distillation has not been investigated before.
“To our knowledge combining pruning and knowledge distillation has not been investigated before.”
The paper relies on this without support.
The translation domain has generally required the largest capacity deep learning models.
“We have chosen to focus on translation as this domain has generally required the largest capacity deep learning models”
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Extracting reasoning matters not because output-only distillation is ineffective, but because reasoning yields a substantially more effective form of capability stealing.”
From Stealing Reasoning Traces from Proprietary LLM APIs (2026) · cites this paper as Kim and Rush, 2016
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
“on the 2 × 500 English → German model the word embeddings account for approximately 63% (50m out of 84m) of the parameters.”
§ Sequence-Level Knowledge Distillation > 5.2 Weight Pruning
Reported 63 · recomputed 59.52381
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
“Student Baseline 2 × 500 (Params: 84m) Word-KD Seq-KD Baseline + Seq-Inter Word-KD + Seq-Inter Seq-KD + Seq-Inter Seq-KD + Word-KD Seq-KD + Seq-Inter + Word-KD | 14.7 15.4 18.9 18.5 18.3 18.9 18.7 18.8 | -+0.7 +4.2 +3.6”
§ Sequence-Level Knowledge Distillation > 7 Conclusion
Reported 3.6 · recomputed 3.8
Citations
Cites
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318
Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation
Cho, K., van Merrienboer, B., Gulcehre, C., Bahdanau, D., Bougares, F., Schwenk, H., & Bengio, Y. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1179 · Cited by 25,545
BLEU: a method for automatic evaluation of machine translation
Papineni, K., Roukos, S., Ward, T., & Zhu, W.-J. (2001) · Proceedings of the 40th Annual Meeting on Association for Computational Linguistics - ACL '02 · doi:10.3115/1073083.1073135 · Cited by 21,344
Sequence to Sequence Learning with Neural Networks
Sutskever, I., Vinyals, O., & Le, Q. V. (2014) · arXiv · doi:10.48550/arxiv.1409.3215 · Cited by 17,766
Neural Machine Translation by Jointly Learning to Align and Translate
Bahdanau, D., Cho, K., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1409.0473 · Cited by 16,213
Distilling the Knowledge in a Neural Network
Hinton, G., Vinyals, O., & Dean, J. (2015) · arXiv · doi:10.48550/arxiv.1503.02531 · Cited by 14,204
Effective Approaches to Attention-based Neural Machine Translation
Luong, T., Pham, H., & Manning, C. D. (2015) · Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d15-1166 · Cited by 8,382
Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
Xu, K., Ba, J., Kiros, R., Cho, K., Courville, A., Salakhutdinov, R., Zemel, R., & Bengio, Y. (2015) · arXiv · doi:10.48550/arxiv.1502.03044 · Cited by 8,248
Show and tell: A neural image caption generator
Vinyals, O., Toshev, A., Bengio, S., & Erhan, D. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298935 · Cited by 6,318
Han, S., Mao, H., & Dally, W. J. (2015) · arXiv · doi:10.48550/arxiv.1510.00149 · Cited by 4,662
Research works printed but not linked: 4
- Cristian Bucila, Rich Caruana, and Alexandru Niculescu-Mizil. 2006. Model Compres- sion. In Proceedings of KDD.
- and Blunsom2013] Nal Kalchbrenner and Phil Blunsom. 2013. Recurrent Continuous Transla- tion Models. In Proceedings of EMNLP.
- Jiwei Li, Michael Galley, Chris Brockett, Jianfeg Gao, and Bill Dolan. 2016. A Diversity- Promoting Objective Function for Neural Conversa- tional Models. In Proceedings of NAACL 2016.
- Mariet and Sra2016] Zelda Mariet and Suvrit Sra. 2016. Diversity Networks. In Proceedings of ICLR.
Cited by
Sorted by citations
Items 201–300 of 841
Accelerating Transformer Inference for Translation via Parallel Decoding
Santilli, A., Severino, S., Postolache, E., Maiorca, V., Mancusi, M., Marin, R., & Rodola, E. (2023) · Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2023.acl-long.689 · Cited by 29
“NAT methods patch up this limitation with sequence-level KD which has the advantage of "not requiring any beam search at test-time" (Kim and Rush, 2016) thanks to learning and distillation from large models.”
BackgroundMethodGreedy Search with Probabilistic N-gram Matching for Neural Machine Translation
Shao, C., Chen, X., & Feng, Y. (2018) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d18-1510 · Cited by 29
End-to-End Transition-Based Online Dialogue Disentanglement
Liu, H., Shi, Z., Gu, J.-C., Liu, Q., Wei, S., & Zhu, X. (2020) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2020/535 · Cited by 29
Adaptive Adapters: An Efficient Way to Incorporate BERT Into Neural Machine Translation
Guo, J., Zhang, Z., Xu, L., Chen, B., & Chen, E. (2021) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2021.3076863 · Cited by 34
Self-Training Sampling with Monolingual Data Uncertainty for Neural Machine Translation
Jiao, W., Wang, X., Tu, Z., Shi, S., Lyu, M. R.-T., & King, I. (2021) · doi:10.18653/v1/2021.acl-long.221 · Cited by 34
Multi-view Teacher–Student Network
Tian, Y., Sun, S., & Tang, J. (2021) · Neural Networks · doi:10.1016/j.neunet.2021.11.002 · Cited by 29
Investigation of Sequence-level Knowledge Distillation Methods for CTC Acoustic Models
Takashima, R., Li, S., & Kawai, H. (2019) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2019.8682671 · Cited by 27
Towards Data Distillation for End-to-end Spoken Conversational Question AnsweringPreprint
You, C., Chen, N., Liu, F., Yang, D., & Zou, Y. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.08923 · Cited by 41
Knowledge Distillation: A Method for Making Neural Machine Translation More Efficient
Jooste, W., Haque, R., & Way, A. (2022) · Information · doi:10.3390/info13020088 · Cited by 36
Mitigating exposure bias in large language model distillation: an imitation learning approach
Pozzi, A., Incremona, A., Tessera, D., & Toti, D. (2025) · Neural Computing and Applications · doi:10.1007/s00521-025-11162-0 · Cited by 28
Results of the WMT17 Neural MT Training Task
Bojar, O., Helcl, J., Kocmi, T., Libovický, J., & Musil, T. (2017) · Conference on Machine Translation (WMT) · doi:10.18653/v1/w17-4757 · Cited by 28
Frustratingly Easy Model Ensemble for Abstractive Summarization
Kobayashi, H. (2018) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d18-1449 · Cited by 28
Partially Non-Autoregressive Image Captioning
Fei, Z. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i2.16219 · Cited by 28
When deep learning meets watermarking: A survey of application, attacks and defenses
Chen, H., Liu, C., Zhu, T., & Zhou, W. (2024) · Computer Standards & Interfaces · doi:10.1016/j.csi.2023.103830 · Cited by 28
Improving Offline Handwritten Chinese Character Recognition by Iterative Refinement
Yang, X., He, D., Zhou, Z., Kifer, D., & Giles, C. L. (2017) · IAPR International Conference on Document Analysis and Recognition (ICDAR) · doi:10.1109/icdar.2017.11 · Cited by 28
Mun'im, R. M., Inoue, N., & Shinoda, K. (2019) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2019.8683171 · Cited by 27
ASR is All You Need: Cross-Modal Distillation for Lip Reading
Afouras, T., Chung, J. S., & Zisserman, A. (2020) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp40776.2020.9054253 · Cited by 25
An Efficient and Lightweight Approach for Intrusion Detection based on Knowledge Distillation
Zhao, R., Chen, Y., Wang, Y., Shi, Y., & Xue, Z. (2021) · International Conference on Communications · doi:10.1109/icc42927.2021.9500574 · Cited by 27
Improving Neural Machine Translation by Bidirectional Training
Ding, L., Wu, D., & Tao, D. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.263 · Cited by 31
Improving Cross-Lingual Transfer Learning for End-to-End Speech Recognition with Speech Translation
Wang, C., Pino, J., & Gu, J. (2020) · Interspeech (USB) · doi:10.21437/interspeech.2020-2955 · Cited by 36
Proceedings of the First Workshop on Neural Machine Translation
(2017) · doi:10.18653/v1/w17-32 · Cited by 30
Survey of Deep Learning Accelerators for Edge and Emerging Computing
Alam, S., Yakopcic, C., Wu, Q., Barnell, M., Khan, S., & Taha, T. M. (2024) · Electronics · doi:10.3390/electronics13152988 · Cited by 38
Cross-modality Data Augmentation for End-to-End Sign Language Translation
Ye, J., Jiao, W., Wang, X., et al. (2023) · doi:10.18653/v1/2023.findings-emnlp.904 · Cited by 26
A Stable and Effective Learning Strategy for Trainable Greedy Decoding
Chen, Y., Li, V. O. K., Cho, K., & Bowman, S. R. (2018) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d18-1035 · Cited by 29
schuBERT: Optimizing Elements of BERT
Khetan, A., & Karnin, Z. S. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.250 · Cited by 28
An Efficient Sharing Grouped Convolution via Bayesian Learning
Chen, T., Duan, B., Sun, Q. L., Zhang, M., Li, G., Geng, H., Zhang, Q., & Yu, B. (2021) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2021.3084900 · Cited by 26
O2NA: An Object-Oriented Non-Autoregressive Approach for Controllable Video Captioning
Liu, F., Ren, X., Wu, X., Yang, B., Ge, S., & Sun, X. (2021) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2021.findings-acl.24 · Cited by 29
End-to-end Spoken Conversational Question Answering: Task, Dataset and Model
You, C., Chen, N., Liu, F., Ge, S., Wu, X., & Zou, Y. (2022) · Findings of the Association for Computational Linguistics: NAACL · doi:10.18653/v1/2022.findings-naacl.91 · Cited by 32
From Algorithm to Hardware: A Survey on Efficient and Safe Deployment of Deep Neural Networks
Geng, X., Wang, Z., Chen, C., Xu, Q., Xu, K., Jin, C., Gupta, M., Yang, X., Chen, Z., Sabry Aly, M. M., et al. (2025) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2024.3394494 · Cited by 26
Sub-Band Knowledge Distillation Framework for Speech Enhancement
Hao, X., Wen, S., Su, X., et al. (2020) · Interspeech (USB) · doi:10.21437/interspeech.2020-1539 · Cited by 25
Dialogue Distillation: Open-Domain Dialogue Augmentation Using Unpaired Data
Zhang, R., Zheng, Y., Shao, J., Mao, X., Xi, Y., & Huang, M. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.277 · Cited by 23
Distilling Linguistic Context for Language Model Compression
Park, G., Kim, G., & Yang, E. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.30 · Cited by 35
Mutual Learning of Joint and Separate Domain Alignments for Multi-Source Domain Adaptation
Xu, Y., Kan, M., Shan, S., & Chen, X. (2022) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv51458.2022.00172 · Cited by 26
Refined U-net: A new semantic technique on hand segmentation
Tsai, T., & Huang, S. (2022) · Neurocomputing · doi:10.1016/j.neucom.2022.04.079 · Cited by 25
Stage-Wise Magnitude-Based Pruning for Recurrent Neural Networks
Li, G., Yang, P., Qian, C., Hong, R., & Tang, K. (2022) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2022.3184730 · Cited by 25
Speeding Up Neural Machine Translation Decoding by Shrinking Run-time Vocabulary
Shi, X., & Knight, K. K. (2017) · Annual Meeting of the Association for Computational Linguistics (ACL) · doi:10.18653/v1/p17-2091 · Cited by 25
Go From the General to the Particular: Multi-Domain Translation with Domain Transformation Networks
Wang, Y., Wang, L., Shi, S., Li, V. O. K., & Tu, Z. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i05.6461 · Cited by 27
Sequence Generation: From Both Sides to the Middle
Zhou, L., Zhang, J., Zong, C., & Yu, H. (2019) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2019/760 · Cited by 24
Li, G., Liu, L., Huang, G., Zhu, C., & Zhao, T. (2019) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d19-1570 · Cited by 24
Diversifying Dialog Generation via Adaptive Label Smoothing
Wang, Y., Zheng, Y., Jiang, Y., & Huang, M. (2021) · doi:10.18653/v1/2021.acl-long.272 · Cited by 26
Improving Stance Detection with Multi-Dataset Learning and Knowledge Distillation
Li, Y., Zhao, C., & Caragea, C. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.511 · Cited by 24
Agriculture Assistant Chatbot Using Artificial Neural Network
Chandolikar, N., Dale, C., Koli, T., Singh, M., & Narkhede, T. (2022) · International Conference on Advanced Computing Technologies and Applications (ICACTA) · doi:10.1109/icacta54488.2022.9753433 · Cited by 30
CTC Alignments Improve Autoregressive Translation
Yan, B., Dalmia, S., Higuchi, Y., Neubig, G., Metze, F., Black, A. W., & Watanabe, S. (2023) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2023.eacl-main.119 · Cited by 34
Machine Translation in the Era of Large Language Models:A Survey of Historical and Emerging Problems
Ataman, D., Birch, A., Habash, N. Y., Federico, M., Koehn, P., & Cho, K. (2025) · Information · doi:10.3390/info16090723 · Cited by 49
Sequence-Level Training for Non-Autoregressive Neural Machine Translation
Shao, C.-Z., Feng, Y., Zhang, J.-C., Meng, F.-D., & Zhou, J. (2021) · International Conference on Computational Logic · doi:10.1162/coli_a_00421 · Cited by 32
DQ-BART: Efficient Sequence-to-Sequence Model via Joint Distillation and Quantization
Li, Z., Wang, Z., Tan, M., Nallapati, R. M., Bhatia, P., Arnold, A., Xiang, B., & Roth, D. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-short.22 · Cited by 23
Multimodal Reaction: Information Modulation for Cross-Modal Representation Learning
Zeng, Y., Mai, S., Yan, W., & Hu, H. (2023) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2023.3293335 · Cited by 25
Multi-Fidelity Neural Architecture Search With Knowledge Distillation
Trofimov, I., Klyuchnikov, N., Salnikov, M., Filippov, A., & Burnaev, E. V. (2023) · IEEE Access · doi:10.1109/access.2023.3234810 · Cited by 29
Heterogeneous Federated Learning Framework for IIoT Based on Selective Knowledge Distillation
Guo, S., Chen, H., Liu, Y., Yang, C., Li, Z., & Jin, C. (2024) · IEEE Transactions on Industrial Informatics · doi:10.1109/tii.2024.3452229 · Cited by 22
Transferable polychromatic optical encoder for neural networks
Choi, M., Xiang, J., Wirth-Singh, A., Baek, S., Shlizerman, E., & Majumdar, A. (2025) · Nature Communications · doi:10.1038/s41467-025-61338-4 · Cited by 23
ORTHROS: non-autoregressive end-to-end speech translation With dual-decoder
Inaguma, H., Higuchi, Y., Duh, K., Kawahara, T., & Watanabe, S. (2021) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp39728.2021.9415093 · Cited by 22
Multi-Teacher Distillation With Single Model for Neural Machine Translation
Liang, X., Wu, L., Li, J., Qin, T., Zhang, M., & Liu, T. (2022) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2022.3153264 · Cited by 25
Image Captioning With Controllable and Adaptive Length Levels
Ding, N., Deng, C., Tan, M., Du, Q., Ge, Z., & Wu, Q. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3328298 · Cited by 23
Computer Vision Model Compression Techniques for Embedded Systems:A Survey
Lopes, A., dos Santos, F. P., de Oliveira, D. C., Schiezaro, M., & Pedrini, H. (2024) · Computers & Graphics · doi:10.1016/j.cag.2024.104015 · Cited by 25
Joint Student-Teacher Learning for Audio-Visual Scene-Aware Dialog
Hori, C., Cherian, A., Marks, T. K., & Hori, T. (2019) · Interspeech (USB) · doi:10.21437/interspeech.2019-3143 · Cited by 22
Cloze Distillation: Improving Neural Language Models with Human Next-Word Prediction
Eisape, T., Zaslavsky, N., & Lévy, R. (2020) · Conference on Computational Natural Language Learning · doi:10.18653/v1/2020.conll-1.49 · Cited by 26
Ranking Distillation: Learning Compact Ranking Models With High Performance for Recommender SystemPreprint
Tang, J., & Wang, K. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1809.07428 · Cited by 22
A Comprehensive Survey of Multilingual Neural Machine TranslationPreprint
Dabre, R., Chu, C.-H., & Kunchukuttan, A. (2020) · arXiv.org · doi:10.48550/arxiv.2001.01115 · Cited by 36
Cross-Modal Knowledge Distillation With Multi-Stage Adaptive Feature Fusion for Speech Separation
Fan, C., Xiang, W., Tao, J., Yi, J., & Lv, Z. (2025) · IEEE Transactions on Audio Speech and Language Processing · doi:10.1109/taslpro.2025.3533359 · Cited by 21
Multi-Source Neural Machine Translation With Missing Data
Nishimura, Y., Sudoh, K., Neubig, G., et al. (2019) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2019.2959224 · Cited by 21
Sequence-Level Consistency Training for Semi-Supervised End-to-End Automatic Speech Recognition
Masumura, R., Ihori, M., Takashima, A., et al. (2020) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp40776.2020.9052940 · Cited by 21
Iterative Refinement in the Continuous Space for Non-Autoregressive Neural Machine Translation
Lee, J., Shu, R., & Cho, K. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.73 · Cited by 22
Pruning-then-Expanding Model for Domain Adaptation of Neural Machine Translation
Gu, S., Feng, Y., & Xie, W. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.308 · Cited by 23
Unsupervised Paraphrasing with Pretrained Language Models
Niu, T., Yavuz, S., Zhou, Y., Keskar, N. S., Wang, H., & Xiong, C. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.417 · Cited by 24
Ding, L., Wang, L., Shi, S., Tao, D., & Tu, Z. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.172 · Cited by 22
Dynamic Self-Supervised Teacher-Student Network Learning
Ye, F., & Borş, A. G. (2022) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2022.3220928 · Cited by 22
Improving Non-autoregressive Neural Machine Translation with Monolingual Data
Zhou, J., & Keung, P. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.171 · Cited by 24
Amalgamating Knowledge from Two Teachers for Task-oriented Dialogue System with Adversarial Training
He, W., Yang, M., Yan, R., et al. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.281 · Cited by 21
Microsoft Research Asia’s Systems for WMT19
Xia, Y., Tan, X., Tian, F., Gao, F., He, D., Chen, W., Fan, Y., Gong, L., Leng, Y.-C., Luo, R., et al. (2019) · Conference on Machine Translation · doi:10.18653/v1/w19-5348 · Cited by 26
Learning to Solve NLP Tasks in an Incremental Number of Languages
Castellucci, G., Filice, S., Croce, D., & Basili, R. (2021) · doi:10.18653/v1/2021.acl-short.106 · Cited by 21
Towards Quantifiable Dialogue Coherence Evaluation
Zheng, Y., Lu, L., Huang, L., Lin, L., & Liang, X. (2021) · doi:10.18653/v1/2021.acl-long.211 · Cited by 20
WACO: Word-Aligned Contrastive Learning for Speech Translation
Ouyang, S., Ye, R., & Li, L. (2023) · Annual Meeting of the Association for Computational Linguistics (ACL) · doi:10.18653/v1/2023.acl-long.216 · Cited by 20
Exploring weight symmetry in deep neural networks
Hu, S. X., Zagoruyko, S., & Komodakis, N. (2019) · Computer Vision and Image Understanding · doi:10.1016/j.cviu.2019.07.006 · Cited by 27
Yin, J., Guo, L., Jiang, W., et al. (2021) · Computer Communications · doi:10.1016/j.comcom.2021.05.005 · Cited by 20
Cross-modal knowledge distillation for continuous sign language recognition
Gao, L., Shi, P., Hu, L., Feng, J., Zhu, L., Wan, L., & Feng, W. (2024) · Neural Networks · doi:10.1016/j.neunet.2024.106587 · Cited by 22
Chuang, S.-P., Chuang, Y.-S., Chang, C.-C., & Lee, H.-Y. (2021) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2021.findings-acl.92 · Cited by 19
An Edge based Federated Learning Framework for Person Re-identification in UAV Delivery Service
Zhang, C., Liu, X., Jia, X., Chen, T., Li, G., Jiang, F., & Li, X. (2021) · IEEE International Conference on Web Services (ICWS) · doi:10.1109/icws53863.2021.00070 · Cited by 19
Transformer-Based Text-to-Speech with Weighted Forced Attention
Okamoto, T., Toda, T., Shiga, Y., & Kawai, H. (2020) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp40776.2020.9053915 · Cited by 19
Imitation Learning for Non-Autoregressive Neural Machine TranslationPreprint
Wei, B., Wang, M., Zhou, H., Lin, J., Xie, J., & Sun, X. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1906.02041 · Cited by 19
An Enhanced and Lightweight YOLOv8-Based Model for Accurate Rice Pest Detection
Liu, G., Di, J., Wang, Q., Zhao, Y., & Yang, Y. (2025) · IEEE Access · doi:10.1109/access.2025.3569819 · Cited by 22
Distilling Multiple Domains for Neural Machine Translation
Currey, A., Mathur, P., & Dinu, G. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.364 · Cited by 21
Incorporating a Local Translation Mechanism into Non-autoregressive Translation
Kong, X., Zhang, Z., & Hovy, E. H. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.79 · Cited by 21
ESPnet-ST IWSLT 2021 Offline Speech Translation System
Inaguma, H., Yan, B., Dalmia, S., Guo, P., Shi, J., Duh, K., & Watanabe, S. (2021) · International Conference on Spoken Language Translation (IWSLT) · doi:10.18653/v1/2021.iwslt-1.10 · Cited by 19
The USTC-NELSLIP Offline Speech Translation Systems for IWSLT 2022
Zhang, W., Ye, Z., Tang, H., Li, X., Zhou, X., Yang, J., Cui, J., Pan, D., Shi, M., Song, Y., et al. (2022) · International Conference on Spoken Language Translation (IWSLT) · doi:10.18653/v1/2022.iwslt-1.15 · Cited by 18
f-Divergence Minimization for Sequence-Level Knowledge Distillation
Wen, Y., Li, Z., Du, W., & Mou, L. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.605 · Cited by 90
EdgeFormer: A Parameter-Efficient Transformer for On-Device Seq2seq Generation
Ge, T., Chen, S., & Wei, F. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.741 · Cited by 21
Distilling Knowledge for Search-based Structured Prediction
Liu, Y., Che, W., Zhao, H., Qin, B., & Liu, T. (2018) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p18-1129 · Cited by 21
Transductive Ensemble Learning for Neural Machine Translation
Wang, Y.-R., Wu, L., Xia, Y., Qin, T., Zhai, C., & Liu, T. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i04.6097 · Cited by 18
A Simplified Fully Quantized Transformer for End-to-end Speech RecognitionPreprint
Bie, A., Venkitesh, B., Monteiro, J., Haidar, M. A., & Rezagholizadeh, M. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1911.03604 · Cited by 32
LLM for Mobile: An Initial Roadmap
Chen, D., Liu, Y., Zhou, M., Zhao, Y., Wang, H., Wang, S., Chen, X., Bissyandé, T. F., Klein, J., & Li, L. (2024) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3708528 · Cited by 27
The USTC-NELSLIP Systems for Simultaneous Speech Translation Task at IWSLT 2021
Liu, D., Du, M., Li, X., et al. (2021) · International Conference on Spoken Language Translation (IWSLT) · doi:10.18653/v1/2021.iwslt-1.2 · Cited by 17
Shao, C., Wu, X., & Feng, Y. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.277 · Cited by 23
Spatial Knowledge Distillation to Aid Visual Reasoning
Aditya, S., Saha, R. R., Yang, Y., & Baral, C. R. (2019) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv.2019.00030 · Cited by 17
A review of Thai–English machine translation
Lyons, S. (2020) · Machine Translation · doi:10.1007/s10590-020-09248-8 · Cited by 18
Autoregressive Knowledge Distillation through Imitation Learning
Lin, A., Wohlwend, J., Chen, H., & Lei, T. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.494 · Cited by 70
Light-TTS: Lightweight Multi-Speaker Multi-Lingual Text-to-Speech
Li, S., Ouyang, B., Li, L., et al. (2021) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp39728.2021.9414400 · Cited by 17
XCode : Towards Cross-Language Code Representation with Large-Scale Pre-Training
Lin, Z., Li, G., Zhang, J., Deng, Y., Zeng, X., Zhang, Y., & Wan, Y. (2022) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3506696 · Cited by 17
Maximum Bayes Smatch Ensemble Distillation for AMR Parsing
Lee, Y., Astudillo, R. F., Lam, H. T., Naseem, T., Florian, R., & Roukos, S. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.393 · Cited by 19
Mimic and Conquer: Heterogeneous Tree Structure Distillation for Syntactic NLP
Fei, H., Ren, Y., & Ji, D. (2020) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2020.findings-emnlp.18 · Cited by 16
Metadata
- Authors
- Kim, Y., & Rush, A. M.
- Affiliations
- Harvard University (US)
- Venue
- Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing
- Published
- 2016
- Pages
- 1317-1327
- OpenAlex
- W2963736842
- Semantic Scholar
- 57a10537978600fd33dcdd48922c791609a4851a
- Cited by
- 836
- References
- 53
- Citations by year
- 2026: 36 · 2025: 59 · 2024: 52 · 2023: 137 · 2022: 115 · 2021: 157 · 2020: 143 · 2019: 75 · 2018: 42 · 2017: 16 · 2016: 4
- FWCI
- 24.59
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Conference on Empirical Methods in Natural Language Processing (EMNLP) (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Keywords
- knowledge distillation, neural machine translation, model compression, weight pruning, teacher-student models, parameter reduction
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-10-23, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-23 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
