Distilling Knowledge from Ensembles of Neural Networks for Speech Recognition
The DOI matches its Crossref record; 4 of the 4 references it cites were found in a registry or an index.
Abstract
Speech recognition systems that combine multiple types of acoustic models have been shown to outperform single-model systems. However, such systems can be complex to implement and too resource-intensive to use in production. This paper describes how to use knowledge distillation to combine acoustic models in a way that has the best of many worlds: It improves recognition accuracy significantly, can be implemented with standard training tools, and requires no additional complexity during recognition. First, we identify a simple but particularly strong type of ensemble: a late combination of recurrent neural networks with different architectures and training objectives. To harness such an ensemble, we use a variant of standard cross-entropy training to distill it into a single model and then discriminatively fine-tune the result. An evaluation on 2,000-hour large vocabulary tasks in 5 languages shows that the distilled models provide up to 8.9% relative WER improvement over conventionally-trained baselines with an identical number of parameters.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318 · cited 6×
“This work found a simple but powerful class of acoustic model ensembles consisting of LSTM and CLDNN models built with different training objectives.”
MethodInfluentialFitNets: Hints for Thin Deep Nets
Romero, A., Ballas, N., Kahou, S. E., Chassang, A., Gatta, C., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1412.6550 · Cited by 2,859
Measures of Diversity in Classifier Ensembles and Their Relationship with the Ensemble Accuracy
Kuncheva, L. I., & Whitaker, C. J. (2003) · Machine Learning · doi:10.1023/a:1022859003006 · Cited by 2,335 · cited 1×
“[15, 16]), models that tend to make errors on different inputs can compensate for one another in combination and make fewer errors than any model individually.”
BackgroundConvolutional, Long Short-Term Memory, fully connected Deep Neural Networks
Sainath, T. N., Vinyals, O., Senior, A., & Sak, H. (2015) · 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2015.7178838 · Cited by 1,678
Cited by
Knowledge Distillation: A Survey
Gou, J., Yu, B., Maybank, S. J., & Tao, D. (2021) · International Journal of Computer Vision · doi:10.1007/s11263-021-01453-z · Cited by 3,931
On the Efficacy of Knowledge Distillation
Cho, J. H., & Hariharan, B. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00489 · Cited by 637
Towards Understanding Ensemble, Knowledge Distillation and Self-Distillation in Deep Learning
Allen-Zhu, Z., & Li, Y. (2020) · arXiv · doi:10.48550/arxiv.2012.09816 · Cited by 178
A Review of Privacy-Preserving Federated Learning for the Internet-of-Things
Briggs, C., Fan, Z., & Andras, P. (2021) · Studies in Computational Intelligence · doi:10.1007/978-3-030-70604-3_2 · Cited by 101
“network) to emulate the predictions of an ensemble model [29, 30, 31].”
BackgroundContinual Learning for Fake Audio Detection
Ma, H., Yi, J., Tao, J., Bai, Y., Tian, Z., & Wang, C. (2021) · Interspeech 2021 · doi:10.21437/interspeech.2021-794 · Cited by 39
Metadata
- Authors
- Chebotar, Y., & Waters, A.
- Venue
- Interspeech 2016
- Publisher
- ISCA
- Published
- 2016-09-08
- Pages
- 3439-3443
- ISSN
- 2308-457X (ISSN-L)
- OpenAlex
- W2508418541
- Cited by
- 167
- References
- 4
- FWCI
- 7.26
- Type
- conference paper
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.