Amortized Neural Networks for Low-Latency Speech Recognition
Open accessThe DOI matches its Crossref record; 44 of the 44 references it cites were found in a registry or an index.
Abstract
We introduce Amortized Neural Networks (AmNets), a compute cost- and latency-aware network architecture particularly well-suited for sequence modeling tasks. We apply AmNets to the Recurrent Neural Network Transducer (RNN-T) to reduce compute cost and latency for an automatic speech recognition (ASR) task. The AmNets RNN-T architecture enables the network to dynamically switch between encoder branches on a frame-by-frame basis. Branches are constructed with variable levels of compute cost and model capacity. Here, we achieve variable compute for two well-known candidate techniques: one using sparse pruning and the other using matrix factorization. Frame-by-frame switching is determined by an arbitrator network that requires negligible compute overhead. We present results using both architectures on LibriSpeech data and show that our proposed architecture can reduce inference cost by up to 45\% and latency to nearly real-time without incurring a loss in accuracy.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source doesn’t support the claim as written.
In the citing paper
“The network is slightly simplified with respect to state-of-the-art RNNTs; the long short-term memory (LSTM) blocks are unidirectional, rather than bidirectional, and the decoding scheme is greedy rather than beam-search, which increases the WER slightly but makes online continuous-streaming use much more straightforward27.”
From An analog-AI chip for energy-efficient speech recognition and transcription (2023) · cites this paper as [27]
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
$rm K$-SVD: An Algorithm for Designing Overcomplete Dictionaries for Sparse Representation
Aharon, M., Elad, M., & Bruckstein, A. (2006) · IEEE Transactions on Signal Processing · doi:10.1109/tsp.2006.881199 · Cited by 9,451
Speech recognition with deep recurrent neural networks
Graves, A., Mohamed, A.-R., & Hinton, G. (2013) · 2013 IEEE International Conference on Acoustics, Speech and Signal Processing · doi:10.1109/icassp.2013.6638947 · Cited by 8,893
Greff, K., Srivastava, R. K., Koutnik, J., Steunebrink, B. R., & Schmidhuber, J. (2017) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2016.2582924 · Cited by 6,955
“Librispeech: an ASR corpus based on public domain audio books,”
Panayotov, V., Chen, G., Povey, D., & Khudanpur, S. (2015) · 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2015.7178964 · Cited by 6,072
Categorical Reparameterization with Gumbel-Softmax
Jang, E., Gu, S., & Poole, B. (2016) · arXiv · doi:10.48550/arxiv.1611.01144 · Cited by 3,616
Statistical Learning with Sparsity: The Lasso and Generalizations
Hastie, T., Tibshirani, R., & Wainwright, M. (2015) · Chapman & Hall/CRC eBooks · doi:10.1201/b18401 · Cited by 2,520
Hybrid speech recognition with Deep Bidirectional LSTM
Graves, A., Jaitly, N., & Mohamed, A.-R. (2013) · 2013 IEEE Workshop on Automatic Speech Recognition and Understanding · doi:10.1109/asru.2013.6707742 · Cited by 1,736
Bivariate Exponential Distributions
Gumbel, E. J. (1960) · Journal of the American Statistical Association · doi:10.1080/01621459.1960.10483368 · Cited by 1,321
To prune, or not to prune: exploring the efficacy of pruning for model compression
Zhu, M., & Gupta, S. (2017) · arXiv · doi:10.48550/arxiv.1710.01878 · Cited by 805
Streaming End-to-end Speech Recognition for Mobile Devices
He, Y., Sainath, T. N., Prabhavalkar, R., McGraw, I., Alvarez, R., Zhao, D., Rybach, D., Kannan, A., Wu, Y., Pang, R., et al. (2019) · ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2019.8682336 · Cited by 601
The State of Sparsity in Deep Neural Networks
Gale, T., Elsen, E., & Hooker, S. (2019) · arXiv · doi:10.48550/arxiv.1902.09574 · Cited by 440
Rao, K., Sak, H., & Prabhavalkar, R. (2017) · 2017 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU) · doi:10.1109/asru.2017.8268935 · Cited by 341
Adaptive Computation Time for Recurrent Neural Networks
Graves, A. (2016) · arXiv · doi:10.48550/arxiv.1603.08983 · Cited by 335
SVD Compression for Magnetic Resonance Fingerprinting in the Time Domain
McGivney, D. F., Pierre, E., Ma, D., Jiang, Y., Saybasili, H., Gulani, V., & Griswold, M. A. (2014) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2014.2337321 · Cited by 314
Singular Value Decomposition (SVD) Image Coding
Andrews, H., & Patterson, C. (1976) · IEEE Transactions on Communications · doi:10.1109/tcom.1976.1093309 · Cited by 304
Simple Recurrent Units for Highly Parallelizable Recurrence
Lei, T., Zhang, Y., Wang, S. I., Dai, H., & Artzi, Y. (2018) · Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-1477 · Cited by 289
A Streaming On-Device End-To-End Model Surpassing Server-Side Conventional Model Quality and Latency
Sainath, T. N., He, Y., Li, B., Narayanan, A., Pang, R., Bruguier, A., Chang, S.-Y., Li, W., Alvarez, R., Chen, Z., et al. (2020) · ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp40776.2020.9054188 · Cited by 185
Large-Scale Multilingual Speech Recognition with a Streaming End-to-End Model
Kannan, A., Datta, A., Sainath, T. N., Weinstein, E., Ramabhadran, B., Wu, Y., Bapna, A., Chen, Z., & Lee, S. (2019) · Interspeech 2019 · doi:10.21437/interspeech.2019-2858 · Cited by 155
Sparse low rank factorization for deep neural network compression
Swaminathan, S., Garg, D., Kannan, R., & Andres, F. (2020) · Neurocomputing · doi:10.1016/j.neucom.2020.02.035 · Cited by 147
Efficient Neural Network Compression
Kim, H., Khan, M. U. K., & Kyung, C. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.01285 · Cited by 142
On the Comparison of Popular End-to-End Models for Large Scale Speech Recognition
Li, J., Wu, Y., Gaur, Y., Wang, C., Zhao, R., & Liu, S. (2020) · Interspeech (USB) · doi:10.21437/interspeech.2020-2846 · Cited by 137
Lossy image compression using singular value decomposition and wavelet difference reduction
Rufai, A. M., Anbarjafari, G., & Demirel, H. (2013) · Digital Signal Processing · doi:10.1016/j.dsp.2013.09.008 · Cited by 134
Conditional Computation in Neural Networks for faster models
Bengio, E., Bacon, P., Pineau, J., & Precup, D. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1511.06297 · Cited by 130
Developing RNN-T Models Surpassing High-Performance Hybrid Models with Customization Capability
Li, J., Zhao, R., Meng, Z., Liu, Y., Wei, W., Parthasarathy, S., Mazalov, V., Wang, Z., He, L., Zhao, S., et al. (2020) · Interspeech 2020 · doi:10.21437/interspeech.2020-3016 · Cited by 98
Prabhavalkar, R., Alsharif, O., Bruguier, A., & McGraw, L. (2016) · 2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2016.7472823 · Cited by 97
Self-Attention Transducers for End-to-End Speech Recognition
Tian, Z., Yi, J., Tao, J., Bai, Y., & Wen, Z. (2019) · Interspeech (USB) · doi:10.21437/interspeech.2019-2203 · Cited by 64
Efficient Knowledge Distillation for RNN-Transducer Models
Panchapagesan, S., Park, D., Chiu, C., Shangguan, Y., Liang, Q., & Gruenstein, A. (2021) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp39728.2021.9413905 · Cited by 61
Exploring multidimensional lstms for large vocabulary ASR
Li, J., Mohamed, A., Zweig, G., & Gong, Y. (2016) · 2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2016.7472617 · Cited by 56
Optimizing Speech Recognition For The Edge
Shangguan, Y., Li, J., Liang, Q., Álvarez, R., & McGraw, I. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1909.12408 · Cited by 53
Dynamic Sparsity Neural Networks for Automatic Speech Recognition
Wu, Z., Zhao, D. X., Liang, Q., Yu, J., Gulati, A., & Pang, R. (2021) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp39728.2021.9414505 · Cited by 51
On the Efficient Representation and Execution of Deep Acoustic Models
Alvarez, R., Prabhavalkar, R., & Bakhtin, A. (2016) · Interspeech 2016 · doi:10.21437/interspeech.2016-128 · Cited by 44
Prabhavalkar, R., Alsharif, O., Bruguier, A., et al. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1603.08042 · Cited by 43
Variable Computation in Recurrent Neural Networks
Jernite, Y., Grave, É., Joulin, A., & Mikolov, T. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1611.06188 · Cited by 41
Jia, Y., Wu, Z., Xu, Y., Ke, D., & Su, K. (2017) · Journal of Robotics · doi:10.1155/2017/2061827 · Cited by 36
Quantization Aware Training with Absolute-Cosine Regularization for Automatic Speech Recognition
Nguyen, H. D., Alexandridis, A., & Mouchtaris, A. (2020) · Interspeech 2020 · doi:10.21437/interspeech.2020-1991 · Cited by 35
Addressing Sparsity in Deep Neural Networks
Zhou, X., Chen, Y., Du, Z., Zhang, S., Zhang, L., Lan, H., Liu, S., Li, L., Guo, Q., & Chen, T. (2018) · IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems · doi:10.1109/tcad.2018.2864289 · Cited by 21
Layer sparsity in neural networks
Hebiri, M., Lederer, J., & Taheri, M. (2024) · Journal of Statistical Planning and Inference · doi:10.1016/j.jspi.2024.106195 · Cited by 18
Bifocal Neural ASR: Exploiting Keyword Spotting for Inference Optimization
Macoskey, J. J., Strimel, G. P., & Rastrow, A. (2021) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp39728.2021.9414652 · Cited by 15
Self-citationCross Attention with Monotonic Alignment for Speech Transformer
Zhao, Y., Ni, C., Leung, C.-C., Joty, S., Chng, E. S., & Ma, B. (2020) · Interspeech (USB) · doi:10.21437/interspeech.2020-1198 · Cited by 13
Fast Intent Classification for Spoken Language Understanding Systems
Tyagi, A., Sharma, V., Gupta, R., Samson, L., Zhuang, N., Wang, Z., & Campbell, B. (2020) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp40776.2020.9054240 · Cited by 10
Dynamic Sparsity Neural Networks for Automatic Speech RecognitionPreprint
Wu, Z., Zhao, D. X., Liang, Q., Yu, J., Gulati, A., & Pang, R. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2005.10627 · Cited by 2
Layer Sparsity in Neural NetworksPreprint
Hebiri, M., & Lederer, J. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.15604
Research works printed but not linked: 2
- [11][11] R. Pang, T. N. Sainath, and R. P. et al., 'Compression of end-toend models,' in Proc., Interspeech , Sep. 2018. -
- [19][19] T. Bolukbasi, J. Wang, O. Dekel, and V. Saligrama, 'Adaptive neural networks for efficient inference,' in 34th International Conference on Machine Learning, ICML , vol. 2, 2017. -
Cited by
Recent Advances in End-to-End Automatic Speech Recognition
Li, J. (2022) · APSIPA Transactions on Signal and Information Processing · doi:10.1561/116.00000050 · Cited by 395
An analog-AI chip for energy-efficient speech recognition and transcription
Ambrogio, S., Narayanan, P., Okazaki, A., Fasoli, A., Mackin, C., Hosokawa, K., Nomura, A., Yasuda, T., Chen, A., Friz, A., et al. (2023) · Nature · doi:10.1038/s41586-023-06337-5 · Cited by 269
“…with respect to state-of-the-art RNNTs; the long short-term memory (LSTM) blocks are unidirectional, rather than bidirectional, and the decoding scheme is greedy rather than beam-search, which increases the WER slightly but makes online continuous-streaming use much more straightforward 27 .”
Background
Metadata
- Authors
- Macoskey, J., Strimel, G. P., Su, J., & Rastrow, A.
- Affiliations
- Amazon (United States) (US)
- ORCID
- Macoskey, Jonathan
- Venue
- Interspeech 2021
- Publisher
- ISCA
- Published
- 2021-08-30
- ISSN
- 2308-457X (ISSN-L)
- OpenAlex
- W3178511319
- Cited by
- 12
- References
- 44
- Citations by year
- 2024: 1 · 2023: 8 · 2022: 3
- FWCI
- 1.37
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- cc-by
- Locations
- Interspeech (USB) (publishedVersion)
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University) (cc-by)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Speech Recognition and Synthesis
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Computer Science › Signal Processing › Speech and Audio Processing
- Keywords
- RNN-T, matrix factorization, inference cost reduction, real-time inference, sequence modeling
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.