The DOI matches its Crossref record; 21 of the 21 references it cites were found in a registry or an index.
ESE: Efficient Speech Recognition Engine with Sparse LSTM on FPGA
Abstract
Long Short-Term Memory (LSTM) is widely used in speech recognition. In order to achieve higher prediction accuracy, machine learning scientists have built increasingly larger models. Such large model is both computation intensive and memory intensive. Deploying such bulky model results in high power consumption and leads to a high total cost of ownership (TCO) of a data center. To speedup the prediction and make it energy efficient, we first propose a load-balance-aware pruning method that can compress the LSTM model size by 20x (10x from pruning and 2x from quantization) with negligible loss of the prediction accuracy. The pruned model is friendly for parallel processing. Next, we propose a scheduler that encodes and partitions the compressed model to multiple PEs for parallelism and schedule the complicated LSTM data flow. Finally, we design the hardware architecture, named Efficient Speech Recognition Engine (ESE) that works directly on the sparse LSTM model.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Citations
Cites
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318 · cited 1×
“In this work, we evaluated the most complex one: LSTM [14].”
MethodHan, S., Mao, H., & Dally, W. J. (2015) · arXiv · doi:10.48550/arxiv.1510.00149 · Cited by 4,662
Self-citationKaldi Speech Recognition Toolkit
Povey, D. (2024) · Infoscience (Ecole Polytechnique Fédérale de Lausanne) · doi:10.57702/jb3fvbn9 · Cited by 4,609
Long short-term memory recurrent neural network architectures for large scale acoustic modeling
Sak, H., Senior, A., & Beaufays, F. (2014) · Interspeech 2014 · doi:10.21437/interspeech.2014-80 · Cited by 2,973
Deep Speech 2: End-to-End Speech Recognition in English and Mandarin
Amodei, D., Anubhai, R., Battenberg, E., Case, C. T., Casper, J., Catanzaro, B., Chen, J., Chrzanowski, M., Coates, A., Diamos, G., et al. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1512.02595 · Cited by 2,110 · cited 1×
“Deep neural network is becoming the state-of-the-art method for speech recognition [6, 13].”
MethodSelf-citationDaDianNao: A Machine-Learning Supercomputer
Chen, Y., Luo, T., Liu, S., Zhang, S., He, L., Wang, J., Li, L., Chen, T., Xu, Z., Sun, N., et al. (2014) · 2014 47th Annual IEEE/ACM International Symposium on Microarchitecture · doi:10.1109/micro.2014.58 · Cited by 1,492 · cited 3×
“DaDianNao [3] and ShiDianNao [5] eliminate the DRAM access by having all weights on-chip (eDRAM or SRAM).”
BackgroundLearning both Weights and Connections for Efficient Neural Networks
Han, S., Pool, J., Tran, J., & Dally, W. J. (2015) · arXiv · doi:10.48550/arxiv.1506.02626 · Cited by 1,481 · cited 2×
“It has been widely observed that deep neural networks usually have a lot of redundancy [11, 12].”
BackgroundSelf-citationDianNao: a small-footprint high-throughput accelerator for ubiquitous machine-learning
Chen, T., Du, Z., Sun, N., Wang, J., Wu, C., Chen, Y., & Temam, O. (2014) · Proceedings of the 19th international conference on Architectural support for programming languages and operating systems · doi:10.1145/2541940.2541967 · Cited by 1,297 · cited 1×
“DianNao [2] implements an array of multiply-add units to map large DNN onto its core architecture.”
MethodDARPA TIMIT: acoustic-phonetic continuous speech corpus CD-ROM, NIST speech disc 1-1.1
Garofolo, J. S., Lamel, L. F., Fisher, W. M., Fiscus, J. G., Pallett, D. S., & Dahlgren, N. L. (1993) · doi:10.6028/nist.ir.4930 · Cited by 1,267
Going Deeper with Embedded FPGA Platform for Convolutional Neural Network
Qiu, J., Wang, J., Yao, S., Guo, K., Li, B., Zhou, E., Yu, J., Tang, T., Xu, N., Song, S., et al. (2016) · Proceedings of the 2016 ACM/SIGDA International Symposium on Field-Programmable Gate Arrays · doi:10.1145/2847263.2847265 · Cited by 1,252
Self-citation
Research works printed but not linked: 2
- A. Hannun , C. Case , J. Casper , B. Catanzaro , G. Diamos , E. Elsen , R. Prenger , S. Satheesh , S. Sengupta , A. Coates , and A. Ng . Deep speech: Scaling up end-to-end speech recognition. arXiv, preprint arXiv:1412.5567 , 2014 . A. Hannun, C. Case, J. Casper, B. Catanzaro, G. Diamos, E. Elsen, R. Prenger, S. Satheesh, S. Sengupta, A. Coates, and A. Ng. Deep speech: Scaling up end-to-end speech recognition. arXiv, preprint arXiv:1412.5567, 2014. · arXiv
- M. Lee , K. Hwang , J. Park , S. Choi , S. Shin , and W. Sung . Fpga-based low-power speech recognition with recurrent neural networks. arXiv preprint arXiv:1610.00552 , 2016 . M. Lee, K. Hwang, J. Park, S. Choi, S. Shin, and W. Sung. Fpga-based low-power speech recognition with recurrent neural networks. arXiv preprint arXiv:1610.00552, 2016. · arXiv
Cited by
Deep Learning With Edge Computing: A Review
Chen, J., & Ran, X. (2019) · Proceedings of the IEEE · doi:10.1109/jproc.2019.2921977 · Cited by 1,540
Convergence of Edge Computing and Deep Learning: A Comprehensive Survey
Wang, X., Han, Y., Leung, V. C. M., Niyato, D., Yan, X., & Chen, X. (2020) · IEEE Communications Surveys & Tutorials · doi:10.1109/comst.2020.2970550 · Cited by 1,498
“Therefore, hardware architectures should be designed to work directly for optimized models [145].”
BackgroundMethodAMC: AutoML for Model Compression and Acceleration on Mobile Devices
He, Y., Lin, J., Liu, Z., Wang, H., Li, L.-J., & Han, S. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01234-2_48 · Cited by 1,400
“Fine-grained pruning method prunes neural networks based on individual connections to achieve sparsity in both weights and activations, which is able to achieve higher compression ratio and can be accelerated with specialized hardware such as [18, 17, 39].”
BackgroundMethodSelf-citationTowards artificial general intelligence with hybrid Tianjic chip architecture
Pei, J., Deng, L., Song, S., Zhao, M., Zhang, Y., Wu, S., Wang, G., Zou, Z., Wu, Z., He, W., et al. (2019) · Nature · doi:10.1038/s41586-019-1424-8 · Cited by 1,082
Self-citationModel Compression and Hardware Acceleration for Neural Networks: A Comprehensive Survey
Deng, B. L., Li, G., Han, S., Shi, L., & Xie, Y. (2020) · Proceedings of the IEEE · doi:10.1109/jproc.2020.2976475 · Cited by 996
“Quantization and sparsification are the most widely combined ones to simultaneously reduce the data precision and the number of operations for higher throughput and lower energy [13], [255], [284], [286]–[289].”
BackgroundMethodSelf-citation
Metadata
- Authors
- Han, S., Kang, J., Mao, H., Hu, Y., Li, X., Li, Y., Xie, D., Luo, H., Yao, S., Wang, Y., Yang, H., & Dally, W. (. J.
- Affiliations
- Stanford University (US)
- DeePhi, Beijing, China
- Tsinghua University (CN)
- Stanford University & NVIDIA, Stanford, USA
- Venue
- Proceedings of the 2017 ACM/SIGDA International Symposium on Field-Programmable Gate Arrays
- Publisher
- ACM
- Published
- 2017-02-22
- Pages
- 75-84
- OpenAlex
- W2585720638
- Cited by
- 591
- References
- 21
- Citations by year
- 2026: 18 · 2025: 27 · 2024: 34 · 2023: 37 · 2022: 63 · 2021: 85 · 2020: 110 · 2019: 109 · 2018: 83 · 2017: 25
- FWCI
- 26.14
- Type
- conference paper
- Language
- es
- Open access
- closed
- Locations
- ACM/SIGDA International Symposium on Field-Programmable Gate Arrays (FPGA) (publishedVersion)
- Funders
- National Natural Science Foundation of China (61261160501, 61373026, 61622403)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Computer Science › Hardware and Architecture › Parallel Computing and Optimization Techniques
- Physical Sciences › Computer Science › Hardware and Architecture › Embedded Systems Design Techniques
- Keywords
- speech recognition, LSTM, model compression, pruning, quantization, load balancing, parallel computing, hardware architecture, energy efficiency
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: ESE: Efficient Speech Recognition Engine with Sparse LSTM on FPGA. https://app.cochat.ai/papers/10.1145/3020078.3021745
- The paper
- Han, S., Kang, J., Mao, H., Hu, Y., Li, X., Li, Y., Xie, D., Luo, H., Yao, S., Wang, Y., Yang, H., & Dally, W. (. J. (2017). ESE: Efficient Speech Recognition Engine with Sparse LSTM on FPGA. Proceedings of the 2017 ACM/SIGDA International Symposium on Field-Programmable Gate Arrays. https://doi.org/10.1145/3020078.3021745
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.