ESE: Efficient Speech Recognition Engine with Sparse LSTM on FPGA
The DOI matches its Crossref record; 21 of the 21 references it cites were found in a registry or an index.
Abstract
Long Short-Term Memory (LSTM) is widely used in speech recognition. In order to achieve higher prediction accuracy, machine learning scientists have built increasingly larger models. Such large model is both computation intensive and memory intensive. Deploying such bulky model results in high power consumption and leads to a high total cost of ownership (TCO) of a data center. To speedup the prediction and make it energy efficient, we first propose a load-balance-aware pruning method that can compress the LSTM model size by 20x (10x from pruning and 2x from quantization) with negligible loss of the prediction accuracy. The pruned model is friendly for parallel processing. Next, we propose a scheduler that encodes and partitions the compressed model to multiple PEs for parallelism and schedule the complicated LSTM data flow. Finally, we design the hardware architecture, named Efficient Speech Recognition Engine (ESE) that works directly on the sparse LSTM model.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Citations
Cites
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318 · cited 1×
“In this work, we evaluated the most complex one: LSTM [14].”
MethodHan, S., Mao, H., & Dally, W. J. (2015) · arXiv · doi:10.48550/arxiv.1510.00149 · Cited by 4,662
Self-citationKaldi Speech Recognition Toolkit
Povey, D. (2024) · Infoscience (Ecole Polytechnique Fédérale de Lausanne) · doi:10.57702/jb3fvbn9 · Cited by 4,609
Long short-term memory recurrent neural network architectures for large scale acoustic modeling
Sak, H., Senior, A., & Beaufays, F. (2014) · Interspeech 2014 · doi:10.21437/interspeech.2014-80 · Cited by 2,973
Deep Speech 2: End-to-End Speech Recognition in English and Mandarin
Amodei, D., Anubhai, R., Battenberg, E., Case, C. T., Casper, J., Catanzaro, B., Chen, J., Chrzanowski, M., Coates, A., Diamos, G., et al. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1512.02595 · Cited by 2,110 · cited 1×
“Deep neural network is becoming the state-of-the-art method for speech recognition [6, 13].”
MethodSelf-citationDaDianNao: A Machine-Learning Supercomputer
Chen, Y., Luo, T., Liu, S., Zhang, S., He, L., Wang, J., Li, L., Chen, T., Xu, Z., Sun, N., et al. (2014) · 2014 47th Annual IEEE/ACM International Symposium on Microarchitecture · doi:10.1109/micro.2014.58 · Cited by 1,492 · cited 3×
“DaDianNao [3] and ShiDianNao [5] eliminate the DRAM access by having all weights on-chip (eDRAM or SRAM).”
BackgroundLearning both Weights and Connections for Efficient Neural Networks
Han, S., Pool, J., Tran, J., & Dally, W. J. (2015) · arXiv · doi:10.48550/arxiv.1506.02626 · Cited by 1,481 · cited 2×
“It has been widely observed that deep neural networks usually have a lot of redundancy [11, 12].”
BackgroundSelf-citationDianNao: a small-footprint high-throughput accelerator for ubiquitous machine-learning
Chen, T., Du, Z., Sun, N., Wang, J., Wu, C., Chen, Y., & Temam, O. (2014) · Proceedings of the 19th international conference on Architectural support for programming languages and operating systems · doi:10.1145/2541940.2541967 · Cited by 1,297 · cited 1×
“DianNao [2] implements an array of multiply-add units to map large DNN onto its core architecture.”
MethodDARPA TIMIT: acoustic-phonetic continuous speech corpus CD-ROM, NIST speech disc 1-1.1
Garofolo, J. S., Lamel, L. F., Fisher, W. M., Fiscus, J. G., Pallett, D. S., & Dahlgren, N. L. (1993) · doi:10.6028/nist.ir.4930 · Cited by 1,267
Going Deeper with Embedded FPGA Platform for Convolutional Neural Network
Qiu, J., Wang, J., Yao, S., Guo, K., Li, B., Zhou, E., Yu, J., Tang, T., Xu, N., Song, S., et al. (2016) · Proceedings of the 2016 ACM/SIGDA International Symposium on Field-Programmable Gate Arrays · doi:10.1145/2847263.2847265 · Cited by 1,252
Self-citationEIE: Efficient Inference Engine on Compressed Deep Neural Network
Han, S., Liu, X., Mao, H., Pu, J., Pedram, A., Horowitz, M. A., & Dally, W. J. (2016) · 2016 ACM/IEEE 43rd Annual International Symposium on Computer Architecture (ISCA) · doi:10.1109/isca.2016.30 · Cited by 1,077
Self-citationShiDianNao: shifting vision processing closer to the sensor
Du, Z., Fasthuber, R., Chen, T., Ienne, P., Li, L., Luo, T., Feng, X., Chen, Y., & Temam, O. (2015) · Proceedings of the 42nd Annual International Symposium on Computer Architecture · doi:10.1145/2749469.2750389 · Cited by 933 · cited 1×
“DaDianNao [3] and ShiDianNao [5] eliminate the DRAM access by having all weights on-chip (eDRAM or SRAM).”
Darpa Timit Acoustic-Phonetic Continuous Speech Corpus CD-ROM {TIMIT} | NIST
Garofolo, J. S., Lamel, L. F., Fisher, W. M., Fiscus, J. G., Pallett, D. S., & Dahlgren, N. L. (1993) · Cited by 680
2013 IEEE Workshop on Automatic Speech Recognition and Understanding
(2013) · doi:10.1109/asru31144.2013 · Cited by 402
Sparse Matrix-Vector multiplication on FPGAs
Zhuo, L., & Prasanna, V. K. (2005) · Proceedings of the 2005 ACM/SIGDA 13th international symposium on Field-programmable gate arrays · doi:10.1145/1046192.1046202 · Cited by 225 · cited 1×
“[21] proposed an FPGA-based design on Virtex-II Pro for SpMV.”
MethodAccelerating recurrent neural networks in analytics servers: Comparison of FPGA, CPU, GPU, and ASIC
Nurvitadhi, E., Sim, J., Sheffield, D., Mishra, A., Krishnan, S., & Marr, D. (2016) · 2016 26th International Conference on Field Programmable Logic and Applications (FPL) · doi:10.1109/fpl.2016.7577314 · Cited by 175
Recurrent Neural Networks Hardware Implementation on FPGAPreprint
Chang, A. X. M., Martini, B., & Culurciello, E. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1511.05552 · Cited by 152 · cited 1×
“2 layers and 128 hidden units in hardware [1].”
BackgroundA High Memory Bandwidth FPGA Accelerator for Sparse Matrix-Vector Multiplication
Fowers, J., Ovtcharov, K., Strauß, K., Chung, E. S., & Stitt, G. (2014) · IEEE International Symposium on Field-Programmable Custom Computing Machines (FCCM) · doi:10.1109/fccm.2014.23 · Cited by 144
A scalable sparse matrix-vector multiplication kernel for energy-efficient sparse-blas on FPGAs
Dorrance, R., Ren, F., & Markovic, D. M. (2014) · ACM/SIGDA International Symposium on Field-Programmable Gate Arrays (FPGA) · doi:10.1145/2554688.2554785 · Cited by 89
Angel-Eye: A Complete Design Flow for Mapping CNN onto Customized Hardware
Guo, K., Sui, L., Qiu, J., Yao, S., Han, S., Wang, Y., & Yang, H. (2016) · IEEE Computer Society Annual Symposium on VLSI · doi:10.1109/isvlsi.2016.129 · Cited by 79
Self-citationAn Overview of Modern Speech Recognition.
Huang, X., & Deng, L. (2010) · Cited by 58
Research works printed but not linked: 2
- A. Hannun , C. Case , J. Casper , B. Catanzaro , G. Diamos , E. Elsen , R. Prenger , S. Satheesh , S. Sengupta , A. Coates , and A. Ng . Deep speech: Scaling up end-to-end speech recognition. arXiv, preprint arXiv:1412.5567 , 2014 . A. Hannun, C. Case, J. Casper, B. Catanzaro, G. Diamos, E. Elsen, R. Prenger, S. Satheesh, S. Sengupta, A. Coates, and A. Ng. Deep speech: Scaling up end-to-end speech recognition. arXiv, preprint arXiv:1412.5567, 2014. · arXiv
- M. Lee , K. Hwang , J. Park , S. Choi , S. Shin , and W. Sung . Fpga-based low-power speech recognition with recurrent neural networks. arXiv preprint arXiv:1610.00552 , 2016 . M. Lee, K. Hwang, J. Park, S. Choi, S. Shin, and W. Sung. Fpga-based low-power speech recognition with recurrent neural networks. arXiv preprint arXiv:1610.00552, 2016. · arXiv
Cited by
Deep Learning With Edge Computing: A Review
Chen, J., & Ran, X. (2019) · Proceedings of the IEEE · doi:10.1109/jproc.2019.2921977 · Cited by 1,540
Convergence of Edge Computing and Deep Learning: A Comprehensive Survey
Wang, X., Han, Y., Leung, V. C. M., Niyato, D., Yan, X., & Chen, X. (2020) · IEEE Communications Surveys & Tutorials · doi:10.1109/comst.2020.2970550 · Cited by 1,498
“Therefore, hardware architectures should be designed to work directly for optimized models [145].”
BackgroundMethodAMC: AutoML for Model Compression and Acceleration on Mobile Devices
He, Y., Lin, J., Liu, Z., Wang, H., Li, L.-J., & Han, S. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01234-2_48 · Cited by 1,400
“Fine-grained pruning method prunes neural networks based on individual connections to achieve sparsity in both weights and activations, which is able to achieve higher compression ratio and can be accelerated with specialized hardware such as [18, 17, 39].”
BackgroundMethodSelf-citationTowards artificial general intelligence with hybrid Tianjic chip architecture
Pei, J., Deng, L., Song, S., Zhao, M., Zhang, Y., Wu, S., Wang, G., Zou, Z., Wu, Z., He, W., et al. (2019) · Nature · doi:10.1038/s41586-019-1424-8 · Cited by 1,082
Self-citationModel Compression and Hardware Acceleration for Neural Networks: A Comprehensive Survey
Deng, B. L., Li, G., Han, S., Shi, L., & Xie, Y. (2020) · Proceedings of the IEEE · doi:10.1109/jproc.2020.2976475 · Cited by 996
“Quantization and sparsification are the most widely combined ones to simultaneously reduce the data precision and the number of operations for higher throughput and lower energy [13], [255], [284], [286]–[289].”
BackgroundMethodSelf-citation
Metadata
- Authors
- Han, S., Kang, J., Mao, H., Hu, Y., Li, X., Li, Y., Xie, D., Luo, H., Yao, S., Wang, Y., Yang, H., & Dally, W. (. J.
- Affiliations
- Stanford University (US)
- DeePhi, Beijing, China
- Tsinghua University (CN)
- Stanford University & NVIDIA, Stanford, USA
- Venue
- Proceedings of the 2017 ACM/SIGDA International Symposium on Field-Programmable Gate Arrays
- Publisher
- ACM
- Published
- 2017-02-22
- Pages
- 75-84
- OpenAlex
- W2585720638
- Cited by
- 591
- References
- 21
- Citations by year
- 2026: 18 · 2025: 27 · 2024: 34 · 2023: 37 · 2022: 63 · 2021: 85 · 2020: 110 · 2019: 109 · 2018: 83 · 2017: 25
- FWCI
- 26.14
- Type
- conference paper
- Language
- es
- Open access
- closed
- Locations
- ACM/SIGDA International Symposium on Field-Programmable Gate Arrays (FPGA) (publishedVersion)
- Funders
- National Natural Science Foundation of China (61261160501, 61373026, 61622403)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Computer Science › Hardware and Architecture › Parallel Computing and Optimization Techniques
- Physical Sciences › Computer Science › Hardware and Architecture › Embedded Systems Design Techniques
- Keywords
- speech recognition, LSTM, model compression, pruning, quantization, load balancing, parallel computing, hardware architecture, energy efficiency
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.