Compressing RNNs to Kilobyte Budget for IoT Devices Using Kronecker Products
The DOI matches its Crossref record; 37 of the 37 references it cites were found in a registry or an index.
Abstract
Micro-controllers (MCUs) make up most of the processors in the world with widespread applicability from automobile to medical devices. The Internet of Things promises to enable these resource-constrained MCUs with machine learning algorithms to provide always-on intelligence. Many Internet of Things applications consume time-series data that are naturally suitable for recurrent neural networks (RNNs) like LSTMs and GRUs. However, RNNs can be large and difficult to deploy on these devices, as they have few kilobytes of memory. As a result, there is a need for compression techniques that can significantly compress RNNs without negatively impacting task accuracy. This article introduces a method to compress RNNs for resource-constrained environments using the Kronecker product (KP). KPs can compress RNN layers by 16× to 38× with minimal accuracy loss. By quantizing the resulting models to 8 bits, we further push the compression factor to 50×. We compare KP with other state-of-the-art compression techniques across seven benchmarks spanning five different applications and show that KP can beat the task accuracy achieved by other techniques by a large margin while simultaneously improving the inference runtime. Sometimes the KP compression mechanism can introduce an accuracy loss. We develop a hybrid KP approach to mitigate this. Our hybrid KP algorithm provides fine-grained control over the compression ratio, enabling us to regain accuracy lost during compression by adding a small…
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Citations
Cites
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318
Gradient-based learning applied to document recognition
Lecun, Y., Bottou, L., Bengio, Y., & Haffner, P. (1998) · Proceedings of the IEEE · doi:10.1109/5.726791 · Cited by 58,805
Bidirectional recurrent neural networks
Schuster, M., & Paliwal, K. K. (1997) · IEEE Transactions on Signal Processing · doi:10.1109/78.650093 · Cited by 10,172
Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation
Cho, K., van Merrienboer, B., Gulcehre, C., Bahdanau, D., Bougares, F., Schwenk, H., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1406.1078 · Cited by 3,317
Trefethen, L. N., & Bau, III (1997) · doi:10.1137/1.9780898719574 · Cited by 3,301
(2008) · Choice Reviews Online · doi:10.5860/choice.45-5628 · Cited by 2,725
A database for handwritten text recognition research
Hull, J. J. (1994) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/34.291440 · Cited by 2,244
Recurrent Neural Network Regularization
Zaremba, W., Sutskever, I., & Vinyals, O. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1409.2329 · Cited by 2,123
Speech Commands: A Dataset for Limited-Vocabulary Speech Recognition
Warden, P. (2018) · arXiv · doi:10.48550/arxiv.1804.03209 · Cited by 1,033
Predicting Parameters in Deep Learning
Denil, M., Shakibi, B., Dinh, L., Ranzato, M., & de Freitas, N. (2013) · arXiv · doi:10.48550/arxiv.1306.0543 · Cited by 853
To prune, or not to prune: exploring the efficacy of pruning for model compression
Zhu, M., & Gupta, S. (2017) · arXiv · doi:10.48550/arxiv.1710.01878 · Cited by 805
Collecting complex activity datasets in highly rich networked sensor environments
Roggen, D., Calatroni, A., Rossi, M., Holleczek, T., Forster, K., Troster, G., Lukowicz, P., Bannach, D., Pirkl, G., Ferscha, A., et al. (2010) · 2010 Seventh International Conference on Networked Sensing Systems (INSS) · doi:10.1109/inss.2010.5573462 · Cited by 788
Understanding the exploding gradient problemPreprint
Pascanu, R., Mikolov, T., & Bengio, Y. (2012) · arXiv (Cornell University) · Cited by 575
BinaryNet: Training Deep Neural Networks with Weights and Activations Constrained to +1 or -1Preprint
Courbariaux, M., & Bengio, Y. (2016) · arXiv (Cornell University) · Cited by 484
The State of Sparsity in Deep Neural Networks
Gale, T., Elsen, E., & Hooker, S. (2019) · arXiv · doi:10.48550/arxiv.1902.09574 · Cited by 440
An Exploration of Parameter Redundancy in Deep Networks with Circulant Projections
Cheng, Y., Yu, F. X., Feris, R. S., Kumar, S., Choudhary, A., & Chang, S.-F. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.327 · Cited by 303
Speeding-up Convolutional Neural Networks Using Fine-tuned CP-Decomposition
Lebedev, V., Ganin, Y., Rakhuba, M., Oseledets, I. V., & Lempitsky, V. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1412.6553 · Cited by 283
C-LSTM: Enabling Efficient LSTM using Structured Compression Techniques on FPGAs
Wang, S., Li, Z., Ding, C., Yuan, B., Qiu, Q., Wang, Y., & Liang, Y. (2018) · Proceedings of the 2018 ACM/SIGDA International Symposium on Field-Programmable Gate Arrays · doi:10.1145/3174243.3174253 · Cited by 220
Neural Networks with Few Multiplications
Lin, Z., Courbariaux, M., Memisevic, R., & Bengio, Y. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1510.03009 · Cited by 141
Structured Bayesian Pruning via Log-Normal Multiplicative Noise
Neklyudov, K., Molchanov, D., Ashukha, A., & Vetrov, D. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1705.07283 · Cited by 122
Factorization tricks for LSTM networks
Kuchaiev, O., & Ginsburg, B. (2017) · arXiv · doi:10.48550/arxiv.1703.10722 · Cited by 99
HubaraItay, CourbariauxMatthieu, SoudryDaniel, El-YanivRan, & BengioYoshua (2017) · Journal of Machine Learning Research · doi:10.5555/3122009.3242044 · Cited by 93
FastGRNN: A Fast, Accurate, Stable and Tiny Kilobyte Sized Gated Recurrent Neural Network
Kusupati, A., Singh, M., Bhatia, K., Kumar, A., Jain, P., & Varma, M. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1901.02358 · Cited by 91
The Power of Sparsity in Convolutional Neural Networks
Changpinyo, S., Sandler, M., & Zhmoginov, A. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1702.06257 · Cited by 80
Kronecker Recurrent UnitsPreprint
Jose, C., Cisse, M., & Fleuret, F. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1705.10142 · Cited by 47
Fast Orthogonal Projection Based on Kronecker Product
Zhang, X., Yu, F. X., Guo, R., Kumar, S., Wang, S., & Chang, S.-F. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.335 · Cited by 43
Structured Weight Matrices-Based Hardware Accelerators in Deep Neural Networks: FPGAs and ASICs
Ding, C., Ren, A., Yuan, G., Ma, X., Li, J., Liu, N., Yuan, B., & Wang, Y. (2018) · ACM Great Lakes Symposium on VLSI (GLSVLSI) · doi:10.1145/3194554.3194625 · Cited by 28
Run-Time Efficient RNN Compression for Inference on Edge Devices
Thakker, U., Beu, J., Gope, D., Dasika, G., & Mattina, M. (2019) · 2019 2nd Workshop on Energy Efficient Machine Learning and Cognitive Computing for Embedded Applications (EMC2) · doi:10.1109/emc249363.2019.00013 · Cited by 24
Self-citationSkipping RNN State Updates without Retraining the Original Model
Tao, J., Thakker, U., Dasika, G., & Beu, J. (2019) · Proceedings of the 1st Workshop on Machine Learning on Edge in Sensor Systems · doi:10.1145/3362743.3362965 · Cited by 19
Self-citationTernary MobileNets via Per-Layer Hybrid Filter Banks
Gope, D., Beu, J., Thakker, U., & Mattina, M. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw50498.2020.00362 · Cited by 15
Self-citationRank and run-time aware compression of NLP Applications
Thakker, U., Beu, J. G., Gope, D., Dasika, G. S., & Mattina, M. (2020) · Workshop on Simple and Efficient Natural Language Processing (SustaiNLP) · doi:10.18653/v1/2020.sustainlp-1.2 · Cited by 14
Self-citationMeasuring scheduling efficiency of RNNs for NLP applications
Thakker, U., Dasika, G. S., Beu, J. G., & Mattina, M. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1904.03302 · Cited by 13
Self-citationCompression of Fully-Connected Layer in Neural Network by Kronecker ProductPreprint
Zhou, S., & Wu, J. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1507.05775 · Cited by 9
ADMMiRNN: Training RNN with Stable Convergence via an Efficient ADMM Approach
Tang, Y. H., Kan, Z., Sun, D., Qiao, L., Xiao, J., Lai, Z., & Li, D. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-67661-2_1 · Cited by 9
Compressing Language Models using Doped Kronecker Products
Thakker, U., Whatmough, P. N., Liu, Z.-G., Mattina, M., & Beu, J. (2020) · arXiv · doi:10.48550/arxiv.2001.08896 · Cited by 7
Self-citationPushing the Envelope of Dynamic Spatial Gating technologies
Huang, X., Thakker, U., Gope, D., & Beu, J. (2020) · Proceedings of the 2nd International Workshop on Challenges in Artificial Intelligence and Machine Learning for Internet of Things · doi:10.1145/3417313.3429380 · Cited by 7
Self-citationUnderstanding the Impact of Dynamic Channel Pruning on Conditionally Parameterized Convolutions
Raju, R., Gope, D., Thakker, U., & Beu, J. G. (2020) · International Workshop on Challenges in Artificial Intelligence and Machine Learning for Internet of Things (AIChallengeIoT) · doi:10.1145/3417313.3429381 · Cited by 7
Self-citation
Research works printed but not linked: 25
- Martín Abadi , Ashish Agarwal , Paul Barham , Eugene Brevdo , Zhifeng Chen , Craig Citro , Greg S. Corrado , et al . 2015 . TensorFlow: Large- Scale Machine Learning on Heterogeneous Systems. Retrieved June 1, 2021 from https://www.tensorflow.org/. (Software available from tensorflow.org.) Martín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S. Corrado, et al. 2015. TensorFlow: Large-Scale Machine Learning on Heterogeneous Systems. Retrieved June 1, 2021 from https://www.tensorflow.org/. (Software available from tensorflow.org.)
- Colby R. Banbury , Vijay Janapa Reddi , Max Lam, William Fu, Amin Fazel, Jeremy Holleman, Xinyuan Huang, et al. 2020 . Benchmarking TinyML systems: Challenges and direction. arxiv:cs.PF/2003.04821. Colby R. Banbury, Vijay Janapa Reddi, Max Lam, William Fu, Amin Fazel, Jeremy Holleman, Xinyuan Huang, et al. 2020. Benchmarking TinyML systems: Challenges and direction. arxiv:cs.PF/2003.04821.
- Giuseppe Giovanni Calvi , Ahmad Moniri , Mahmoud Mahfouz , Zeyang Yu , Qibin Zhao , and Danilo P . Mandic . 2019 . Tucker tensor layer in fully connected neural networks. arxiv:1903.06133. Giuseppe Giovanni Calvi, Ahmad Moniri, Mahmoud Mahfouz, Zeyang Yu, Qibin Zhao, and Danilo P. Mandic. 2019. Tucker tensor layer in fully connected neural networks. arxiv:1903.06133. · arXiv
- 10.5555/3294771.3294792 · doi:10.5555/3294771.3294792
- 10.5555/2999792.2999852 · doi:10.5555/2999792.2999852
- 10.5555/3086952 · doi:10.5555/3086952
- Gael Guennebau and Benoit Jacob . 2009 . Eigen Library . Retrieved December 21, 2018 from http://eigen.tuxfamily.org/. Gael Guennebau and Benoit Jacob. 2009. Eigen Library. Retrieved December 21, 2018 from http://eigen.tuxfamily.org/.
- 10.5555/3060832.3060835 · doi:10.5555/3060832.3060835
- Song Han , Huizi Mao , and William J. Dally . 2016. Deep compression: Compressing deep neural networks with pruning, trained quantization and Huffman coding . In Proceedings of the International Conference on Learning Representations (ICLR’16) . Song Han, Huizi Mao, and William J. Dally. 2016. Deep compression: Compressing deep neural networks with pruning, trained quantization and Huffman coding. In Proceedings of the International Conference on Learning Representations (ICLR’16).
- Qinyao He He Wen Shuchang Zhou Yuxin Wu Cong Yao Xinyu Zhou and Yuheng Zou. 2016. Effective quantization methods for recurrent neural networks. arxiv:1611.10176. Qinyao He He Wen Shuchang Zhou Yuxin Wu Cong Yao Xinyu Zhou and Yuheng Zou. 2016. Effective quantization methods for recurrent neural networks. arxiv:1611.10176. · arXiv
- Qinyao He He Wen Shuchang Zhou Yuxin Wu Cong Yao Xinyu Zhou and Yuheng Zou. 2016. Effective quantization methods for recurrent neural networks. arxiv:1611.10176. Qinyao He He Wen Shuchang Zhou Yuxin Wu Cong Yao Xinyu Zhou and Yuheng Zou. 2016. Effective quantization methods for recurrent neural networks. arxiv:1611.10176. · arXiv
- 10.5555/3122009.3242044 · doi:10.5555/3122009.3242044 · Paper page
- 10.5555/3305381.3305581 · doi:10.5555/3305381.3305581
- 10.5555/3327546.3327577 · doi:10.5555/3327546.3327577
- 10.5555/1062366 · doi:10.5555/1062366
- Zhouhan Lin Matthieu Courbariaux Roland Memisevic and Yoshua Bengio. 2015. Neural networks with few multiplications. arxiv:1510.03009. https://doi.org/10.1109/5.726791 · DOI · Paper page
- Christos Louizos , Max Welling , and Diederik P . Kingma . 2017 . Learning sparse neural networks through regularization. arxiv:1712.01312. Christos Louizos, Max Welling, and Diederik P. Kingma. 2017. Learning sparse neural networks through regularization. arxiv:1712.01312. · arXiv
- Sharan Narang , Eric Undersander , and Gregory F . Diamos . 2017 . Block-sparse recurrent neural networks. arxiv:1711.02782. Sharan Narang, Eric Undersander, and Gregory F. Diamos. 2017. Block-sparse recurrent neural networks. arxiv:1711.02782. · arXiv
- 10.5555/3295222.3295422 · doi:10.5555/3295222.3295422
- 10.5555/2969442.2969584 · doi:10.5555/2969442.2969584
- 10.5555/3327546.3327580 · doi:10.5555/3327546.3327580
- Vincent Vanhoucke , Andrew Senior , and Mark Z. Mao . 2011. Improving the speed of neural networks on CPUs . In Proceedings of theDeep Learning and Unsupervised Feature Learning Workshop (NIPS’11) . Vincent Vanhoucke, Andrew Senior, and Mark Z. Mao. 2011. Improving the speed of neural networks on CPUs. In Proceedings of theDeep Learning and Unsupervised Feature Learning Workshop (NIPS’11).
- 10.5555/3157382.3157643 · doi:10.5555/3157382.3157643
- Jiong Zhang , Qi Lei , and Inderjit S . Dhillon . 2018 . Stabilizing gradients for deep neural networks via efficient SVD parameterization. arxiv:1803.09327. Jiong Zhang, Qi Lei, and Inderjit S. Dhillon. 2018. Stabilizing gradients for deep neural networks via efficient SVD parameterization. arxiv:1803.09327. · arXiv
- Yundong Zhang , Naveen Suda , Liangzhen Lai , and Vikas Chandra . 2017 . Hello Edge: Keyword spotting on microcontrollers. arxiv:1711.07128. Yundong Zhang, Naveen Suda, Liangzhen Lai, and Vikas Chandra. 2017. Hello Edge: Keyword spotting on microcontrollers. arxiv:1711.07128. · arXiv
Cited by
Machine Learning for Microcontroller-Class Hardware: A Review
Saha, S. S., Sandha, S. S., & Srivastava, M. (2022) · IEEE Sensors Journal · doi:10.1109/jsen.2022.3210773 · Cited by 253
Ciliberto, M., Fortes Rey, V., Calatroni, A., Lukowicz, P., & Roggen, D. (2021) · Frontiers in Computer Science · doi:10.3389/fcomp.2021.792065 · Cited by 35
“…Roggen (2016); Morales and Roggen (2016), and it has been used recently in fi elds as varied as machine learning model compression for embedded systems Thakker et al. (2021), transfer learning research Kalabakov et al. (2021), ontology-based activity recognition Noor et al. (2018), unsupervised…”
Metadata
- Authors
- Thakker, U., Fedorov, I., Zhou, C., Gope, D., Mattina, M., Dasika, G., & Beu, J.
- Affiliations
- Arm ML Research Lab, Austin, TX
- AMD Research, Austin, TX
- Arm ML Research Lab, Boston, MA
- Venue
- ACM Journal on Emerging Technologies in Computing Systems
- Published
- 2021-07-14
- Volume
- 17
- Issue
- 4
- Pages
- 1-18
- ISSN
- 1550-4832 (ISSN-L)
- 1550-4840
- OpenAlex
- W3180864037
- Cited by
- 8
- References
- 37
- Citations by year
- 2025: 2 · 2024: 2 · 2023: 2 · 2022: 1 · 2021: 1
- FWCI
- 0.54
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- ACM Journal on Emerging Technologies in Computing Systems (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Computer Science › Computer Networks and Communications › IoT and Edge/Fog Computing
- Physical Sciences › Computer Science › Artificial Intelligence › Neural Networks and Applications
- Keywords
- Kronecker product, Internet of Things, microcontroller, resource-constrained devices, 8-bit quantization, LSTM, GRU, hybrid compression, inference runtime, model size reduction, time series forecasting
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.