The DOI matches its Crossref record; 37 of the 37 references it cites were found in a registry or an index.
Compressing RNNs to Kilobyte Budget for IoT Devices Using Kronecker Products
Abstract
Micro-controllers (MCUs) make up most of the processors in the world with widespread applicability from automobile to medical devices. The Internet of Things promises to enable these resource-constrained MCUs with machine learning algorithms to provide always-on intelligence. Many Internet of Things applications consume time-series data that are naturally suitable for recurrent neural networks (RNNs) like LSTMs and GRUs. However, RNNs can be large and difficult to deploy on these devices, as they have few kilobytes of memory. As a result, there is a need for compression techniques that can significantly compress RNNs without negatively impacting task accuracy. This article introduces a method to compress RNNs for resource-constrained environments using the Kronecker product (KP). KPs can compress RNN layers by 16× to 38× with minimal accuracy loss. By quantizing the resulting models to 8 bits, we further push the compression factor to 50×. We compare KP with other state-of-the-art compression techniques across seven benchmarks spanning five different applications and show that KP can beat the task accuracy achieved by other techniques by a large margin while simultaneously improving the inference runtime. Sometimes the KP compression mechanism can introduce an accuracy loss. We develop a hybrid KP approach to mitigate this. Our hybrid KP algorithm provides fine-grained control over the compression ratio, enabling us to regain accuracy lost during compression by adding a small…
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Citations
Cites
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318
Gradient-based learning applied to document recognition
Lecun, Y., Bottou, L., Bengio, Y., & Haffner, P. (1998) · Proceedings of the IEEE · doi:10.1109/5.726791 · Cited by 58,805
Bidirectional recurrent neural networks
Schuster, M., & Paliwal, K. K. (1997) · IEEE Transactions on Signal Processing · doi:10.1109/78.650093 · Cited by 10,172
Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation
Cho, K., van Merrienboer, B., Gulcehre, C., Bahdanau, D., Bougares, F., Schwenk, H., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1406.1078 · Cited by 3,317
Trefethen, L. N., & Bau, III (1997) · doi:10.1137/1.9780898719574 · Cited by 3,301
(2008) · Choice Reviews Online · doi:10.5860/choice.45-5628 · Cited by 2,725
A database for handwritten text recognition research
Hull, J. J. (1994) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/34.291440 · Cited by 2,244
Recurrent Neural Network Regularization
Zaremba, W., Sutskever, I., & Vinyals, O. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1409.2329 · Cited by 2,123
Speech Commands: A Dataset for Limited-Vocabulary Speech Recognition
Warden, P. (2018) · arXiv · doi:10.48550/arxiv.1804.03209 · Cited by 1,033
Predicting Parameters in Deep Learning
Denil, M., Shakibi, B., Dinh, L., Ranzato, M., & de Freitas, N. (2013) · arXiv · doi:10.48550/arxiv.1306.0543 · Cited by 853
Research works printed but not linked: 25
- Martín Abadi , Ashish Agarwal , Paul Barham , Eugene Brevdo , Zhifeng Chen , Craig Citro , Greg S. Corrado , et al . 2015 . TensorFlow: Large- Scale Machine Learning on Heterogeneous Systems. Retrieved June 1, 2021 from https://www.tensorflow.org/. (Software available from tensorflow.org.) Martín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S. Corrado, et al. 2015. TensorFlow: Large-Scale Machine Learning on Heterogeneous Systems. Retrieved June 1, 2021 from https://www.tensorflow.org/. (Software available from tensorflow.org.)
- Colby R. Banbury , Vijay Janapa Reddi , Max Lam, William Fu, Amin Fazel, Jeremy Holleman, Xinyuan Huang, et al. 2020 . Benchmarking TinyML systems: Challenges and direction. arxiv:cs.PF/2003.04821. Colby R. Banbury, Vijay Janapa Reddi, Max Lam, William Fu, Amin Fazel, Jeremy Holleman, Xinyuan Huang, et al. 2020. Benchmarking TinyML systems: Challenges and direction. arxiv:cs.PF/2003.04821.
- Giuseppe Giovanni Calvi , Ahmad Moniri , Mahmoud Mahfouz , Zeyang Yu , Qibin Zhao , and Danilo P . Mandic . 2019 . Tucker tensor layer in fully connected neural networks. arxiv:1903.06133. Giuseppe Giovanni Calvi, Ahmad Moniri, Mahmoud Mahfouz, Zeyang Yu, Qibin Zhao, and Danilo P. Mandic. 2019. Tucker tensor layer in fully connected neural networks. arxiv:1903.06133. · arXiv
- 10.5555/3294771.3294792 · doi:10.5555/3294771.3294792
- 10.5555/2999792.2999852 · doi:10.5555/2999792.2999852
- 10.5555/3086952 · doi:10.5555/3086952
- Gael Guennebau and Benoit Jacob . 2009 . Eigen Library . Retrieved December 21, 2018 from http://eigen.tuxfamily.org/. Gael Guennebau and Benoit Jacob. 2009. Eigen Library. Retrieved December 21, 2018 from http://eigen.tuxfamily.org/.
- 10.5555/3060832.3060835 · doi:10.5555/3060832.3060835
- Song Han , Huizi Mao , and William J. Dally . 2016. Deep compression: Compressing deep neural networks with pruning, trained quantization and Huffman coding . In Proceedings of the International Conference on Learning Representations (ICLR’16) . Song Han, Huizi Mao, and William J. Dally. 2016. Deep compression: Compressing deep neural networks with pruning, trained quantization and Huffman coding. In Proceedings of the International Conference on Learning Representations (ICLR’16).
- Qinyao He He Wen Shuchang Zhou Yuxin Wu Cong Yao Xinyu Zhou and Yuheng Zou. 2016. Effective quantization methods for recurrent neural networks. arxiv:1611.10176. Qinyao He He Wen Shuchang Zhou Yuxin Wu Cong Yao Xinyu Zhou and Yuheng Zou. 2016. Effective quantization methods for recurrent neural networks. arxiv:1611.10176. · arXiv
- Qinyao He He Wen Shuchang Zhou Yuxin Wu Cong Yao Xinyu Zhou and Yuheng Zou. 2016. Effective quantization methods for recurrent neural networks. arxiv:1611.10176. Qinyao He He Wen Shuchang Zhou Yuxin Wu Cong Yao Xinyu Zhou and Yuheng Zou. 2016. Effective quantization methods for recurrent neural networks. arxiv:1611.10176. · arXiv
- 10.5555/3122009.3242044 · doi:10.5555/3122009.3242044
- 10.5555/3305381.3305581 · doi:10.5555/3305381.3305581
- 10.5555/3327546.3327577 · doi:10.5555/3327546.3327577
- 10.5555/1062366 · doi:10.5555/1062366
- Zhouhan Lin Matthieu Courbariaux Roland Memisevic and Yoshua Bengio. 2015. Neural networks with few multiplications. arxiv:1510.03009. https://doi.org/10.1109/5.726791 · DOI · Paper page
- Christos Louizos , Max Welling , and Diederik P . Kingma . 2017 . Learning sparse neural networks through regularization. arxiv:1712.01312. Christos Louizos, Max Welling, and Diederik P. Kingma. 2017. Learning sparse neural networks through regularization. arxiv:1712.01312. · arXiv
- Sharan Narang , Eric Undersander , and Gregory F . Diamos . 2017 . Block-sparse recurrent neural networks. arxiv:1711.02782. Sharan Narang, Eric Undersander, and Gregory F. Diamos. 2017. Block-sparse recurrent neural networks. arxiv:1711.02782. · arXiv
- 10.5555/3295222.3295422 · doi:10.5555/3295222.3295422
- 10.5555/2969442.2969584 · doi:10.5555/2969442.2969584
- 10.5555/3327546.3327580 · doi:10.5555/3327546.3327580
- Vincent Vanhoucke , Andrew Senior , and Mark Z. Mao . 2011. Improving the speed of neural networks on CPUs . In Proceedings of theDeep Learning and Unsupervised Feature Learning Workshop (NIPS’11) . Vincent Vanhoucke, Andrew Senior, and Mark Z. Mao. 2011. Improving the speed of neural networks on CPUs. In Proceedings of theDeep Learning and Unsupervised Feature Learning Workshop (NIPS’11).
- 10.5555/3157382.3157643 · doi:10.5555/3157382.3157643
- Jiong Zhang , Qi Lei , and Inderjit S . Dhillon . 2018 . Stabilizing gradients for deep neural networks via efficient SVD parameterization. arxiv:1803.09327. Jiong Zhang, Qi Lei, and Inderjit S. Dhillon. 2018. Stabilizing gradients for deep neural networks via efficient SVD parameterization. arxiv:1803.09327. · arXiv
- Yundong Zhang , Naveen Suda , Liangzhen Lai , and Vikas Chandra . 2017 . Hello Edge: Keyword spotting on microcontrollers. arxiv:1711.07128. Yundong Zhang, Naveen Suda, Liangzhen Lai, and Vikas Chandra. 2017. Hello Edge: Keyword spotting on microcontrollers. arxiv:1711.07128. · arXiv
Cited by
Machine Learning for Microcontroller-Class Hardware: A Review
Saha, S. S., Sandha, S. S., & Srivastava, M. (2022) · IEEE Sensors Journal · doi:10.1109/jsen.2022.3210773 · Cited by 253
Ciliberto, M., Fortes Rey, V., Calatroni, A., Lukowicz, P., & Roggen, D. (2021) · Frontiers in Computer Science · doi:10.3389/fcomp.2021.792065 · Cited by 35
“…Roggen (2016); Morales and Roggen (2016), and it has been used recently in fi elds as varied as machine learning model compression for embedded systems Thakker et al. (2021), transfer learning research Kalabakov et al. (2021), ontology-based activity recognition Noor et al. (2018), unsupervised…”
Metadata
- Authors
- Thakker, U., Fedorov, I., Zhou, C., Gope, D., Mattina, M., Dasika, G., & Beu, J.
- Affiliations
- Arm ML Research Lab, Austin, TX
- AMD Research, Austin, TX
- Arm ML Research Lab, Boston, MA
- Venue
- ACM Journal on Emerging Technologies in Computing Systems
- Published
- 2021-07-14
- Volume
- 17
- Issue
- 4
- Pages
- 1-18
- ISSN
- 1550-4832 (ISSN-L)
- 1550-4840
- OpenAlex
- W3180864037
- Cited by
- 8
- References
- 37
- Citations by year
- 2025: 2 · 2024: 2 · 2023: 2 · 2022: 1 · 2021: 1
- FWCI
- 0.54
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- ACM Journal on Emerging Technologies in Computing Systems (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Computer Science › Computer Networks and Communications › IoT and Edge/Fog Computing
- Physical Sciences › Computer Science › Artificial Intelligence › Neural Networks and Applications
- Keywords
- Kronecker product, Internet of Things, microcontroller, resource-constrained devices, 8-bit quantization, LSTM, GRU, hybrid compression, inference runtime, model size reduction, time series forecasting
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Compressing RNNs to Kilobyte Budget for IoT Devices Using Kronecker Products. https://app.cochat.ai/papers/10.1145/3440016
- The paper
- Thakker, U., Fedorov, I., Zhou, C., Gope, D., Mattina, M., Dasika, G., & Beu, J. (2021). Compressing RNNs to Kilobyte Budget for IoT Devices Using Kronecker Products. ACM Journal on Emerging Technologies in Computing Systems. https://doi.org/10.1145/3440016
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.