Random sketch learning for deep neural networks in edge computing
Open accessAbstract
Despite the great potential of deep neural networks (DNNs), they require massive weights and huge computational resources, creating a vast gap when deploying artificial intelligence at low-cost edge devices. Current lightweight DNNs, achieved by high-dimensional space pre-training and post-compression, present challenges when covering the resources deficit, making tiny artificial intelligence hard to be implemented. Here we report an architecture named random sketch learning, or Rosler, for computationally efficient tiny artificial intelligence. We build a universal compressing-while-training framework that directly learns a compact model and, most importantly, enables computationally efficient on-device learning. As validated on different models and datasets, it attains substantial memory reduction of ~50-90× (16-bits quantization), compared with fully connected DNNs. We demonstrate it on low-cost hardware, whereby the computation is accelerated by >180× and the energy consumption is reduced by ~10×. Our method paves the way for deploying tiny artificial intelligence in many scientific and industrial applications.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Li et al. established an architecture named random sketch learning for deep neural networks in edge computing, which demonstrated substantial memory reduction of ~50-90× (16-bits quantization), computation acceleration of 180× and energy consumption reduction of ~10× compared with fully connected deep neural networks.
“Li et al. established an architecture named random sketch learning for deep neural networks in edge computing. 225 This method demonstrated substantial memory reduction of ~50-90× (16-bits quantization), computation acceleration of 180× and energy consumption reduction of ~10× compared with fully connected deep neural networks”
From Energy system and resource utilization in space: A state-of-the-art review (2024) · cites this paper as [225]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946 · see citing works
LeCun, Y., Bengio, Y., & Hinton, G. (2015) · Nature · doi:10.1038/nature14539 · Cited by 84,126 · see citing works
Mastering the game of Go with deep neural networks and tree search
Silver, D., Huang, A., Maddison, C. J., Guez, A., Sifre, L., van den Driessche, G., Schrittwieser, J., Antonoglou, I., Panneershelvam, V., Lanctot, M., et al. (2016) · Nature · doi:10.1038/nature16961 · Cited by 15,612 · see citing works
Communication-Efficient Learning of Deep Networks from Decentralized Data
McMahan, H. B., Moore, E., Ramage, D., Hampson, S., & Arcas, B. A. Y. (2016) · arXiv · doi:10.48550/arxiv.1602.05629 · Cited by 14,215 · see citing works
(2010) · Cited by 14,134 · see citing works
Federated Machine Learning: Concept and Applications
Yang, Q., Liu, Y., Chen, T., & Tong, Y. (2019) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3298981 · Cited by 6,322 · see citing works
Deep learning and process understanding for data-driven Earth system science
Reichstein, M., Camps-Valls, G., Stevens, B., Jung, M., Denzler, J., Carvalhais, N., & Prabhat (2019) · Nature · doi:10.1038/s41586-019-0912-1 · Cited by 5,673 · see citing works
Han, S., Mao, H., & Dally, W. J. (2015) · arXiv · doi:10.48550/arxiv.1510.00149 · Cited by 4,662 · see citing works
Efficient Processing of Deep Neural Networks: A Tutorial and Survey
Sze, V., Chen, Y.-H., Yang, T.-J., & Emer, J. S. (2017) · Proceedings of the IEEE · doi:10.1109/jproc.2017.2761740 · Cited by 4,215 · see citing works
Training and operation of an integrated neuromorphic network based on metal-oxide memristors
Prezioso, M., Merrikh-Bayat, F., Hoskins, B. D., Adam, G. C., Likharev, K. K., & Strukov, D. B. (2015) · Nature · doi:10.1038/nature14441 · Cited by 3,001 · see citing works
Smith, W. A., & Randall, R. B. (2015) · Mechanical Systems and Signal Processing · doi:10.1016/j.ymssp.2015.04.021 · Cited by 2,734 · see citing works
Deep reinforcement learning for autonomous driving: A survey
Kiran, B. R., Sobh, I., Talpaert, V., Mannion, P., Sallab, A. A. A., Yogamani, S., & Perez, P. (2022) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2021.3054625 · Cited by 2,342 · see citing works
Learning Structured Sparsity in Deep Neural NetworksPreprint
Wen, W. X., Wu, C., Wang, Y., Chen, Y., & Li, H. H. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1608.03665 · Cited by 1,861 · see citing works
Deep learning for smart manufacturing: Methods and applications
Wang, J., Ma, Y., Zhang, L., Gao, R. X., & Wu, D. (2018) · Journal of Manufacturing Systems · doi:10.1016/j.jmsy.2018.01.003 · Cited by 1,740 · see citing works
The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks
Frankle, J., & Carbin, M. (2018) · arXiv · doi:10.48550/arxiv.1803.03635 · Cited by 1,529 · see citing works
Burger, B., Maffettone, P. M., Gusev, V. V., Aitchison, C. M., Bai, Y., Wang, X., Li, X., Alston, B. M., Li, B., Clowes, R., et al. (2020) · Nature · doi:10.1038/s41586-020-2442-2 · Cited by 1,516 · see citing works
Self-citationLearning both Weights and Connections for Efficient Neural Networks
Han, S., Pool, J., Tran, J., & Dally, W. J. (2015) · arXiv · doi:10.48550/arxiv.1506.02626 · Cited by 1,481 · see citing works
Analogue signal and image processing with large memristor crossbars
Li, C., Hu, M., Li, Y., Jiang, H., Ge, N., Montgomery, E., Zhang, J., Song, W., Dávila, N., Graves, C. E., et al. (2017) · Nature Electronics · doi:10.1038/s41928-017-0002-z · Cited by 1,292 · see citing works
Self-citationSpeeding up Convolutional Neural Networks with Low Rank Expansions
Jaderberg, M., Vedaldi, A., & Zisserman, A. (2014) · Proceedings of the British Machine Vision Conference 2014 · doi:10.5244/c.28.88 · Cited by 1,164 · see citing works
Federated learning of predictive models from federated Electronic Health Records
Brisimi, T. S., Chen, R., Mela, T., Olshevsky, A., Paschalidis, I. C., & Shi, W. (2018) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2018.01.007 · Cited by 997 · see citing works
Predicting Parameters in Deep Learning
Denil, M., Shakibi, B., Dinh, L., Ranzato, M., & de Freitas, N. (2013) · arXiv · doi:10.48550/arxiv.1306.0543 · Cited by 853 · see citing works
SNIP: Single-shot Network Pruning based on Connection Sensitivity
Lee, N., Ajanthan, T., & Torr, P. H. S. (2018) · arXiv · doi:10.48550/arxiv.1810.02340 · Cited by 685 · see citing works
Federated Machine Learning: Concept and Applications
Yang, Q., Liu, Y., Chen, T., & Tong, Y. (2019) · arXiv · doi:10.48550/arxiv.1902.04885 · Cited by 600 · see citing works
Lu, Y., Huang, X., Zhang, K., Maharjan, S., & Zhang, Y. (2020) · IEEE Transactions on Industrial Informatics · doi:10.1109/tii.2020.3017668 · Cited by 500 · see citing works
Relative-Error $CUR$ Matrix Decompositions
Drineas, P., Mahoney, M. W., & Muthukrishnan, S. (2008) · SIAM Journal on Matrix Analysis and Applications · doi:10.1137/07070471x · Cited by 469 · see citing works
Practical Secure Aggregation for Federated Learning on User-Held DataPreprint
Bonawitz, K., Ivanov, V., Kreuter, B., Marcedone, A., McMahan, H. B., Patel, S., Ramage, D., Segal, A., & Seth, K. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1611.04482 · Cited by 468 · see citing works
Scaling for edge inference of deep neural networks
Xu, X., Ding, Y., Hu, S. X., Niemier, M., Cong, J., Hu, Y., & Shi, Y. (2018) · Nature Electronics · doi:10.1038/s41928-018-0059-3 · Cited by 462 · see citing works
Gao, M., Pu, J., Yang, X., Horowitz, M., & Kozyrakis, C. (2017) · International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS) · doi:10.1145/3037697.3037702 · Cited by 421 · see citing works
On Compressing Deep Models by Low Rank and Sparse Decomposition
Yu, X., Liu, T., Wang, X., & Tao, D. (2017) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2017.15 · Cited by 419 · see citing works
Silva, S., Gutman, B. A., Romero, E., Thompson, P. M., Altmann, A., & Lorenzi, M. (2019) · IEEE International Symposium on Biomedical Imaging (ISBI) · doi:10.1109/isbi.2019.8759317 · Cited by 212 · see citing works
Network Pruning via Transformable Architecture SearchPreprint
Dong, X., & Yang, Y. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1905.09717 · Cited by 177 · see citing works
LogNet: Energy-efficient neural networks using logarithmic computation
Lee, E. H., Miyashita, D., Chai, E., Murmann, B., & Wong, S. S. (2017) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2017.7953288 · Cited by 150 · see citing works
Improving CUR Matrix Decomposition and the Nyström Approximation via Adaptive SamplingPreprint
Wang, S., & Zhang, Z. (2013) · arXiv (Cornell University) · doi:10.48550/arxiv.1303.4207 · Cited by 141 · see citing works
Lym, S., Choukse, E., Zangeneh, S., Wen, W., Sanghavi, S., & Erez, M. (2019) · International Conference for High Performance Computing, Networking, Storage and Analysis (SC) · doi:10.1145/3295500.3356156 · Cited by 75 · see citing works
Multi-View Fusion with Extreme Learning Machine for Clustering
Zhang, Y., Wu, J. X., Zhou, C., Cai, Z., Yang, J. Y., & Yu, P. S. (2019) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3340268 · Cited by 71 · see citing works
NAT: Neural Architecture Transformer for Accurate and Compact ArchitecturesPreprint
Guo, Y. F., Zheng, Y., Tan, M., Chen, Q., Chen, J., Zhao, P., & Huang, J. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1910.14488 · Cited by 67 · see citing works
Gao, M., Pu, J., Yang, X., et al. (2017) · ACM SIGARCH Computer Architecture News · doi:10.1145/3093337.3037702 · Cited by 60 · see citing works
Simons, F. J., Nolet, G., Georgieff, P., Babcock, J. M., Regier, L. A., & Davis, R. E. (2009) · Journal of Geophysical Research Atmospheres · doi:10.1029/2008jb006088 · Cited by 57 · see citing works
Measurement Science Roadmap for Prognostics and Health Management for Smart Manufacturing Systems
Pellegrino, J. L., Justiniano, M., Raghunathan, A., & Weiss, B. A. (2016) · doi:10.6028/nist.ams.100-2 · Cited by 39 · see citing works
Vaughan, O. P. H. (2019) · Nature Electronics · doi:10.1038/s41928-018-0198-6 · Cited by 16 · see citing works
Randomized Approximate Channel Estimator in Massive-MIMO Communication
Li, B., Wang, S., Zhang, J., et al. (2020) · IEEE Communications Letters · doi:10.1109/lcomm.2020.3001931 · Cited by 11 · see citing works
Self-citationRandom Sketch Learning for Tiny AI
Li, B., Liu, H., & Chen, P. (2021) · Code Ocean · doi:10.24433/co.5227764.v1 · Cited by 1 · see citing works
Self-citation
Flagged first, then by citations
Research works printed but not linked: 7
- [4][4] Park Jihong et al. 'Wireless network intelligence at the edge'. In: Proceedings of the IEEE 107.11 (2019), pp. 2204-2239. -
- [13][13] D. Hiroshi, M. Roberto, and Jan H ¨ o ller. 'Bringing Machine Learning to the Deepest IoT Edge with TinyML as-a-Service'. In: IEEE Internet of Things (IoT) Newsletter (May, 2020). -
- [21][21] Mingyu Gao et al. 'TETRIS: Scalable and Efficient Neural Network Acceleration with 3 D Memory'. In: in Proc. of the 22 International Conference on Architectural Support for Programming Languages and Operating Systems 45.1 (2017), pp. 751-764. -
- [32][32] Tianyi Zhou and Dacheng Tao. 'GoDec: Randomized Low-rank & Sparse Matrix Decomposition in Noisy Case'. In: in Prof. of International Conference on Machine Learning (ICML) (2011), pp. 33-40. -
- [37][37] Davis W Blalock et al. 'What is the State of Neural Network Pruning'. In: arXiv: Learning (2020). -
- [42][42] K. Simonyan and A. Zisserman. 'Very Deep Convolutional Networks for Large-Scale Image Recognition'. In: International Conference on Learning Representations (ICLR) . 2015. -
- [43][43] Sangkug Lym et al. 'PruneTrain: fast neural network training by dynamic sparse model reconfiguration'. In: In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis (2019), pp. 1-13. -
Cited by
Sorted by citations
A review on TinyML: State-of-the-art and prospects
Ray, P. P. (2022) · Journal of King Saud University - Computer and Information Sciences · doi:10.1016/j.jksuci.2021.11.019 · Cited by 460 · see citing works
“For instance, in (Li et al., 2021a), an architecture called Rosler is presented to showcase a compressing-while-training framework.”
Lead federated neuromorphic learning for wireless edge artificial intelligence
Yang, H., Lam, K.-Y., Xiao, L., Xiong, Z., Hu, H., Niyato, D., & Vincent Poor, H. (2022) · Nature Communications · doi:10.1038/s41467-022-32020-w · Cited by 127 · see citing works
Energy system and resource utilization in space: A state-of-the-art review
Wu, W., Shen, J., Kong, H., Yang, Y., Ren, E., Liu, Z., Wang, W., Dong, M., Han, L., Yang, C., et al. (2024) · The Innovation Energy · doi:10.59717/j.xinn-energy.2024.100029 · Cited by 44 · see citing works
Self-citationNakhle, F., & Harfouche, A. L. (2021) · Patterns · doi:10.1016/j.patter.2021.100323 · Cited by 28 · see citing works
“…generated models are compressed using compression techniques such as network pruning, low-rank approximation, weight quantization, and network architecture transform 127 to allow for their deployment in devices with low computational resources, without signifi-cant decrease in their performance.”
Method
- Authors
- Li, B., Chen, P., Liu, H., Guo, W., Cao, X., Du, J., Zhao, C., & Zhang, J.
- Affiliations
- Beijing Institute of Technology (CN)
- Beijing University of Posts and Telecommunications (CN)
- The Alan Turing Institute (GB)
- Beihang University (CN)
- China Electronics Corporation (China) (CN)
- Corresponding author
- Li, Bin
- Cao, Xianbin
- Venue
- Nature Computational Science
- Published
- 2021-03-25
- Volume
- 1
- Issue
- 3
- Pages
- 221-228
- ISSN
- 2662-8457 (ISSN-L)
- OpenAlex
- W3138547151
- Cited by
- 40
- References
- 42
- Citations by year
- 2026: 4 · 2025: 8 · 2024: 7 · 2023: 9 · 2022: 6 · 2021: 6
- FWCI
- 2.52
- Type
- journal article
- Language
- en
- Open access
- Licence
- Locations
- Nature Computational Science (publishedVersion)
- PubMed (publishedVersion)
- CERES (Cranfield University) (submittedVersion, cc-by-nc)
- Funders
- National Natural Science Foundation of China (91738301, U1805262)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Computer Science › Computer Networks and Communications › IoT and Edge/Fog Computing
- Physical Sciences › Computer Science › Artificial Intelligence › Neural Networks and Applications
- Keywords
- TinyML, edge computing, lightweight neural networks, model compression, on-device learning, memory reduction, energy efficiency, resource-constrained devices
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
