Enabling Large Neural Networks on Tiny Microcontrollers with Swapping
Open accessAbstract
Running neural networks (NNs) on microcontroller units (MCUs) is becoming increasingly important, but is very difficult due to the tiny SRAM size of MCU. Prior work proposes many algorithm-level techniques to reduce NN memory footprints, but all at the cost of sacrificing accuracy and generality, which disqualifies MCUs for many important use cases. We investigate a system solution for MCUs to execute NNs out of core: dynamically swapping NN data chunks between an MCU's tiny SRAM and its large, low-cost external flash. Out-of-core NNs on MCUs raise multiple concerns: execution slowdown, storage wear out, energy consumption, and data security. We present a study showing that none is a showstopper; the key benefit -- MCUs being able to run large NNs with full accuracy and generality -- triumphs the overheads. Our findings suggest that MCUs can play a much greater role in edge intelligence.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
SwapNN adds more energy efficiency to the IoT device by minimizing overall power drop for the microcontroller unit.
“SwapNN adds more energy efficiency to the IoT device by minimizing overall power drop for the microcontroller unit.”
From A review on TinyML: State-of-the-art and prospects (2022) · cites this paper as Miao and Lin, 2021
Section 5.5 (pp. 11–12) says swapping adds energy consumption to an MCU already performing computation. Its measured system energy rose by 42%, from 0.07 Wh to 0.10 Wh, rather than falling. The paper describes this as a modest overhead, not an energy-efficiency gain. (LLM, conf=0.88)
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredThe microcontroller can use Tile0 to compute the next Tile1 while swapping tiles between on-board memory and external flash.
“Microcontroller can then use this Tile0 to compute the next Tile1 while swapping the tiles between the on-board memory and the external flash.”
From A review on TinyML: State-of-the-art and prospects (2022) · cites this paper as Miao and Lin, 2021
Section 2.2 (p. 4) says the MCU can compute Tile1 while writing the completed Tile0 to external flash, and can preload Tile1’s inputs while computing Tile0. It does not say the MCU uses Tile0 to compute Tile1; it describes them as separate output tiles. (LLM, conf=0.88)
Experimental — accuracy not yet measuredThe SwapNN method hides swapping delays with improved parallelism at minute granular levels.
“For example, hiding swapping delays with improved parallelism at minute granular levels.”
From A review on TinyML: State-of-the-art and prospects (2022) · cites this paper as Miao and Lin, 2021
Section 3 (p. 5) says SwapNN schedules I/O and computation across tiles, layers, and frames in parallel to reduce swapping overhead. But the Introduction (p. 2) says that although small tiles provide finer-grained parallelism, their added I/O costs outweigh that benefit on typical hardware; it does not support fine granularity as the means of hiding delays. (LLM, conf=0.84)
Experimental — accuracy not yet measuredThe benefit of SwapNN is to compute both memory and IO bound layers in parallel for different frames.
“The benefit of this method is to compute both memory and IO bound layers in parallel for different frames.”
From A review on TinyML: State-of-the-art and prospects (2022) · cites this paper as Miao and Lin, 2021
Section 2.2 (p. 4) says SwapNN can execute compute-bound and IO-bound layers for different frames in parallel, giving the example of swapping one frame’s FC layer while computing another frame’s Conv layer. The claim says “memory-bound” rather than “compute-bound”; our copy does not establish that those terms are interchangeable. (LLM, conf=0.84)
Experimental — accuracy not yet measuredLarge tile size allows the IO-bound layers to increase delay.
“Moreover, the large tile size allows the IO-bound layers to increase delay.”
From A review on TinyML: State-of-the-art and prospects (2022) · cites this paper as Miao and Lin, 2021
Due to parallel execution, overall IO overhead is reduced in SwapNN.
“Due to the parallel execution, overall IO overhead is reduced.”
From A review on TinyML: State-of-the-art and prospects (2022) · cites this paper as Miao and Lin, 2021
The SwapNN process starts with extraction of CPU/IO parallelism to hide input/output delays.
“The process starts with extraction of CPU/IO parallelism to hide input/output delays.”
From A review on TinyML: State-of-the-art and prospects (2022) · cites this paper as Miao and Lin, 2021
SwapNN performs parallelism in the neural network layer to produce Tile0.
“Firstly, it performs parallelism in the neural network layer to produce Tile0.”
From A review on TinyML: State-of-the-art and prospects (2022) · cites this paper as Miao and Lin, 2021
Each tile can be loaded into a tiny memory space of an IoT device.
“Each of these tiles can be loaded into a tiny memory space of an IoT device.”
From A review on TinyML: State-of-the-art and prospects (2022) · cites this paper as Miao and Lin, 2021
SwapNN has very little impact on the external SD card in terms of durability loss and lifetime.
“This study shows that dong SwapNN has very less impact on the external SD card in terms of durability loss and lifetime.”
From A review on TinyML: State-of-the-art and prospects (2022) · cites this paper as Miao and Lin, 2021
The out-of-core neural network allows splitting one neural network layer's working into multiple series of tiles.
“The out-of-core neural network allows splitting one neural network layer's working into multiple series of tiles.”
From A review on TinyML: State-of-the-art and prospects (2022) · cites this paper as Miao and Lin, 2021
Miao and Lin (2021) presented a method to execute neural networks in IoT-centric microcontrollers by using swapping.
“In (Miao and Lin, 2021) a new method is presented to execute neural networks in IoT centric microcontrollers by using swapping.”
From A review on TinyML: State-of-the-art and prospects (2022) · cites this paper as Miao and Lin, 2021
In the swapping approach, neural networks are swapped between the tiny microcontroller's memory and the external large flash memory.
“In this approach neural networks are swapped between the tiny microcontroller's memory and the external large flash memory.”
From A review on TinyML: State-of-the-art and prospects (2022) · cites this paper as Miao and Lin, 2021
Upon demand, a requested chunk of the neural network layer is swapped from the external flash or memory SD card to the main memory.
“Upon a demand is raised, a requested chunk of the neural network layer is swapped from the external flash or memory SD card to the main memory.”
From A review on TinyML: State-of-the-art and prospects (2022) · cites this paper as Miao and Lin, 2021
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · see citing works
Very Deep Convolutional Networks for Large-Scale Image Recognition
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1409.1556 · Cited by 74,248 · see citing works
Going deeper with convolutions
Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., & Rabinovich, A. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298594 · Cited by 48,562 · see citing works
MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
Howard, A. G., Zhu, M., Chen, B., Kalenichenko, D., Wang, W., Weyand, T., Andreetto, M., & Adam, H. (2017) · arXiv · doi:10.48550/arxiv.1704.04861 · Cited by 19,015 · see citing works
Pruning Filters for Efficient ConvNets
Li, H., Kadav, A., Durdanovic, I., Samet, H., & Graf, H. P. (2016) · arXiv · doi:10.48550/arxiv.1608.08710 · Cited by 2,928 · see citing works
TVM: an automated end-to-end optimizing compiler for deep learning
(2018) · arXiv (Cornell University) · doi:10.5555/3291168.3291211 · Cited by 1,170 · see citing works
To prune, or not to prune: exploring the efficacy of pruning for model compression
Zhu, M., & Gupta, S. (2017) · arXiv · doi:10.48550/arxiv.1710.01878 · Cited by 805 · see citing works
Deep Learning with Limited Numerical Precision
Gupta, S., Agrawal, A., Gopalakrishnan, K., & Narayanan, P. (2015) · Cited by 707 · see citing works
Alwani, M., Chen, H., Ferdman, M., & Milder, P. (2016) · 2016 49th Annual IEEE/ACM International Symposium on Microarchitecture (MICRO) · doi:10.1109/micro.2016.7783725 · Cited by 662 · see citing works
TensorFlow Lite Micro: Embedded Machine Learning on TinyML Systems
David, R., Duke, J., Jain, A., Reddi, V. J., Jeffries, N., Li, J., Kreeger, N., Nappier, I., Natraj, M., Regev, S., et al. (2020) · arXiv · doi:10.48550/arxiv.2010.08678 · Cited by 470 · see citing works
MCUNet: Tiny Deep Learning on IoT DevicesPreprint
Lin, J., Chen, W.-M., Lin, Y., Cohn, J., Gan, C., & Han, S. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2007.10319 · Cited by 425 · see citing works
vDNN: Virtualized deep neural networks for scalable, memory-efficient neural network design
Rhu, M., Gimelshein, N. E., Clemons, J., Zulfiqar, A., & Keckler, S. W. (2016) · IEEE/ACM International Symposium on Microarchitecture (MICRO) · doi:10.1109/micro.2016.7783721 · Cited by 416 · see citing works
Hello Edge: Keyword Spotting on MicrocontrollersPreprint
Zhang, Y., Suda, N., Lai, L., & Chandra, V. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1711.07128 · Cited by 356 · see citing works
CMSIS-NN: Efficient Neural Network Kernels for Arm Cortex-M CPUsPreprint
Lai, L., Suda, N., & Chandra, V. (2018) · arXiv (Cornell University) · doi:10.4230/oasics.ng-res.2025.4 · Cited by 304 · see citing works
Gobieski, G., Lucia, B., & Beckmann, N. (2019) · International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS) · doi:10.1145/3297858.3304011 · Cited by 203 · see citing works
SwapAdvisor: Pushing Deep Learning Beyond the GPU Memory Limit via Smart Swapping
Huang, C.-C., Jin, G., & Li, J. (2020) · Proceedings of the Twenty-Fifth International Conference on Architectural Support for Programming Languages and Operating Systems · doi:10.1145/3373376.3378530 · Cited by 158 · cited 2× · see citing works
“While prior systems have swapped NN tiles between a server’s CPU/GPU memories for training [32], applying the idea to MCU for inference, in particular swapping between its small SRAM and a wimpy SD card, raises multiple concerns: loss of SD card durability, slowdown in NN execution, energy increase, and safety/securit…”
BackgroundHeap data allocation to scratch-pad memory in embedded systems
Dominguez, A., Udayakumaran, S., & Barua, R. K. (2005) · Journal of Embedded Computing · doi:10.3233/emc-2005-00051 · Cited by 155 · see citing works
Dynamic control flow in large-scale machine learning
Yu, Y., Abadi, M., Barham, P., Brevdo, E., Burrows, M., Davis, A., Dean, J., Ghemawat, S., Harley, T., Hawkins, P., et al. (2018) · European Conference on Computer Systems · doi:10.1145/3190508.3190551 · Cited by 86 · see citing works
All the AES You Need on Cortex-M3 and M4
Schwabe, P., & Stoffelen, K. (2017) · Lecture notes in computer science · doi:10.1007/978-3-319-69453-5_10 · Cited by 86 · see citing works
Memory Requirements for Convolutional Neural Network Hardware Accelerators
Siu, K., Stuart, D. M., Mahmoud, M., & Moshovos, A. (2018) · IEEE International Symposium on Workload Characterization · doi:10.1109/iiswc.2018.8573527 · Cited by 75 · see citing works
Xu, M., Qian, F., Mei, Q., Huang, K. R., & Liu, X. (2018) · Proceedings of the ACM on Interactive Mobile Wearable and Ubiquitous Technologies · doi:10.1145/3287075 · Cited by 74 · see citing works
Yang, H., Fritzsche, M., Bartz, C., & Meinel, C. (2017) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3123266.3129393 · Cited by 56 · see citing works
IOS: Inter-Operator Scheduler for CNN AccelerationPreprint
Ding, Y., Zhu, L., Jia, Z., Pekhimenko, G. G., & Han, S. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2011.01302 · Cited by 51 · see citing works
Approximate query service on autonomous IoT cameras
Xu, M., Zhang, X., Liu, Y., Huang, G., Liu, X., & Lin, F. X. (2020) · International Conference on Mobile Systems, Applications, and Services · doi:10.1145/3386901.3388948 · Cited by 48 · see citing works
Self-citationTian, J. W., & Hong, S. (2019) · International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS) · doi:10.1145/3297858.3304038 · Cited by 41 · see citing works
Out-of-core Training for Extremely Large-Scale Neural Networks With Adaptive Window-Based SchedulingPreprint
Hayakawa, A., & Narihira, T. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.14109 · Cited by 3 · see citing works
Flagged first, then by citations
2 of the 28 references couldn’t be matched to a known paper.
Research works printed but not linked: 9
- [16][16] Performance of state-of-the-art cryptography on arm-based microprocessors. https://csrc . nist . gov/csrc/media/events/lightweightcryptography-workshop-2015/documents/presentations/session7vincent . pdf, 2020. -
- [21][21] Manoj Alwani, Han Chen, Michael Ferdman, and Peter Milder. Fusedlayer cnn accelerators. In The 49th Annual IEEE/ACM International Symposium on Microarchitecture , page 22. IEEE Press, 2016. -
- [28][28] Song Han, Huizi Mao, and William J Dally. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. arXiv preprint arXiv:1510.00149 , 2015. - · arXiv
- [33][33] Tian Jin and Seokin Hong. Split-cnn: Splitting window-based operations in convolutional neural networks for memory system optimization. In Proceedings of the Twenty-Fourth International Conference on Architectural Support for Programming Languages and Operating Systems , pages 835-847, 2019. -
- [34][34] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. In Advances in neural information processing systems , pages 1097-1105, 2012. -
- [40][40] Haichen Shen, Seungyeop Han, Matthai Philipose, and Arvind Krishnamurthy. Fast video classification via adaptive cascading of deep models. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 3646-3654, 2017. -
- [44][44] Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 1-9, 2015. -
- [45][45] Mengwei Xu, Feng Qian, Qiaozhu Mei, Kang Huang, and Xuanzhe Liu. Deeptype: On-device deep learning for input personalization service with minimal privacy concern. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies , 2(4):1-26, 2018. -
- [47][47] Haojin Yang, Martin Fritzsche, Christian Bartz, and Christoph Meinel. Bmxnet: An open-source binary neural network implementation based on mxnet. In Proceedings of the 25th ACM international conference on Multimedia , pages 1209-1212. ACM, 2017. -
Cited by
Sorted by citations
A review on TinyML: State-of-the-art and prospects
Ray, P. P. (2022) · Journal of King Saud University - Computer and Information Sciences · doi:10.1016/j.jksuci.2021.11.019 · Cited by 460 · see citing works
Tiny Machine Learning: Progress and Futures [Feature]
Lin, J., Zhu, L., Chen, W.-M., Wang, W.-C., & Han, S. (2023) · IEEE Circuits and Systems Magazine · doi:10.1109/mcas.2023.3302182 · Cited by 216 · see citing works
AI augmented Edge and Fog computing: Trends and challenges
Tuli, S., Mirhakimi, F., Pallewatta, S., Zawad, S., Casale, G., Javadi, B., Yan, F., Buyya, R., & Jennings, N. R. (2023) · Journal of Network and Computer Applications · doi:10.1016/j.jnca.2023.103648 · Cited by 141 · see citing works
Enabling Resource-Efficient AIoT System With Cross-Level Optimization: A Survey
Liu, S., Guo, B., Fang, C., Wang, Z., Luo, S., Zhou, Z., & Yu, Z. (2023) · IEEE Communications Surveys & Tutorials · doi:10.1109/comst.2023.3319952 · Cited by 73 · see citing works
EdgeMoE: Empowering Sparse Large Language Models on Mobile Devices
Yi, R., Guo, L., Wei, S., Zhou, A., Wang, S., & Xu, M. (2025) · IEEE Transactions on Mobile Computing · doi:10.1109/tmc.2025.3546466 · Cited by 69 · see citing works
MCUNetV2: Memory-Efficient Patch-based Inference for Tiny Deep LearningPreprint
Lin, J., Chen, W.-M., Cai, H., Gan, C., & Han, S. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.15352 · Cited by 67 · see citing works
DTMM: Deploying TinyML Models on Extremely Weak IoT Devices with Pruning
Han, L., Xiao, Z., & Li, Z. (2024) · IEEE Conference on Computer Communications (INFOCOM) · doi:10.1109/infocom52122.2024.10621387 · Cited by 23 · see citing works
SwapNet: Efficient Swapping for DNN Inference on Edge AI Devices Beyond the Memory Budget
Wang, K., Cao, J., Zhou, Z., & Li, Z. (2024) · IEEE Transactions on Mobile Computing · doi:10.1109/tmc.2024.3355764 · Cited by 23 · see citing works
Demand Layering for Real-Time DNN Inference with Minimized Memory Usage
Ji, M., Yi, S., Koo, C., Ahn, S., Seo, D., Dutt, N. D., & Kim, J.-C. (2022) · IEEE Real-Time Systems Symposium (RTSS) · doi:10.1109/rtss55097.2022.00033 · Cited by 22 · see citing works
Developing a TinyML Image Classifier in an Hour
Berta, R., Dabbous, A., Lazzaroni, L., Pau, D. P., & Bellotti, F. (2024) · IEEE Open Journal of the Industrial Electronics Society · doi:10.1109/ojies.2024.3451959 · Cited by 20 · see citing works
Power-Performance Characterization of TinyML Systems
Zhang, Y., Wijerathne, D., Li, Z., & Mitra, T. (2022) · IEEE International Conference on Computer Design: VLSI in Computers and Processors (ICCD) · doi:10.1109/iccd56317.2022.00099 · Cited by 15 · see citing works
TinyTS: Memory-Efficient TinyML Model Compiler Framework on Microcontrollers
Liu, Y.-Y., Zheng, H.-S., Hu, Y. F., Hsu, C.-N., & Yeh, T. T. (2024) · IEEE International Symposium on High-Performance Computer Architecture (HPCA) · doi:10.1109/hpca57654.2024.00070 · Cited by 10 · see citing works
Hemadasa, N., Venzke, M., Turau, V., & Huang, Y. (2024) · Chronicle of computing · doi:10.55432/978-1-6692-0005-5_9 · Cited by 1 · see citing works
- Authors
- Miao, H., & Lin, F. X.
- Affiliations
- Purdue University West Lafayette (US)
- University of Virginia (US)
- ORCID
- Lin, Felix Xiaozhu
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2021
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2101.08744
- OpenAlex
- W3128736549
- Cited by
- 12
- References
- 28
- Citations by year
- 2025: 1 · 2024: 5 · 2023: 3 · 2022: 2 · 2021: 1
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Computer Networks and Communications › IoT and Edge/Fog Computing
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Computer Science › Hardware and Architecture › Parallel Computing and Optimization Techniques
- Keywords
- energy consumption, data security, edge AI, large-scale neural networks
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
