Deeploy: Enabling Energy-Efficient Deployment of Small Language Models on Heterogeneous Microcontrollers
Open accessAbstract
With the rise of embodied foundation models (EFMs), most notably small language models (SLMs), adapting Transformers for the edge applications has become a very active field of research. However, achieving the end-to-end deployment of SLMs on the microcontroller (MCU)-class chips without high-bandwidth off-chip main memory access is still an open challenge. In this article, we demonstrate high efficiency end-to-end SLM deployment on a multicore RISC-V (RV32) MCU augmented with ML instruction extensions and a hardware neural processing unit (NPU). To automate the exploration of the constrained, multidimensional memory versus computation tradeoffs involved in the aggressive SLM deployment on the heterogeneous (multicore+NPU) resources, we introduce Deeploy, a novel deep neural network (DNN) compiler, which generates highly optimized C code requiring minimal runtime support. We demonstrate that Deeploy generates the end-to-end code for executing SLMs, fully exploiting the RV32 cores’ instruction extensions and the NPU. We achieve leading-edge energy and throughput of$490 \; \mu $J per token, at 340 token per second for an SLM trained on the TinyStories dataset, running for the first time on an MCU-class device without the external memory.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
TLMs require less memory and processing power, which translates to lower energy consumption.
“TLMs are designed for efficiency, requiring less memory and processing power, which translates to lower energy consumption [70].”
From Tiny Language Models for Automation and Control: Overview, Potential Applications, and Future Research Directions (2025) · cites this paper as [70]
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · see citing works
Jakubův, J., Chvalovský, K., Goertzel, Z., et al. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.itp.2023.19 · Cited by 76,648 · see citing works
Howard, A., Sandler, M., Chen, B., Wang, W., Chen, L.-C., Tan, M., Chu, G., Vasudevan, V., Zhu, Y., Pang, R., et al. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00140 · Cited by 9,708 · see citing works
Llama 2: Open Foundation and Fine-Tuned Chat Models
Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. (2023) · arXiv · doi:10.48550/arxiv.2307.09288 · Cited by 2,604 · see citing works
A Survey of Quantization Methods for Efficient Neural Network Inference
Gholami, A., Kim, S., Dong, Z., Yao, Z., Mahoney, M. W., & Keutzer, K. (2022) · Low-Power Computer Vision · doi:10.1201/9781003162810-13 · Cited by 1,219 · see citing works
TVM: an automated end-to-end optimizing compiler for deep learning
(2018) · arXiv (Cornell University) · doi:10.5555/3291168.3291211 · Cited by 1,170 · see citing works
Gemini: A Family of Highly Capable Multimodal Models
Gemini Team, Anil, R., Borgeaud, S., Alayrac, J.-B., Yu, J., Soricut, R., Schalkwyk, J., Dai, A. M., Hauth, A., Millican, K., et al. (2023) · arXiv · doi:10.48550/arxiv.2312.11805 · Cited by 990 · see citing works
Do As I Can, Not As I Say: Grounding Language in Robotic Affordances
Ahn, M., Brohan, A., Brown, N., Chebotar, Y., Cortes, O., David, B., Finn, C., Fu, C., Gopalakrishnan, K., Hausman, K., et al. (2022) · arXiv · doi:10.48550/arxiv.2204.01691 · Cited by 624 · see citing works
PaLM-E: An Embodied Multimodal Language Model
Driess, D., Xia, F., Sajjadi, M. S. M., Lynch, C., Chowdhery, A., Ichter, B., Wahid, A., Tompson, J., Vuong, Q., Yu, T., et al. (2023) · arXiv · doi:10.48550/arxiv.2303.03378 · Cited by 591 · see citing works
MLIR: Scaling Compiler Infrastructure for Domain Specific Computation
Lattner, C., Amini, M., Bondhugula, U. K. R., Cohen, A., Davis, A., Pienaar, J. A., Riddle, R., Shpeisman, T., Vasilache, N., & Zinenko, O. (2021) · Proceedings of the CGO/Proceedings of the ... CGO · doi:10.1109/cgo51591.2021.9370308 · Cited by 550 · see citing works
Jouppi, N. P., Kurian, G. T., Li, S., Ma, P., Nagarajan, R., Nai, L., Patil, N. P., Subramanian, S., Swing, A., Towles, B., et al. (2023) · Proceedings - International Symposium on Computer Architecture/Proceedings · doi:10.1145/3579371.3589350 · Cited by 528 · see citing works
TensorFlow Lite Micro: Embedded Machine Learning on TinyML Systems
David, R., Duke, J., Jain, A., Reddi, V. J., Jeffries, N., Li, J., Kreeger, N., Nappier, I., Natraj, M., Regev, S., et al. (2020) · arXiv · doi:10.48550/arxiv.2010.08678 · Cited by 470 · see citing works
CMSIS-NN: Efficient Neural Network Kernels for Arm Cortex-M CPUsPreprint
Lai, L., Suda, N., & Chandra, V. (2018) · arXiv (Cornell University) · doi:10.4230/oasics.ng-res.2025.4 · Cited by 304 · see citing works
Foundation models in robotics: Applications, challenges, and the future
Firoozi, R., Tucker, J., Tian, S., Majumdar, A., Sun, J., Liu, W., Zhu, Y., Song, S., Kapoor, A., Hausman, K., et al. (2024) · The International Journal of Robotics Research · doi:10.1177/02783649241281508 · Cited by 267 · see citing works
Root Mean Square Layer Normalization
Zhang, B., & Sennrich, R. (2019) · arXiv · doi:10.48550/arxiv.1910.07467 · Cited by 262 · see citing works
SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models
Xiao, G., Lin, J., Seznec, M., Wu, H., Demouth, J., & Han, S. (2022) · arXiv · doi:10.48550/arxiv.2211.10438 · Cited by 247 · see citing works
GAP-8: A RISC-V SoC for AI at the Edge of the IoT
Flamand, É., Rossi, D., Conti, F., Loi, I., Antonio Pullini, D., Rotenberg, F., & Luca Benini, C. (2018) · IEEE International Conference on Application-specific Systems, Architectures and Processors (ASAP) · doi:10.1109/asap.2018.8445101 · Cited by 216 · see citing works
Self-citationBRECQ: Pushing the Limit of Post-Training Quantization by Block Reconstruction
Li, Y., Gong, R., Tan, X., Yang, Y., Hu, P., Zhang, Q., Yu, F., Wang, W., & Gu, S. (2021) · arXiv · doi:10.48550/arxiv.2102.05426 · Cited by 190 · see citing works
1.1 Deep Learning Hardware: Past, Present, and Future
LeCun, Y. (2019) · Digest of technical papers/Digest of technical papers - IEEE International Solid-State Circuits Conference · doi:10.1109/isscc.2019.8662396 · Cited by 188 · see citing works
I-BERT: Integer-only BERT Quantization
Kim, S., Gholami, A., Yao, Z., Mahoney, M. W., & Keutzer, K. (2021) · arXiv · doi:10.48550/arxiv.2101.01321 · Cited by 186 · see citing works
Tumanov, A., Zhu, T., Park, J. W., Kozuch, M. A., Harchol‐Balter, M., & Ganger, G. R. (2016) · European Conference on Computer Systems · doi:10.1145/2901318.2901355 · Cited by 170 · see citing works
NVIDIA Hopper H100 GPU: Scaling Performance
Choquette, J. (2023) · IEEE Micro · doi:10.1109/mm.2023.3256796 · Cited by 156 · see citing works
DORY: Automatic End-to-End Deployment of Real-World DNNs on Low-Cost IoT MCUs
Burrello, A., Garofalo, A., Bruschi, N., Tagliavini, G., Rossi, D., & Conti, F. (2021) · IEEE Transactions on Computers · doi:10.1109/tc.2021.3066883 · Cited by 141 · see citing works
Self-citationBanbury, C., Reddi, V. J., Torelli, P., Holleman, J., Jeffries, N., Kiraly, C., Montino, P., Kanter, D., Ahmed, S., Pau, D., et al. (2021) · arXiv · doi:10.48550/arxiv.2106.07597 · Cited by 121 · see citing works
Jain, S. R., Gural, A., Wu, M., & Dick, C. H. (2019) · arXiv · doi:10.48550/arxiv.1903.08066 · Cited by 115 · see citing works
DIANA: An End-to-End Energy-Efficient Digital and ANAlog Hybrid Neural Network SoC
Ueyoshi, K., Papistas, I. A., Houshmand, P., Sarda, G. M., Jain, V., Shi, M., Zheng, Q., Giraldo, S., Vrancx, P., Doevenspeck, J., et al. (2022) · Digest of technical papers/Digest of technical papers - IEEE International Solid-State Circuits Conference · doi:10.1109/isscc42614.2022.9731716 · Cited by 89 · see citing works
TinyLlama: An Open-Source Small Language ModelPreprint
Zhang, P., Zeng, G., Wang, T., & Lu, W. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2401.02385 · Cited by 86 · see citing works
XpulpNN: Accelerating Quantized Neural Networks on RISC-V Processors Through ISA Extensions
Garofalo, A., Tagliavini, G., Conti, F., Rossi, D., & Luca Benini, C. (2020) · Design, Automation, and Test in Europe · doi:10.23919/date48585.2020.9116529 · Cited by 80 · see citing works
Self-citationAn efficient profile-based algorithm for scratchpad memory partitioning
Angiolini, F., Benini, L., & Caprara, A. (2005) · IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems · doi:10.1109/tcad.2005.852299 · Cited by 53 · see citing works
Self-citationTinyStories: How Small Can Language Models Be and Still Speak Coherent English?
Eldan, R., & Li, Y. (2023) · arXiv · doi:10.48550/arxiv.2305.07759 · Cited by 52 · see citing works
Memory-efficient Patch-based Inference for Tiny Deep Learning
Lin, J., Chen, W.-M., Cai, H., Gan, C., & Han, S. (2021) · neural information processing systems · Cited by 38 · see citing works
A review of on-device machine learning for IoT: An energy perspective
Tekin, N., Arış, A., Acar, A., Uluagac, A. S., & Güngör, V. Ç. (2023) · Ad Hoc Networks · doi:10.1016/j.adhoc.2023.103348 · Cited by 37 · see citing works
Siracusa: A 16 nm Heterogenous RISC-V SoC for Extended Reality With At-MRAM Neural Engine
Prasad, A. S., Scherer, M., Conti, F., Rossi, D., Mauro, A. D., Eggimann, M., Gomez, J. T., Li, Z., Sarwar, S. S., Wang, Z., et al. (2024) · IEEE Journal of Solid-State Circuits · doi:10.1109/jssc.2024.3385987 · Cited by 24 · see citing works
Self-citationTelaMalloc: Efficient On-Chip Memory Allocation for Production Machine Learning Accelerators
Maas, M., Beaugnon, U., Chauhan, A., & Ilbeyi, B. (2022) · International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS) · doi:10.1145/3567955.3567961 · Cited by 21 · see citing works
BrainBERT: Self-supervised representation learning for intracranial recordingsPreprint
Wang, C., Subramaniam, V., Yaari, A., Kreiman, G., Katz, B., Cases, I., & Barbu, A. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2302.14367 · Cited by 17 · see citing works
MobileLLM: Optimizing Sub-billion Parameter Language Models for On-Device Use CasesPreprint
Liu, Z., Zhao, C., Iandola, F., Lai, C., Tian, Y., Fedorov, I., Xiong, Y., Chang, E., Shi, Y., Krishnamoorthi, R., et al. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2402.14905 · Cited by 16 · see citing works
MobileVLM : A Fast, Strong and Open Vision Language Assistant for Mobile DevicesPreprint
Chu, X., Qiao, L., Lin, X., Xu, S., Yang, Y., Hu, Y., Wei, F., Zhang, X., Zhang, B., Wei, X. K., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2312.16886 · Cited by 13 · see citing works
EEGFormer: Towards Transferable and Interpretable Large-Scale EEG Foundation ModelPreprint
Chen, Y., Ren, K., Song, K., Wang, Y., Wang, Y., Li, D., & Qiu, L. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2401.10278 · Cited by 7 · see citing works
MiniMalloc: A Lightweight Memory Allocator for Hardware-Accelerated Machine Learning
Moffitt, M. D. (2023) · International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS) · doi:10.1145/3623278.3624752 · Cited by 6 · see citing works
Flagged first, then by citations
2 of the 41 references couldn’t be matched to a known paper.
Cited by
Sorted by citations
Lamaakal, I., Maleh, Y., El Makkaoui, K., Ouahbi, I., Pławiak, P., Alfarraj, O., Almousa, M., & Abd El-Latif, A. A. (2025) · Sensors · doi:10.3390/s25051318 · Cited by 60 · see citing works
Assessing the carbon footprint of language models: Towards sustainability in AI
Jeanquartier, F., Jean-Quartier, C., Rieder, P., Misirlić, V., Pasero, C., Hohensinner, R., Müller, H., & Holzinger, A. (2025) · Resources Conservation and Recycling · doi:10.1016/j.resconrec.2025.108670 · Cited by 22 · see citing works
State of the Art and Future Directions of Small Language Models: A Systematic Review
Corradini, F., Leonesi, M., & Piangerelli, M. (2025) · Big Data and Cognitive Computing · doi:10.3390/bdcc9070189 · Cited by 28 · see citing works
Wiese, P., İslamoğlu, G., Scherer, M., Macan, L., Jung, V. J.-B., Burrello, A., Conti, F., & Benini, L. (2025) · IEEE Design & Test · doi:10.1109/mdat.2025.3527371 · Cited by 9 · see citing works
Self-citationA Flexible Template for Edge Generative AI With High-Accuracy Accelerated Softmax and GELU
Belano, A., Tortorella, Y., Garofalo, A., Luca Benini, C., Rossi, D., & Conti, F. (2025) · IEEE Journal on Emerging and Selected Topics in Circuits and Systems · doi:10.1109/jetcas.2025.3562734 · Cited by 9 · see citing works
Self-citationOpen Source Heterogeneous SoCs for Artificial Intelligence: The PULP Platform experience
Conti, F., Garofalo, A., Rossi, D., Tagliavini, G., & Luca Benini, C. (2025) · IEEE Solid-State Circuits Magazine · doi:10.1109/mssc.2025.3537987 · Cited by 6 · see citing works
Self-citationJPPO: Joint Power and Prompt Optimization for Accelerated Large Language Model Services
You, F., Du, H., Huang, K., & Jamalipour, A. (2025) · International Conference on Communications · doi:10.1109/icc52391.2025.11161529 · Cited by 6 · see citing works
Sinigaglia, M., Kiamarzi, A., Bertuletti, M., Ghionda, L., Orlandi, M., Tedeschi, R., Giampietro, A. D., Tortorella, Y., Bertaccini, L., Benatti, S., et al. (2025) · IEEE Transactions on Circuits and Systems I Regular Papers · doi:10.1109/tcsi.2025.3580939 · Cited by 3 · see citing works
Self-citationReal-time and offline large language models on edge devices: a systematic review
Dinçer, E., & Kilimci, Z. H. (2026) · PeerJ Computer Science · doi:10.7717/peerj-cs.3769 · Cited by 3 · see citing works
Distributed Inference with Minimal Off-Chip Traffic for Transformers on Low-Power MCUs
Bochem, S., Jung, V. J. B., Prasad, A. S., Conti, F., & Benini, L. (2025) · Design, Automation, and Test in Europe · doi:10.23919/date64628.2025.10992712 · Cited by 4 · see citing works
Self-citationThroughput Maximization for Transformer Inference on Processing Near-Memory Architectures
Ge, M., Zhong, Y., Chen, S., & Kang, Y. (2025) · IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems · doi:10.1109/tcad.2025.3574235 · Cited by 3 · see citing works
Zhang, K. (2025) · Journal of Computational Methods in Sciences and Engineering · doi:10.1177/14727978251346034 · Cited by 2 · see citing works
NanoHydra: Energy-Efficient Time-Series Classification at the Edge
Cioflan, C., Fonseca, J., Wang, X., & Benini, L. (2025) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn64981.2025.11229151 · Cited by 2 · see citing works
Self-citationEndoscopy Image Classification for Wireless Capsules with CNNs on Microcontroller-Based Platforms
Busia, P., Pinna, A., & Meloni, P. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-87897-8_5 · Cited by 2 · see citing works
Kwon, J., & Park, D. (2025) · Applied Sciences · doi:10.3390/app15031056 · Cited by 1 · see citing works
JPPO++: Joint Power and Denoising-Inspired Prompt Optimization for Mobile LLM Services
You, F., Du, H., Huang, K., & Jamalipour, A. (2026) · IEEE Transactions on Mobile Computing · doi:10.1109/tmc.2026.3669711 · Cited by 1 · see citing works
Driss, M., Atitallah, S. B., Benjdira, B., AlHalawani, S., Ali, A. M., & Boulila, W. (2026) · Internet of Things · doi:10.1016/j.iot.2026.102020 · Cited by 1 · see citing works
Hsieh, S.-W., Chen, Y.-S., Tsai, P.-Y., Lin, M., Cheng, C.-Y., Hsu, L.-F., Huang, P.-H., Chih, H.-W., Chiang, P.-H., Chang, C.-M., et al. (2026) · Digest of technical papers/Digest of technical papers - IEEE International Solid-State Circuits Conference · doi:10.1109/isscc49663.2026.11409294
Busia, P., Pinna, A., & Meloni, P. (2026) · Journal of Systems Architecture · doi:10.1016/j.sysarc.2026.103851
Digital Sovereignty Beyond the Cloud: An Edge AI Framework for Small Island Developing States
Shareef, M. (2026) · International Conference on Artificial Intelligence, Systems, and Emerging Technologies (ICAISET) · doi:10.1109/icaiset66439.2026.11542056
Wang, R., Jung, V. J. B., Wiese, P., Conti, F., Burrello, A., & Luca Benini, C. (2026) · Design, Automation, and Test in Europe · doi:10.23919/date69613.2026.11539322
Self-citationLate Breaking Results: Conversion of Neural Networks into Logic Flows for Edge Computing
Stein, D., Huang, S., Drechsler, R., Li, B., & Zhang, G. L. (2026) · Design, Automation, and Test in Europe · doi:10.23919/date69613.2026.11539745
Survey and Roadmap for Efficiency-Accuracy Cooptimization Towards Embodied AI Chip Design
Zhao, J., Yu, H., & Li, Y. (2026) · International Symposium on Intelligent Robotics and Systems (ISoIRS) · doi:10.1109/isoirs70157.2026.11545322
Petrolo, V., Guella, F., Masera, G., & Martina, M. (2026) · Computers · doi:10.3390/computers15100667
Ortiz, L. A. P., Sánchez, A. C., & Soliz, I. F. M. (2026) · Future Internet · doi:10.3390/fi18110589
- Authors
- Scherer, M., Macan, L., Jung, V. J. B., Wiese, P., Bompani, L., Burrello, A., Conti, F., & Benini, L.
- Affiliations
- ETH Zurich (CH)
- University of Bologna (IT)
- Politecnico di Torino (IT)
- Integrated Systems Laboratory, ETH Zürich, Zürich, Switzerland
- ORCID
- Venue
- IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems
- Published
- 2024-11
- Volume
- 43
- Issue
- 11
- Pages
- 4009-4020
- ISSN
- 0278-0070 (ISSN-L)
- 1937-4151
- OpenAlex
- W4404101614
- Cited by
- 26
- References
- 41
- Citations by year
- 2026: 12 · 2025: 14
- FWCI
- 8.04
- Type
- journal article
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems (publishedVersion)
- arXiv (Cornell University) (submittedVersion, cc-by-sa)
- PORTO@Iris (submittedVersion, other-oa)
- Archivio istituzionale della ricerca (Alma Mater Studiorum Università di Bologna) (submittedVersion, other-oa)
- arXiv (Cornell University)
- Funders
- Spoke 1 on Future HPC of the Italian Research Center on High-Performance Computing, Big Data and Quantum Computing (ICSC) funded by MUR Mission 4-Next Generation EU
- Chips Joint Undertaking (Chips-JU) TRISTAN Project (101095947)
- CONVOLVE Project (101070374)
- EU Horizon Europe project NeuroSoC (101070634)
- Chips-JU, CONVOLVE, and NeuroSoC receive support from the European Union’s Horizon Europe Research and Innovation Program
- Topics
- Physical Sciences › Computer Science › Hardware and Architecture › Embedded Systems Design Techniques
- Physical Sciences › Computer Science › Hardware and Architecture › Parallel Computing and Optimization Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Keywords
- small language models, edge deployment, microcontroller, RISC-V, neural processing units, energy efficiency, transformer inference, hardware acceleration, TinyStories
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
