CoChat
Sign Up
PreprintCited by 470Advanced Neural Network ApplicationsEmbedded Systems Design TechniquesIoT and Edge/Fog Computing

TensorFlow Lite Micro: Embedded Machine Learning on TinyML Systems

Open access
David, Robert, Duke, Jared, Jain, Advait, Reddi, Vijay Janapa, Jeffries, Nat, Li, Jian, Kreeger, Nick, Nappier, Ian +5 more·2020·arXiv
DOI 10.48550/arxiv.2010.08678arXiv 2010.08678
Open this paper in CoChat — free accountOpen PDF

Abstract

Deep learning inference on embedded devices is a burgeoning field with myriad applications because tiny embedded devices are omnipresent. But we must overcome major challenges before we can benefit from this opportunity. Embedded processors are severely resource constrained. Their nearest mobile counterparts exhibit at least a 100 -- 1,000x difference in compute capability, memory availability, and power consumption. As a result, the machine-learning (ML) models and associated ML inference framework must not only execute efficiently but also operate in a few kilobytes of memory. Also, the embedded devices' ecosystem is heavily fragmented. To maximize efficiency, system vendors often omit many features that commonly appear in mainstream systems, including dynamic memory allocation and virtual memory, that allow for cross-platform interoperability. The hardware comes in many flavors (e.g., instruction-set architecture and FPU support, or lack thereof). We introduce TensorFlow Lite Micro (TF Micro), an open-source ML inference framework for running deep-learning models on embedded systems. TF Micro tackles the efficiency requirements imposed by embedded-system resource constraints and the fragmentation challenges that make cross-platform interoperability nearly impossible. The framework adopts a unique interpreter-based approach that provides flexibility while overcoming these challenges. This paper explains the design decisions behind TF Micro and describes its implementation details. Also, we present an evaluation to demonstrate its low resource requirement and minimal run-time performance overhead.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2010.08678

Cites

1 of 1 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • The proposed framework's deployment stage leverages multi-tenancy concepts for dynamic memory sharing among model components.

    “The deployment stage then addresses the practicalities of running these models on MCUs, including: (1) leveraging multi-tenancy concepts [26] for dynamic memory sharing among model components (e.g., cascade stages), and (2) applying model and library optimizations for maximal efficiency.”

    From A cascade framework for on-device uncertainty-aware event detection on microcontrollers (2026) · cites this paper as [26]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 20

  1. Banbury, C. R., Reddi, V. J., Lam, M., Fu, W., Fazel, A., Holleman, J., Huang, X., Hurtado, R., Kanter, D., Lokhmotov, A., et al. Benchmarking tinyml systems: Challenges and direction. arXiv preprint arXiv:2003.04821 , 2020. · arXiv
  2. Chavarriaga, R., Sagha, H., Calatroni, A., Digumarti, S. T., Tr¨ oster, G., Mill´ an, J. d. R., and Roggen, D. The opportunity challenge: A benchmark database for on-body sensor-based activity recognition. Pattern Recognition Letters , 34(15):2033-2042, 2013.
  3. Chen, G., Parada, C., and Heigold, G. Small-footprint keyword spotting using deep neural networks. In 2014 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pp. 4087-4091. IEEE, 2014.
  4. Chen, T., Moreau, T., Jiang, Z., Zheng, L., Yan, E., Shen, H., Cowan, M., Wang, L., Hu, Y., Ceze, L., et al. TVM: An automated end-to-end optimizing compiler for deep learning. In 13th { USENIX } Symposium on Operating Systems Design and Implementation ( { OSDI } 18) , pp. 578-594, 2018.
  5. Chowdhery, A., Warden, P., Shlens, J., Howard, A., and Rhodes, R. Visual wake words dataset. arXiv preprint arXiv:1906.05721 , 2019. · arXiv
  6. Garey, M. R., Graham, R. L., and Ullman, J. D. Worst-case analysis of memory allocation algorithms. In Proceedings of the fourth annual ACM symposium on Theory of computing , pp. 143-150, 1972.
  7. Gruenstein, A., Alvarez, R., Thornton, C., and Ghodrat, M. A cascade architecture for keyword spotting on mobile devices. arXiv preprint arXiv:1712.03603 , 2017. · arXiv
  8. Intel. Intel-64 and ia-32 architectures software developer's manual. Volume 3A: System Programming Guide, Part , 1 (64), 2013.
  9. Koizumi, Y., Saito, S., Uematsu, H., Harada, N., and Imoto, K. ToyADMOS: A dataset of miniaturemachine operating sounds for anomalous sound detection. In Proceedings of IEEE Workshop on Applications of Signal Processing to Audio and Acoustics (WASPAA) , pp. 308-312, November 2019. URL https: //ieeexplore . ieee . org/document/8937164 .
  10. Krishnamoorthi, R. Quantizing deep convolutional networks for efficient inference: A whitepaper. arXiv preprint arXiv:1806.08342 , 2018. · arXiv
  11. Kumar, A., Goyal, S., and Varma, M. Resource-efficient machine learning in 2 kb ram for the internet of things. In International Conference on Machine Learning , pp. 1935-1944, 2017.
  12. Lai, L., Suda, N., and Chandra, V. CMSIS-NN: Efficient neural network kernels for Arm Cortex-M cpus. arXiv preprint arXiv:1801.06601 , 2018. · arXiv
  13. Lin, J., Chen, W.-M., Lin, Y., Cohn, J., Gan, C., and Han, S. Mcunet: Tiny deep learning on IoT devices. arXiv preprint arXiv:2007.10319 , 2020. · arXiv
  14. Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Doll´ ar, P., and Zitnick, C. L. Microsoft COCO: Common objects in context. In European conference on computer vision , pp. 740-755. Springer, 2014.
  15. Martello, S. Chapter 8: Bin packing, knapsack problems: algorithms and computer implementations. WileyInterscience series in discrete mathematics and optimization , 1990.
  16. Mattson, P., Reddi, V. J., Cheng, C., Coleman, C., Diamos, G., Kanter, D., Micikevicius, P., Patterson, D., Schmuelling, G., Tang, H., et al. MLPerf: An industry standard benchmark suite for machine learning performance. IEEE Micro , 40(2):8-16, 2020.
  17. Reddi, V. J., Cheng, C., Kanter, D., Mattson, P., Schmuelling, G., Wu, C.-J., Anderson, B., Breughe, M., Charlebois, M., Chou, W., et al. MLPerf inference benchmark. In 2020 ACM/IEEE 47th Annual International Symposium on Computer Architecture (ISCA) , pp. 446459. IEEE, 2020.
  18. Rotem, N., Fix, J., Abdulrasool, S., Catron, G., Deng, S., Dzhabarov, R., Gibson, N., Hegeman, J., Lele, M., Levenstein, R., et al. Glow: Graph lowering compiler techniques for neural networks. arXiv preprint arXiv:1805.00907 , 2018. · arXiv
  19. Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, I., and Salakhutdinov, R. Dropout: a simple way to prevent neural networks from overfitting. The journal of machine learning research , 15(1):1929-1958, 2014.
  20. Susto, G. A., Schirru, A., Pampuri, S., McLoone, S., and Beghi, A. Machine learning for predictive maintenance: A multiple classifier approach. IEEE Transactions on Industrial Informatics , 11(3):812-820, 2014.

Cited by

Sorted by citations

Authors
David, R., Duke, J., Jain, A., Reddi, V. J., Jeffries, N., Li, J., Kreeger, N., Nappier, I., Natraj, M., Regev, S., Rhodes, R., Wang, T., & Warden, P.
ORCID
Venue
arXiv
Publisher
arXiv
Published
2020
Volume
3
Pages
800-811
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2010.08678
arXiv
2010.08678
OpenAlex
W3093348380
Cited by
470
References
1
Citations by year
2026: 5 · 2025: 59 · 2024: 128 · 2023: 143 · 2022: 75 · 2021: 56 · 2020: 3 · 2019: 1
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
  • Proceedings of Machine Learning and Systems
Funders
Harvard University
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
  • Physical Sciences › Computer Science › Hardware and Architecture › Embedded Systems Design Techniques
  • Physical Sciences › Computer Science › Computer Networks and Communications › IoT and Edge/Fog Computing
Keywords
TensorFlow Lite Micro, embedded machine learning, TinyML, deep learning inference, resource-constrained devices, cross-platform interoperability, low-power inference

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review