MLPerf Tiny Benchmark
Open accessAbstract
Advancements in ultra-low-power tiny machine learning (TinyML) systems promise to unlock an entirely new class of smart applications. However, continued progress is limited by the lack of a widely accepted and easily reproducible benchmark for these systems. To meet this need, we present MLPerf Tiny, the first industry-standard benchmark suite for ultra-low-power tiny machine learning systems. The benchmark suite is the collaborative effort of more than 50 organizations from industry and academia and reflects the needs of the community. MLPerf Tiny measures the accuracy, latency, and energy of machine learning inference to properly evaluate the tradeoffs between systems. Additionally, MLPerf Tiny implements a modular design that enables benchmark submitters to show the benefits of their product, regardless of where it falls on the ML deployment stack, in a fair and reproducible manner. The suite features four benchmarks: keyword spotting, visual wake words, image classification, and anomaly detection.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The quality target metric is measured as % (Top-1) except for the ToyADMOS (5*128) dataset, which needs area under curve (AUC) metric.
“Quality target metric is measured % (Top-1) except the ToyADMOS (5*128) dataset which needs area under curve (AUC) metric.”
From A review on TinyML: State-of-the-art and prospects (2022) · cites this paper as Banbury et al., 2021
MLPerf Tiny v0.5 supports four use cases: keyword spotting, visual wake words, image classification, and anomaly detection.
“Four use cases are supported by this benchmark that are as follows keyword spotting, visual wake words, image classification, and anomaly detection.”
From A review on TinyML: State-of-the-art and prospects (2022) · cites this paper as Banbury et al., 2021
Generic versions of numeric format are chosen as 8-bit integers to offer performance boost with minimum impact on model accuracy.
“It is also noticed that generic versions of numeric format are chosen as 8-bit integers to offer performance boost with minimum impact on the model accuracy.”
From A review on TinyML: State-of-the-art and prospects (2022) · cites this paper as Banbury et al., 2021
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · see citing works
Flagged first, then by citations
Research works printed but not linked: 13
- [4][4] C. Banbury, C. Zhou, I. Fedorov, R. Matas, U. Thakker, D. Gope, V. Janapa Reddi, M. Mattina, and P. Whatmough. Micronets: Neural network architectures for deploying tinyml applications on commodity microcontrollers. Proceedings of Machine Learning and Systems , 3, 2021. -
- [5][5] T. Bouguera, J.-F. Diouris, J.-J. Chaillout, R. Jaouadi, and G. Andrieux. Energy consumption model for sensor nodes based on lora and lorawan. Sensors , 18(7):2104, 2018. -
- [6][6] A. Chowdhery, P. Warden, J. Shlens, A. Howard, and R. Rhodes. Visual wake words dataset. CoRR , abs/1906.05721, 2019. URL http://arxiv.org/abs/1906.05721 . - · arXiv
- [7][7] R. David, J. Duke, A. Jain, V. J. Reddi, N. Jeffries, J. Li, N. Kreeger, I. Nappier, M. Natraj, S. Regev, et al. Tensorflow lite micro: Embedded machine learning on tinyml systems. arXiv preprint arXiv:2010.08678 , 2020. - · arXiv
- [8][8] I. Fedorov, R. P. Adams, M. Mattina, and P. Whatmough. Sparse: Sparse architecture search for cnns on resource-constrained microcontrollers. In Advances in Neural Information Processing Systems 32 , pages 4978-4990. Curran Associates, Inc., 2019. -
- [11][11] A. G. Howard, M. Zhu, B. Chen, D. Kalenichenko, W. Wang, T. Weyand, M. Andreetto, and H. Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv:1704.04861 , 2017. - · arXiv
- [12][12] Y. Koizumi, S. Saito, H. Uematsu, N. Harada, and K. Imoto. Toyadmos: A dataset of miniaturemachine operating sounds for anomalous sound detection. In 2019 IEEE Workshop on Applications of Signal Processing to Audio and Acoustics (WASPAA) , pages 313-317. IEEE, 2019. -
- [13][13] Y. Koizumi, Y. Kawaguchi, K. Imoto, T. Nakamura, Y. Nikaido, R. Tanabe, H. Purohit, K. Suefusa, T. Endo, M. Yasuda, and N. Harada. Description and discussion on DCASE2020 challenge task2: Unsupervised anomalous sound detection for machine condition monitoring. In arXiv e-prints: 2006.05822 , pages 1-4, June 2020. URL https://arxiv.org/abs/2006.05822 . - · arXiv
- [15][15] T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision , pages 740-755. Springer, 2014. -
- [17][17] H. Purohit, R. Tanabe, K. Ichige, T. Endo, Y. Nikaido, K. Suefusa, and Y. Kawaguchi. Mimii dataset: Sound dataset for malfunctioning industrial machine investigation and inspection. arXiv preprint arXiv:1909.09347 , 2019. - · arXiv
- [20][20] A. Torralba, R. Fergus, and W. T. Freeman. 80 million tiny images: A large data set for nonparametric object and scene recognition. IEEE transactions on pattern analysis and machine intelligence , 30(11):1958-1970, 2008. -
- [21][21] P. Warden. Speech commands: A dataset for limited-vocabulary speech recognition. arXiv preprint arXiv:1804.03209 , 2018. - · arXiv
- [22][22] Y. Zhang, N. Suda, L. Lai, and V. Chandra. Hello edge: Keyword spotting on microcontrollers. arXiv preprint arXiv:1711.07128 , 2017. · arXiv
Cited by
Sorted by citations
A compute-in-memory chip based on resistive random-access memory
Wan, W., Kubendran, R., Schaefer, C., Eryilmaz, S. B., Zhang, W., Wu, D., Deiss, S., Raina, P., Qian, H., Gao, B., et al. (2022) · Nature · doi:10.1038/s41586-022-04992-8 · Cited by 943 · see citing works
A review on TinyML: State-of-the-art and prospects
Ray, P. P. (2022) · Journal of King Saud University - Computer and Information Sciences · doi:10.1016/j.jksuci.2021.11.019 · Cited by 460 · see citing works
Giordano, M., Baumann, N., Crabolu, M., Fischer, R., Bellusci, G., & Magno, M. (2022) · IEEE Transactions on Instrumentation and Measurement · doi:10.1109/tim.2022.3165816 · Cited by 72 · see citing works
Widening Access to Applied Machine Learning with TinyML
Janapa Reddi, V., Plancher, B., Kennedy, S., Moroney, L., Warden, P., Suzuki, L., Agarwal, A., Banbury, C., Banzi, M., Bennett, M., et al. (2022) · Harvard Data Science Review · doi:10.1162/99608f92.762d171a · Cited by 62 · see citing works
Self-citationScherer, M., Macan, L., Jung, V. J. B., Wiese, P., Bompani, L., Burrello, A., Conti, F., & Benini, L. (2024) · IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems · doi:10.1109/tcad.2024.3443718 · Cited by 26 · see citing works
Wiese, P., İslamoğlu, G., Scherer, M., Macan, L., Jung, V. J.-B., Burrello, A., Conti, F., & Benini, L. (2025) · IEEE Design & Test · doi:10.1109/mdat.2025.3527371 · Cited by 9 · see citing works
QUTE: Quantifying Uncertainty in TinyML with Early-exit-assisted ensembles for model-monitoring
Ghanathe, N. P., & Wilton, S. J. E. (2024) · arXiv · doi:10.48550/arxiv.2404.12599
- Authors
- Banbury, C., Reddi, V. J., Torelli, P., Holleman, J., Jeffries, N., Kiraly, C., Montino, P., Kanter, D., Ahmed, S., Pau, D., Thakker, U., Torrini, A., Warden, P., Cordaro, J., Di Guglielmo, G., Duarte, J., Gibellini, S., Parekh, V., Tran, H., Tran, N., Wenxu, N., & Xuesong, X.
- Affiliations
- Harvard University (US)
- University of North Carolina at Charlotte (US)
- Fermi National Accelerator Laboratory (US)
- University of Wisconsin–Madison (US)
- University of California San Diego (US)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2021
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2106.07597
- OpenAlex
- W3174684282
- Cited by
- 121
- References
- 1
- Citations by year
- 2026: 5 · 2025: 15 · 2024: 34 · 2023: 42 · 2022: 22 · 2021: 2 · 2020: 1
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- Defense Advanced Research Projects Agency
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Computer Science › Computer Networks and Communications › IoT and Edge/Fog Computing
- Physical Sciences › Engineering › Electrical and Electronic Engineering › Low-power high-performance VLSI design
- Keywords
- TinyML, benchmark suite, machine learning inference, energy efficiency, latency, keyword spotting, image classification, anomaly detection, reproducibility
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
