MLPerf: An Industry Standard Benchmark Suite for Machine Learning Performance
The DOI matches its Crossref record; 6 of the 6 references it cites were found in a registry or an index.
Abstract
In this article, we describe the design choices behind MLPerf, a machine learning performance benchmark that has become an industry standard. The first two rounds of the MLPerf Training benchmark helped drive improvements to software-stack performance and scalability, showing a 1.3× speedup in the top 16-chip results despite higher quality targets and a 5.5× increase in system scale. The first round of MLPerf Inference received over 500 benchmark results from 14 different organizations, showing growing adoption.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“due to high communication overhead, on-time model update and possible leakage of confidential data, some edge systems prefer to train the model locally, i.e., on-device learning [222].”
From Bringing ai to edge: from deep learning’s perspective (2022) · cites this paper as [222]
Citations
Cites
Reddi, V. J., Cheng, C., Kanter, D., Mattson, P., Schmuelling, G., Wu, C.-J., Anderson, B., Breughe, M., Charlebois, M., Chou, W., et al. (2020) · 2020 ACM/IEEE 47th Annual International Symposium on Computer Architecture (ISCA) · doi:10.1109/isca45697.2020.00045 · Cited by 505
Self-citationMattson, P., Cheng, C., Coleman, C., Diamos, G., Micikevicius, P., Patterson, D., Tang, H., Wei, G.-Y., Bailis, P., Bittorf, V., et al. (2019) · arXiv · doi:10.48550/arxiv.1910.01500 · Cited by 206 · cited 2×
“See the work by Mattson et al. 5 and Reddi et al. 6 for more on related work.”
MethodSelf-citationFathom: reference workloads for modern deep learning methods
Adolf, R., Rama, S., Reagen, B., Wei, G.-Y., & Brooks, D. (2016) · 2016 IEEE International Symposium on Workload Characterization (IISWC) · doi:10.1109/iiswc.2016.7581275 · Cited by 169
Self-citationDixit, K. M. (1993) · Elsevier eBooks · Cited by 65
Dixit, K. M. (1991) · Parallel Computing · doi:10.1016/s0167-8191(05)80033-x · Cited by 56
Reddi, V. J., Cheng, C., Kanter, D., Mattson, P. R., Schmuelling, G., Wu, C.-J., Anderson, B., Breughe, M. B., Charlebois, M., Chou, W., et al. (2019) · arXiv (Cornell University) · doi:10.4230/oasics.parma-ditam.2021.4 · Cited by 35
Self-citation
Research works printed but not linked: 1
- coleman, 2017, DAWNBench: An end-to-end deep learning benchmark and competition, Proc 31st Conf Neural Inf Process Syst
Cited by
Reddi, V. J., Cheng, C., Kanter, D., Mattson, P., Schmuelling, G., Wu, C.-J., Anderson, B., Breughe, M., Charlebois, M., Chou, W., et al. (2020) · 2020 ACM/IEEE 47th Annual International Symposium on Computer Architecture (ISCA) · doi:10.1109/isca45697.2020.00045 · Cited by 505
“To keep pace with the changes, we established a process to maintain MLPerf [36].”
MethodSelf-citationServing DNNs like Clockwork: Performance Predictability from the Bottom Up
Gujarati, A., Karimi, R. J., Alzayat, S., Hao, W., Kaufmann, A., Vigfússon, Ý., & Mace, J. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.02464 · Cited by 70
“The Clockwork approach generalizes readily beyond GPUs to other inference-specific hardware accelerators [48], whose performance is arguably even more predictable.”
BackgroundInfluentialTensorFlow Lite Micro: Embedded Machine Learning on TinyML Systems
David, R., Duke, J., Jain, A., Reddi, V. J., Jeffries, N., Li, J., Kreeger, N., Nappier, I., Natraj, M., Regev, S., et al. (2020) · arXiv · doi:10.48550/arxiv.2010.08678 · Cited by 470
“MLPerf is adopting the benchmarks (Mattson et al., 2020; Reddi et al., 2020), and the tinyMLPerf benchmark suite imposes accuracy metrics for them (Banbury et al.”
MethodSelf-citationSustainable AI: Environmental Implications, Challenges and Opportunities
Wu, C.-J., Raghavendra, R., Gupta, U., Acun, B., Ardalani, N., Maeng, K., Chang, G., Behram, F. A., Huang, J., Bai, C., et al. (2021) · arXiv · doi:10.48550/arxiv.2111.00364 · Cited by 405
“The MLPerf benchmark standards can advance the field of AI in an environmentally-competitive manner by enabling the measurement of energy and/or carbon footprint.”
BackgroundSelf-citationMiller, C., Kathirgamanathan, A., Picchetti, B., Arjunan, P., Park, J. Y., Nagy, Z., Raftery, P., Hobson, B. W., Shi, Z., & Meggers, F. (2020) · Scientific Data · doi:10.1038/s41597-020-00712-x · Cited by 271
Metadata
- Authors
- Mattson, P., Reddi, V. J., Cheng, C., Coleman, C., Diamos, G., Kanter, D., Micikevicius, P., Patterson, D., Schmuelling, G., Tang, H., Wei, G.-Y., & Wu, C.-J.
- Affiliations
- Google (United States) (US)
- Harvard University (US)
- Intel (United States) (US)
- Stanford University (US)
- Landing AI
- Real World Technologies
- Nvidia (United Kingdom) (GB)
- University of California, Berkeley (US)
- Microsoft (United States) (US)
- Arizona State University (US)
- Venue
- IEEE Micro
- Published
- 2020-03-01
- Volume
- 40
- Issue
- 2
- Pages
- 8-16
- ISSN
- 0272-1732 (ISSN-L)
- 1937-4143
- OpenAlex
- W3008591352
- Cited by
- 171
- References
- 0
- Citations by year
- 2026: 22 · 2025: 24 · 2024: 21 · 2023: 25 · 2022: 25 · 2021: 31 · 2020: 21
- FWCI
- 21.43
- Type
- journal article
- Language
- en
- Open access
- closed
- Licence
- Locations
- IEEE Micro (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Hardware and Architecture › Parallel Computing and Optimization Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Computer Science › Computer Networks and Communications › Software System Performance and Reliability
- Keywords
- MLPerf, benchmark design, MLPerf Inference, training performance, inference performance, scalability, industry standards
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: MLPerf: An Industry Standard Benchmark Suite for Machine Learning Performance. https://app.cochat.ai/papers/10.1109/mm.2020.2974843
- The paper
- Mattson, P., Reddi, V. J., Cheng, C., Coleman, C., Diamos, G., Kanter, D., Micikevicius, P., Patterson, D., Schmuelling, G., Tang, H., Wei, G.-Y., & Wu, C.-J. (2020). MLPerf: An Industry Standard Benchmark Suite for Machine Learning Performance. IEEE Micro. https://doi.org/10.1109/mm.2020.2974843
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.