The DOI matches its Crossref record; 33 of the 34 references it cites were found in a registry or an index.
How to Evaluate Deep Neural Network Processors: TOPS/W (Alone) Considered Harmful
Open accessAbstract
A significant amount of specialized hardware has been developed for processing deep neural networks (DNNs) in both academia and industry. This article aims to highlight the key concepts required to evaluate and compare these DNN processors. We discuss existing challenges, such as the flexibility and scalability needed to support a wide range of neural networks, as well as design considerations for both the DNN processors and the DNN models themselves. We also describe specific metrics that can be used to evaluate and compare existing solutions beyond the commonly used tera-operations per second per watt (TOPS/W). This article is based on the tutorial "How to Understand and Evaluate Deep Learning Processors" that was given at the 2020 International Solid-State Circuits Conference, as well as excerpts from the book, Efficient Processing of Deep Neural Networks [36].
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“The energy consumption of executing a DNN model is highly dictated by the computation architecture and methods (e.g., ultra-low power compute-in-memory AI accelerator) at the edge computation nodes [249], the architecture of DNN models [250], and the wireless transmission for data exchange during the model inference procedure.”
From Edge artificial intelligence for 6g: vision, enabling technologies, and applications (2022) · cites this paper as [249]
Citations
Cites
ImageNet Large Scale Visual Recognition Challenge
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al. (2015) · International Journal of Computer Vision · doi:10.1007/s11263-015-0816-y · Cited by 39,380
MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
Howard, A. G., Zhu, M., Chen, B., Kalenichenko, D., Wang, W., Weyand, T., Andreetto, M., & Adam, H. (2017) · arXiv · doi:10.48550/arxiv.1704.04861 · Cited by 19,015
MnasNet: Platform-Aware Neural Architecture Search for Mobile
Tan, M., Chen, B., Pang, R., Vasudevan, V., Sandler, M., Howard, A., & Le, Q. V. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00293 · Cited by 3,416
Eyeriss: An Energy-Efficient Reconfigurable Accelerator for Deep Convolutional Neural Networks
Chen, Y.-H., Krishna, T., Emer, J. S., & Sze, V. (2017) · IEEE Journal of Solid-State Circuits · doi:10.1109/jssc.2016.2616357 · Cited by 3,247
Self-citationWilliams, S., Waterman, A., & Patterson, D. A. (2009) · Communications of the ACM · doi:10.1145/1498765.1498785 · Cited by 2,798
A Proof for the Queuing Formula: L = λ W
Little, J. D. C. (1961) · Operations Research · doi:10.1287/opre.9.3.383 · Cited by 2,777
The Case for Energy-Proportional Computing
Barroso, L. A., & Hölzle, U. (2007) · Computer · doi:10.1109/mc.2007.443 · Cited by 2,491
1.1 Computing's energy problem (and what we can do about it)
Horowitz, M. (2014) · 2014 IEEE International Solid-State Circuits Conference Digest of Technical Papers (ISSCC) · doi:10.1109/isscc.2014.6757323 · Cited by 2,069
EIE: efficient inference engine on compressed deep neural network
Han, S., Liu, X., Mao, H., Pu, J., Pedram, A., Horowitz, M. A., & Dally, W. J. (2016) · ACM SIGARCH Computer Architecture News · doi:10.1145/3007787.3001163 · Cited by 1,937
Eyeriss: a spatial architecture for energy-efficient dataflow for convolutional neural networks
Chen, Y.-H., Emer, J., & Sze, V. (2016) · ACM SIGARCH Computer Architecture News · doi:10.1145/3007787.3001177 · Cited by 1,182
Self-citationNeuro-Inspired Computing With Emerging Nonvolatile Memorys
Yu, S. (2018) · Proceedings of the IEEE · doi:10.1109/jproc.2018.2790840 · Cited by 1,170 · cited 1×
“In addition, advanced device and memory technologies can be used to reduce the switching capacitance between compute and memory, for instance, by enabling inmemory computing [28], [29].”
Eyeriss v2: A Flexible Accelerator for Emerging Deep Neural Networks on Mobile Devices
Chen, Y.-H., Yang, T.-J., Emer, J. S., & Sze, V. (2019) · IEEE Journal on Emerging and Selected Topics in Circuits and Systems · doi:10.1109/jetcas.2019.2910232 · Cited by 1,106
Self-citationSCNN: An Accelerator for Compressed-sparse Convolutional Neural Networks
Parashar, A., Rhu, M., Mukkara, A., Puglielli, A., Venkatesan, R., Khailany, B., Emer, J., Keckler, S. W., & Dally, W. J. (2017) · Proceedings of the 44th Annual International Symposium on Computer Architecture · doi:10.1145/3079856.3080254 · Cited by 896
Self-citationDesigning Energy-Efficient Convolutional Neural Networks Using Energy-Aware Pruning
Yang, T.-J., Chen, Y.-H., & Sze, V. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.643 · Cited by 732 · cited 1×
“Various works have proposed techniques for designing DNN models with hardware in the loop [22]–[25].”
MethodSelf-citationCnvlutin: ineffectual-neuron-free deep neural network computing
Albericio, J., Judd, P., Hetherington, T., Aamodt, T., Jerger, N. E., & Moshovos, A. (2016) · ACM SIGARCH Computer Architecture News · doi:10.1145/3007787.3001138 · Cited by 614
NetAdapt: Platform-Aware Neural Network Adaptation for Mobile Applications
Yang, T.-J., Howard, A., Chen, B., Zhang, X., Go, A., Sandler, M., Sze, V., & Adam, H. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01249-6_18 · Cited by 572
Self-citationTimeloop: A Systematic Approach to DNN Accelerator Evaluation
Parashar, A., Raina, P., Shao, Y. S., Chen, Y.-H., Ying, V. A., Mukkara, A., Venkatesan, R., Khailany, B., Keckler, S. W., & Emer, J. (2019) · 2019 IEEE International Symposium on Performance Analysis of Systems and Software (ISPASS) · doi:10.1109/ispass.2019.00042 · Cited by 571
Self-citationEyeriss: A Spatial Architecture for Energy-Efficient Dataflow for Convolutional Neural Networks
Chen, Y.-H., Emer, J., & Sze, V. (2016) · 2016 ACM/IEEE 43rd Annual International Symposium on Computer Architecture (ISCA) · doi:10.1109/isca.2016.40 · Cited by 534
Self-citationIn-Memory Computing: Advances and Prospects
Verma, N., Jia, H., Valavi, H., Tang, Y., Ozatay, M., Chen, L.-Y., Zhang, B., & Deaville, P. (2019) · IEEE Solid-State Circuits Magazine · doi:10.1109/mssc.2019.2922889 · Cited by 473
TETRIS: Scalable and Efficient Neural Network Acceleration with 3D Memory
Gao, M., Pu, J., Yang, X., Horowitz, M., & Kozyrakis, C. (2017) · Proceedings of the Twenty-Second International Conference on Architectural Support for Programming Languages and Operating Systems · doi:10.1145/3037697.3037702 · Cited by 421
Simba: Scaling Deep-Learning Inference with Multi-Chip-Module-Based Architecture
Shao, Y. S., Clemons, J., Venkatesan, R., Zimmer, B., Fojtik, M., Jiang, N., Keller, B., Klinefelter, A., Pinckney, N., Raina, P., et al. (2019) · Proceedings of the 52nd Annual IEEE/ACM International Symposium on Microarchitecture · doi:10.1145/3352460.3358302 · Cited by 406
Self-citationUnsupervised learning of hierarchical representations with convolutional deep belief networks
Lee, H., Grosse, R., Ranganath, R., & Ng, A. Y. (2011) · Communications of the ACM · doi:10.1145/2001269.2001295 · Cited by 388
Understanding Reuse, Performance, and Hardware Cost of DNN Dataflow: A Data-Centric Approach
Kwon, H., Chatarasi, P., Pellauer, M., Parashar, A., Sarkar, V., & Krishna, T. (2019) · Proceedings of the 52nd Annual IEEE/ACM International Symposium on Microarchitecture · doi:10.1145/3352460.3358252 · Cited by 301
MAERI: Enabling Flexible Dataflow Mapping over DNN Accelerators via Reconfigurable Interconnects
Kwon, H., Samajdar, A., & Krishna, T. (2018) · Proceedings of the Twenty-Third International Conference on Architectural Support for Programming Languages and Operating Systems · doi:10.1145/3173162.3173176 · Cited by 270
Accelergy: An Architecture-Level Energy Estimation Methodology for Accelerator Designs
Wu, Y. N., Emer, J. S., & Sze, V. (2019) · 2019 IEEE/ACM International Conference on Computer-Aided Design (ICCAD) · doi:10.1109/iccad45719.2019.8942149 · Cited by 241
Self-citationCnvlutin: Ineffectual-Neuron-Free Deep Neural Network Computing
Albericio, J., Judd, P., Hetherington, T., Aamodt, T., Jerger, N. E., & Moshovos, A. (2016) · 2016 ACM/IEEE 43rd Annual International Symposium on Computer Architecture (ISCA) · doi:10.1109/isca.2016.11 · Cited by 226
Using Dataflow to Optimize Energy Efficiency of Deep Neural Network Accelerators
Chen, Y., Emer, J., & Sze, V. (2017) · IEEE Micro · doi:10.1109/mm.2017.54 · Cited by 171
Self-citationFathom: reference workloads for modern deep learning methods
Adolf, R., Rama, S., Reagen, B., Wei, G.-Y., & Brooks, D. (2016) · 2016 IEEE International Symposium on Workload Characterization (IISWC) · doi:10.1109/iiswc.2016.7581275 · Cited by 169
Efficient Processing of Deep Neural Networks
Sze, V., Chen, Y., Yang, T.-J., & Emer, J. (2020) · Synthesis lectures on computer architecture · doi:10.1007/978-3-031-01766-7 · Cited by 162
Self-citationCamus, V., Mei, L., Enz, C., & Verhelst, M. (2019) · IEEE Journal on Emerging and Selected Topics in Circuits and Systems · doi:10.1109/jetcas.2019.2950386 · Cited by 110
Design Considerations for Efficient Deep Neural Networks on Processing-in-Memory Accelerators
Yang, T.-J., & Sze, V. (2019) · 2019 IEEE International Electron Devices Meeting (IEDM) · doi:10.1109/iedm19573.2019.8993662 · Cited by 55 · cited 1×
“Various works have proposed techniques for designing DNN models with hardware in the loop [22]–[25].”
MethodSelf-citation(2019) · IEEE Hot Chips Symposium · doi:10.1109/hotchips.2019.8875628 · Cited by 36
An Architecture-Level Energy and Area Estimator for Processing-In-Memory Accelerator Designs
Wu, Y. N., Sze, V., & Emer, J. (2020) · IEEE International Symposium on Performance Analysis of Systems and Software (ISPASS) · doi:10.1109/ispass48437.2020.00024 · Cited by 25
Self-citation
Research works printed but not linked: 5
- 0, NVIDIA NVLink High-speed GPU interconnect
- 2013, High-bandwidth Memory (HBM) DRAM
- lecun, 0, The MNIST Database of Handwritten Digits
- 0, Deepbench
- chen, 20, Introducing the CVPR 2018 On-device Visual Intelligence Challenge Google AI blog
Cited by
Edge artificial intelligence for 6g: vision, enabling technologies, and applications
Letaief, K. B., Shi, Y., Lu, J., & Lu, J. (2022) · IEEE Journal on Selected Areas in Communications · doi:10.1109/jsac.2021.3126076 · Cited by 829
“…a DNN model is highly dictated by the computation architecture and methods (e.g., ultra-low power compute-in-memory AI accelerator) at the edge computation nodes [249], the architecture of DNN models [250], and the wireless transmission for data exchange during the model inference procedure.”
MethodAn optical neural network using less than 1 photon per multiplication
Wang, T., Ma, S.-Y., Wright, L. G., Onodera, T., Richard, B. C., & McMahon, P. L. (2022) · Nature Communications · doi:10.1038/s41467-021-27774-8 · Cited by 268
Single chip photonic deep neural network with accelerated training
Bandyopadhyay, S., Sludds, A., Krastanov, S., Hamerly, R., Harris, N., Bunandar, D., Streshinsky, M., Hochberg, M., & Englund, D. (2022) · arXiv · doi:10.48550/arxiv.2208.01623 · Cited by 265
Lightweight Deep Learning for Resource-Constrained Environments: A Survey
Liu, H.-I., Galindo, M., Xie, H., Wong, L.-K., Shuai, H.-H., Li, Y.-H., & Cheng, W.-H. (2024) · ACM Computing Surveys · doi:10.1145/3657282 · Cited by 259
Single-chip photonic deep neural network with forward-only training
Bandyopadhyay, S., Sludds, A., Krastanov, S., Hamerly, R., Harris, N., Bunandar, D., Streshinsky, M., Hochberg, M., & Englund, D. (2024) · Nature Photonics · doi:10.1038/s41566-024-01567-z · Cited by 201
Metadata
- Authors
- Sze, V., Chen, Y.-H., Yang, T.-J., & Emer, J. S.
- Affiliations
- Massachusetts Institute of Technology (US)
- National Taiwan University (TW)
- Venue
- IEEE Solid-State Circuits Magazine
- Published
- 2020
- Volume
- 12
- Issue
- 3
- Pages
- 28-41
- ISSN
- 1943-0582 (ISSN-L)
- 1943-0590
- OpenAlex
- W3080976800
- Cited by
- 80
- References
- 34
- Citations by year
- 2026: 6 · 2025: 11 · 2024: 18 · 2023: 16 · 2022: 11 · 2021: 17 · 2020: 1
- FWCI
- 4.38
- Type
- journal article
- Language
- en
- Open access
- green
- hdl.handle.net
- Licence
- Locations
- IEEE Solid-State Circuits Magazine (publishedVersion)
- DSpace@MIT (Massachusetts Institute of Technology) (submittedVersion, cc-by-nc-sa)
- Funders
- Defense Advanced Research Projects Agency (HR0011-18-3-0007, NSF E2CDA 1639921)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Computer Science › Hardware and Architecture › Parallel Computing and Optimization Techniques
- Physical Sciences › Engineering › Electrical and Electronic Engineering › Advanced Memory and Neural Computing
- Keywords
- energy efficiency metrics, processor evaluation, scalability, hardware flexibility, special-purpose hardware
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.