Accelerating Deep Convolutional Networks using low-precision and sparsity
Open accessThe DOI matches its Crossref record; 14 of the 14 references it cites were found in a registry or an index.
Abstract
We explore techniques to significantly improve the compute efficiency and performance of Deep Convolution Networks without impacting their accuracy. To improve the compute efficiency, we focus on achieving high accuracy with extremely low-precision (2-bit) weight networks, and to accelerate the execution time, we aggressively skip operations on zero-values. We achieve the highest reported accuracy of 76.6% Top-1/93% Top-5 on the Imagenet object classification challenge with low-precision network while reducing the compute requirement by ~3× compared to a full-precision network that achieves similar accuracy. Furthermore, to fully exploit the benefits of our low-precision networks, we build a deep learning accelerator core, DLAC, that can achieve up to 1 TFLOP/mm2equivalent for single-precision floating-point operations (~2 TFLOP/mm2for half-precision), which is ~5× better than Linear Algebra Core [16] and ~4× better than previous deep learning accelerator proposal [8].
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“those accuracies are already comparable or higher than those reported on ternary weight networks42, for example 71.6% top-1 accuracy of ResNet-34 on ImageNet with first layer in FP3245.”
From Accurate deep neural network inference using computational phase-change memory (2020) · cites this paper as [45]
Citations
Cites
Very Deep Convolutional Networks for Large-Scale Image Recognition
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1409.1556 · Cited by 114,148 · cited 2×
“We first evaluate low-precision variant of VGG network on Cifar10 dataset, shown in Figure 3.”
MethodImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333 · cited 12×
“Imagenet [9] This dataset consists of 1.”
BackgroundInfluentialDelving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification
He, K., Zhang, X., Ren, S., & Sun, J. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.123 · Cited by 19,741 · cited 1×
“The accuracy of DNNs is rapidly improving (for example, Top-5 error for Imagenet object classification challenge [9] improved by >6× in 5 years) and is close to human-level accuracy in some cases [12].”
BackgroundIdentity mappings in deep residual networks
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46493-0_38 · Cited by 10,741
Identity Mappings in Deep Residual Networks
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · arXiv · doi:10.48550/arxiv.1603.05027 · Cited by 10,665
Han, S., Mao, H., & Dally, W. J. (2015) · arXiv · doi:10.48550/arxiv.1510.00149 · Cited by 4,662 · cited 2×
“In addition to the low-precision networks, DLAC can also accelerate pruned networks [11].”
BackgroundXNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networks
Rastegari, M., Ordonez, V., Redmon, J., & Farhadi, A. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46493-0_32 · Cited by 4,408 · cited 2×
“These include approaches that reduce the weights to half-precision [7], binary value [17], and ternary (2-bit) value [14].”
BackgroundMethodDeep Speech 2: End-to-End Speech Recognition in English and Mandarin
Amodei, D., Anubhai, R., Battenberg, E., Case, C. T., Casper, J., Catanzaro, B., Chen, J., Chrzanowski, M., Coates, A., Diamos, G., et al. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1512.02595 · Cited by 2,110 · cited 1×
“These include approaches that reduce the weights to half-precision [7], binary value [17], and ternary (2-bit) value [14].”
BackgroundDaDianNao: A Machine-Learning Supercomputer
Chen, Y., Luo, T., Liu, S., Zhang, S., He, L., Wang, J., Li, L., Chen, T., Xu, Z., Sun, N., et al. (2014) · 2014 47th Annual IEEE/ACM International Symposium on Microarchitecture · doi:10.1109/micro.2014.58 · Cited by 1,492 · cited 4×
“Also, the greater compute requirements make DNNs harder to deploy, which has led to a lot of interest recently in specialized hardware solutions, both commercially [3, 5] and in academia [10, 6, 8].”
BackgroundResultInfluentialEIE: Efficient Inference Engine on Compressed Deep Neural Network
Han, S., Liu, X., Mao, H., Pu, J., Pedram, A., Horowitz, M. A., & Dally, W. J. (2016) · 2016 ACM/IEEE 43rd Annual International Symposium on Computer Architecture (ISCA) · doi:10.1109/isca.2016.30 · Cited by 1,077
Li, F., Liu, B., Wang, X., Zhang, B., & Yan, J. (2016) · arXiv · doi:10.48550/arxiv.1605.04711 · Cited by 732
Cnvlutin: Ineffectual-Neuron-Free Deep Neural Network Computing
Albericio, J., Judd, P., Hetherington, T., Aamodt, T., Jerger, N. E., & Moshovos, A. (2016) · 2016 ACM/IEEE 43rd Annual International Symposium on Computer Architecture (ISCA) · doi:10.1109/isca.2016.11 · Cited by 226
Neural Networks with Few Multiplications
Lin, Z., Courbariaux, M., Memisevic, R., & Bengio, Y. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1510.03009 · Cited by 141 · cited 1×
“In this paper, we build on recently proposed low-precision convolution networks [17, 15, 14] to reduce compute requirements of these networks.”
MethodCodesign Tradeoffs for High-Performance, Low-Power Linear Algebra Architectures
Pedram, A., Geijn, R. A., & Gerstlauer, A. (2012) · IEEE Transactions on Computers · doi:10.1109/tc.2012.132 · Cited by 57
Research works printed but not linked: 5
- 0, Training and investigating residual nets
- 0, Tensor processing unit
- 0, Wave dataflow engine
- 0, CIFAR-10 Dataset
- 0, 92 45% on cifar-10 in torch
Cited by
Pruning and quantization for deep neural network acceleration: A survey
Liang, T., Glossner, J., Wang, L., Shi, S., & Zhang, X. (2021) · Neurocomputing · doi:10.1016/j.neucom.2021.07.045 · Cited by 943
SCNN: An Accelerator for Compressed-sparse Convolutional Neural Networks
Parashar, A., Rhu, M., Mukkara, A., Puglielli, A., Venkatesan, R., Khailany, B., Emer, J., Keckler, S. W., & Dally, W. J. (2017) · Proceedings of the 44th Annual International Symposium on Computer Architecture · doi:10.1145/3079856.3080254 · Cited by 896
“The Deep Learning Accelerator Core (DLAC) architecture paper mentions that it can skip computation on zero-valued operands to improve performance, but the architecture does not appear to employ zero compression [32].”
BackgroundSCNN: An Accelerator for Compressed-sparse Convolutional Neural Networks
Parashar, A., Rhu, M., Mukkara, A., Puglielli, A., Venkatesan, R., Khailany, B., Emer, J., Keckler, S. W., & Dally, W. J. (2017) · ACM SIGARCH Computer Architecture News · doi:10.1145/3140659.3080254 · Cited by 413
Accurate deep neural network inference using computational phase-change memory
Joshi, V., Le Gallo, M., Haefeli, S., Boybat, I., Nandakumar, S. R., Piveteau, C., Dazzi, M., Rajendran, B., Sebastian, A., & Eleftheriou, E. (2020) · Nature Communications · doi:10.1038/s41467-020-16108-9 · Cited by 490
Can FPGAs Beat GPUs in Accelerating Next-Generation Deep Neural Networks?
Nurvitadhi, E., Venkatesh, G., Sim, J., Marr, D., Huang, R., Ong Gee Hock, J., Liew, Y. T., Srivatsan, K., Moss, D., Subhaschandra, S., et al. (2017) · Proceedings of the 2017 ACM/SIGDA International Symposium on Field-Programmable Gate Arrays · doi:10.1145/3020078.3021740 · Cited by 467
“Ternary DNNs (i.e., Ternary Weight Networks) [4,5] have recently proposed constraining neural network weights to +1, 0, or -1, allowing for weights to be represented with just 2 bits, while simultaneously introducing more sparsity to these weights.”
BackgroundMethodInfluentialSelf-citation
Metadata
- Authors
- Venkatesh, G., Nurvitadhi, E., & Marr, D.
- Affiliations
- Intel (United States) (US)
- Venue
- 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)
- Publisher
- IEEE
- Published
- 2017-03
- Pages
- 2861-2865
- ISSN
- 1520-6149 (ISSN-L)
- OpenAlex
- W2529546376
- Cited by
- 123
- References
- 13
- Citations by year
- 2026: 1 · 2024: 5 · 2023: 9 · 2022: 7 · 2021: 9 · 2020: 22 · 2019: 23 · 2018: 31 · 2017: 16
- FWCI
- 7.36
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- IEEE International Conference on Acoustics Speech and Signal Processing (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Engineering › Electrical and Electronic Engineering › Low-power high-performance VLSI design
- Physical Sciences › Engineering › Electrical and Electronic Engineering › Advanced Memory and Neural Computing
- Keywords
- deep convolutional neural networks, low-precision arithmetic, sparse computation, computational efficiency, deep learning accelerators, energy efficiency, image classification, ImageNet
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.