Accelerating Deep Convolutional Networks using low-precision and sparsity
Open accessThe DOI matches its Crossref record; 14 of the 14 references it cites were found in a registry or an index.
Abstract
We explore techniques to significantly improve the compute efficiency and performance of Deep Convolution Networks without impacting their accuracy. To improve the compute efficiency, we focus on achieving high accuracy with extremely low-precision (2-bit) weight networks, and to accelerate the execution time, we aggressively skip operations on zero-values. We achieve the highest reported accuracy of 76.6% Top-1/93% Top-5 on the Imagenet object classification challenge with low-precision network while reducing the compute requirement by ~3× compared to a full-precision network that achieves similar accuracy. Furthermore, to fully exploit the benefits of our low-precision networks, we build a deep learning accelerator core, DLAC, that can achieve up to 1 TFLOP/mm2equivalent for single-precision floating-point operations (~2 TFLOP/mm2for half-precision), which is ~5× better than Linear Algebra Core [16] and ~4× better than previous deep learning accelerator proposal [8].
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“those accuracies are already comparable or higher than those reported on ternary weight networks42, for example 71.6% top-1 accuracy of ResNet-34 on ImageNet with first layer in FP3245.”
From Accurate deep neural network inference using computational phase-change memory (2020) · cites this paper as [45]
Citations
Cites
Very Deep Convolutional Networks for Large-Scale Image Recognition
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1409.1556 · Cited by 114,148 · cited 2×
“We first evaluate low-precision variant of VGG network on Cifar10 dataset, shown in Figure 3.”
MethodImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333 · cited 12×
“Imagenet [9] This dataset consists of 1.”
BackgroundInfluentialDelving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification
He, K., Zhang, X., Ren, S., & Sun, J. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.123 · Cited by 19,741 · cited 1×
“The accuracy of DNNs is rapidly improving (for example, Top-5 error for Imagenet object classification challenge [9] improved by >6× in 5 years) and is close to human-level accuracy in some cases [12].”
BackgroundIdentity mappings in deep residual networks
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46493-0_38 · Cited by 10,741
Identity Mappings in Deep Residual Networks
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · arXiv · doi:10.48550/arxiv.1603.05027 · Cited by 10,665
Han, S., Mao, H., & Dally, W. J. (2015) · arXiv · doi:10.48550/arxiv.1510.00149 · Cited by 4,662 · cited 2×
“In addition to the low-precision networks, DLAC can also accelerate pruned networks [11].”
BackgroundXNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networks
Rastegari, M., Ordonez, V., Redmon, J., & Farhadi, A. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46493-0_32 · Cited by 4,408 · cited 2×
“These include approaches that reduce the weights to half-precision [7], binary value [17], and ternary (2-bit) value [14].”
BackgroundMethodDeep Speech 2: End-to-End Speech Recognition in English and Mandarin
Amodei, D., Anubhai, R., Battenberg, E., Case, C. T., Casper, J., Catanzaro, B., Chen, J., Chrzanowski, M., Coates, A., Diamos, G., et al. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1512.02595 · Cited by 2,110 · cited 1×
“These include approaches that reduce the weights to half-precision [7], binary value [17], and ternary (2-bit) value [14].”
BackgroundDaDianNao: A Machine-Learning Supercomputer
Chen, Y., Luo, T., Liu, S., Zhang, S., He, L., Wang, J., Li, L., Chen, T., Xu, Z., Sun, N., et al. (2014) · 2014 47th Annual IEEE/ACM International Symposium on Microarchitecture · doi:10.1109/micro.2014.58 · Cited by 1,492 · cited 4×
“Also, the greater compute requirements make DNNs harder to deploy, which has led to a lot of interest recently in specialized hardware solutions, both commercially [3, 5] and in academia [10, 6, 8].”
BackgroundResultInfluentialEIE: Efficient Inference Engine on Compressed Deep Neural Network
Han, S., Liu, X., Mao, H., Pu, J., Pedram, A., Horowitz, M. A., & Dally, W. J. (2016) · 2016 ACM/IEEE 43rd Annual International Symposium on Computer Architecture (ISCA) · doi:10.1109/isca.2016.30 · Cited by 1,077
Research works printed but not linked: 5
- 0, Training and investigating residual nets
- 0, Tensor processing unit
- 0, Wave dataflow engine
- 0, CIFAR-10 Dataset
- 0, 92 45% on cifar-10 in torch
Cited by
Items 101–131 of 131
Fully parallel FPGA Implementation of an Artificial Neural Network Tuned by Genetic Algorithm
Dumesnil, E., Beaulieu, P.-O., & Boukadoum, M. (2018) · IEEE International New Circuits and Systems Conference (NEWCAS) · doi:10.1109/newcas.2018.8585580 · Cited by 4
“Moreover, many studies have shown that high precision may not be that important to build an efficient ANN [4], [5].”
Evolutionary Cell Aided Design for Neural Network ArchitecturesPreprint
Colangelo, P., Segal, O., Speicher, A., & Margala, M. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1903.02130 · Cited by 4
“Specifically, these reconfigurable architectures are a popular platform for both research and deployment due to their ability to change their fabric routing and resource structures to fit various workloads and optimizations by leveraging the resiliency of neural networks through low-numeric precision [20][21] and spar…”
BackgroundShimoda, M., Sato, S., & Nakahara, H. (2019) · IEICE Transactions on Information and Systems · doi:10.1587/transinf.2018rcp0005 · Cited by 4
Half-Precision Floating-Point Multiplier IP Core Based on 130 Nm CMOS ASIC Technology
Aguilera-Galicia, C. R., Longoria‐Gandara, O., & Pizano‐Escalante, L. (2018) · IEEE Latin-American Conference on Communications · doi:10.1109/latincom.2018.8613231 · Cited by 3
“For example, in the deep-learning field it has been demonstrated that a neural-network accelerator can be trained and implemented using half-precision floatingpoint (FP16) arithmetic [3], achieving high accuracy and performance in image classification, while reducing powerconsumption and computational requirements.”
MethodA Case Study on Optimizing Accurate Half Precision Average
Peou, K., Kelly, A., Falcou, J., & Germain‐Renaud, C. (2018) · International Symposium on Computer Architecture and High Performance Computing (SBAC-PAD) · doi:10.1109/cahpc.2018.8645923 · Cited by 3
“In addition, the field of machine learning is currently exploring methods to reduce the precision in many of their algorithms in order to increase performance [4] [14] [12] [2].”
MethodBarbosa, R. C., Rosa, R. L., da Silva, K. C. N., & Rodríguez, D. Z. (2020) · Brazilian Journal of Development · doi:10.34117/bjdv6n7-619 · Cited by 3
Towards Efficient Hardware Acceleration of Deep Neural Networks on FPGA
Li, S. (2018) · D-Scholarship@Pitt (University of Pittsburgh) · Cited by 3
Optimizing Memory Placement using Evolutionary Graph Reinforcement LearningPreprint
Khadka, S., Aflalo, E., Marder, M., Ben-David, A., Miret, S., Mannor, S., Hazan, T., Tang, H., & Majumdar, S. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2007.07298 · Cited by 7
“re that the solutions are robust to the real-world constraints and uncertainties of the chip. Complimentary to other approaches like compression [Cheng et al., 2018, Kim et al., 2015], sparsification [Venkatesh et al., 2017] and network pruning [Han et al., 2015], EGRL keeps the workload unchanged and instead tackles h…”
BackgroundEarly Prediction of DNN Activation Using Hierarchical Computations
Suresh, B., Pillai, K., Kalsi, G. S., Abuhatzera, A., & Subramoney, S. (2021) · Mathematics · doi:10.3390/math9233130 · Cited by 3
Identifying and Exploiting Ineffectual Computations to Enable Hardware Acceleration of Deep Learning
Moshovos, A., Albericio, J., Judd, P., Delmás, A., Sharify, S., Mahmoud, M., Hetherington, T., Nikolić, M., Stuart, D. M., Siu, K., et al. (2018) · IEEE International New Circuits and Systems Conference (NEWCAS) · doi:10.1109/newcas.2018.8585656 · Cited by 2
Increased Leverage of Transprecision Computing for Machine Vision Applications at the Edge
Minhas, U. I., Lee, J., Mukhanov, L., Karakonstantis, G., Vandierendonck, H., & Woods, R. (2022) · Journal of Signal Processing Systems · doi:10.1007/s11265-022-01784-1 · Cited by 4
“In some cases, arithmetic precision of aspects of the application have been reduced to 8-bits, 4-bits or even down to 2-bits [9–11].”
BackgroundCompiling Optimization for Neural Network Accelerators
Song, J. Y., Zhuang, Y., Chen, X., Zhi, T., & Liu, S. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-29611-7_2 · Cited by 2
Harnessing Tensor Decomposition for High-Dimensional Machine LearningPreprint
Rustik, E., Viktoriia, E., & Tatyana, A. (2025) · doi:10.36227/techrxiv.174417403.38431928/v1 · Cited by 2
Low-precision deep-learning-based automatic modulation recognition system
Kumar, S., Agarwal, A., Varshney, N., & Mahapatra, R. (2022) · ITU Journal on Future and Evolving Technologies · doi:10.52953/ctyj2699 · Cited by 1
Pham, D., & Lai, B.-C. C. (2020) · IET Circuits Devices & Systems · doi:10.1049/iet-cds.2019.0225 · Cited by 1
“Their results have been published in [14, 15, 42, 43].”
CHAMP: Channel Merging Process for Cost-Efficient Highly-Pruned CNN Acceleration
Kwon, H., Byun, Y., Kang, S., & Lee, Y. (2022) · IEEE Transactions on Circuits and Systems I Regular Papers · doi:10.1109/tcsi.2022.3174531 · Cited by 1
Power Awareness in Low Precision Neural Networks
Spingarn, N., Banner, R., Ben-Yaakov, H., Hoffer, E., & Michaeli, T. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-25082-8_5 · Cited by 3
Impact of Vectorization Over 16-bit Data-Types on GPUs
Reis, L., Nobre, R., & Cardoso, J. M. P. (2018) · doi:10.1145/3183767.3183777 · Cited by 1
“, [21]), and some image processing kernels (see, e.”
BackgroundRotateConv: Making Asymmetric Convolutional Kernels Rotatable
Ma, J., Guo, W., Wang, W., & Wang, L. (2018) · Proceedings - International Conference on Pattern Recognition/Proceedings/International Conference on Pattern Recognition · doi:10.1109/icpr.2018.8545623 · Cited by 1
“Weight pruning [12], [13] is a post processing method which deletes the small weight parameters after pre-training.”
MethodAn Optimization Technique for General Neural Network Hardware Architecture
Ravichandran, J., & Nair, R. K. R. (2020) · International Conference on Devices, Circuits and Systems (ICDCS) · doi:10.1109/icdcs48716.2020.243591 · Cited by 1
“[11] G Venkatesh, E Nurvitadhi, and D Marr.”
BackgroundMOSDA: On-Chip Memory Optimized Sparse Deep Neural Network Accelerator With Efficient Index Matching
Xu, H., Shiomi, J., & Onodera, H. (2020) · IEEE Open Journal of Circuits and Systems · doi:10.1109/ojcas.2020.3035402 · Cited by 2
“As a result, the matrices involved in DNN processing are often sparse [24].”
BackgroundAlgorithm/architecture solutions to improve beyond uniform quantization in embedded DNN accelerators
Pedram, A., Ardestani, A. S., Li, L., Abdel-Aziz, H. A., Fang, J., & Hassoun, J. (2022) · Journal of Systems Architecture · doi:10.1016/j.sysarc.2022.102454 · Cited by 1
Multiple-precision MLFMA for efficient and accurate solutions of broadband electromagnetic problems
Kalfa, M. (2020) · Bilkent University Institutional Repository (Bilkent University) · Cited by 1
Multi-Way Data Representation: A Comprehensive Survey on Tensor Decomposition in Machine LearningPreprint
Tatyana, A. (2025) · Engineering Archive (engrXiv) · doi:10.31224/4421 · Cited by 2
FPGA Architectures for Low Precision Machine Learning
Moss, D. J. M. (2017) · The Sydney eScholarship Repository (The University of Sydney)
Cai, R. (2020) · Digital Repository Service (Northeastern University) · doi:10.17760/d20383686 · Cited by 1
A review of data representation formats for intelligent computing with emphasis on POSITs
Balasubramani, S., & Jagadeesan, U. (2023) · AIP conference proceedings · doi:10.1063/5.0126236
“Ganesh Venkatesh et al.,[33] was able to successfully develop a 2 bit CNN with 76% top 1 and 93% top 5 accuracy also they designed a deep learning accelerator core with 1 TFLOP/mm2 [33].”
MethodFloating-Point Formats and Arithmetic for Highly Accurate Multi-Layer Perceptrons
Niknia, F., Wang, Z., Liu, S., Reviriego, P., Louri, A., & Lombardi, F. (2023) · IEEE International Conference on Nanotechnology (IEEE-NANO) · doi:10.1109/nano58406.2023.10231201
“To reduce computational complexity, low-precision formats (such as HP or SP) are extensively used in ANNs [6], [7].”
MethodNaik, P. K. (2024) · Journal of Electrical Systems · doi:10.52783/jes.1772
“These techniques execute object detection throughout various sections of the frame that were used for feature extraction [21].”
MethodCabanes, Q. (2021) · theses.fr (ABES) · doi:10.70675/48eb5418zc515z45fdza5faz645920a7699b · Cited by 1
The Gegelati Framework for Efficient and Reproducible Solutions with Tangled Program Graphs
Vacher, Q., Beuve, N., Dardaillon, M., & Desnos, K. (2026) · Genetic and evolutionary computation · doi:10.1007/978-981-95-6398-2_21
Metadata
- Authors
- Venkatesh, G., Nurvitadhi, E., & Marr, D.
- Affiliations
- Intel (United States) (US)
- Venue
- 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)
- Publisher
- IEEE
- Published
- 2017-03
- Pages
- 2861-2865
- ISSN
- 1520-6149 (ISSN-L)
- OpenAlex
- W2529546376
- Cited by
- 123
- References
- 13
- Citations by year
- 2026: 1 · 2024: 5 · 2023: 9 · 2022: 7 · 2021: 9 · 2020: 22 · 2019: 23 · 2018: 31 · 2017: 16
- FWCI
- 7.36
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- IEEE International Conference on Acoustics Speech and Signal Processing (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Engineering › Electrical and Electronic Engineering › Low-power high-performance VLSI design
- Physical Sciences › Engineering › Electrical and Electronic Engineering › Advanced Memory and Neural Computing
- Keywords
- deep convolutional neural networks, low-precision arithmetic, sparse computation, computational efficiency, deep learning accelerators, energy efficiency, image classification, ImageNet
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.