Model Compression and Hardware Acceleration for Neural Networks: A Comprehensive Survey
The DOI matches its Crossref record; 317 of the 334 references it cites were found in a registry or an index, and 1 are not checked yet.
Abstract
Domain-specific hardware is becoming a promising topic in the backdrop of improvement slow down for general-purpose processors due to the foreseeable end of Moore's Law. Machine learning, especially deep neural networks (DNNs), has become the most dazzling domain witnessing successful applications in a wide spectrum of artificial intelligence (AI) tasks. The incomparable accuracy of DNNs is achieved by paying the cost of hungry memory consumption and high computational complexity, which greatly impedes their deployment in embedded systems. Therefore, the DNN compression concept was naturally proposed and widely used for memory saving and compute acceleration. In the past few years, a tremendous number of compression techniques have sprung up to pursue a satisfactory tradeoff between processing efficiency and application accuracy. Recently, this wave has spread to the design of neural network accelerators for gaining extremely high performance. However, the amount of related works is incredibly huge and the reported approaches are quite divergent. This research chaos motivates us to provide a comprehensive survey on the recent advances toward the goal of efficient compression and execution of DNNs without significantly compromising accuracy, involving both the high-level algorithms and their applications in hardware design. In this article, we review the mainstream compression approaches such as compact model, tensor decomposition, data quantization, and network sparsification…
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“but there is no review discussing all three categories which in future will be integrated seamlessly to design effective and efficient DNNs in the new edge intelligence era.”
From Bringing ai to edge: from deep learning’s perspective (2022) · cites this paper as [28]
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · cited 3×
“Specifically, an example of the identity connection used in ResNet [30] [see Fig.”
BackgroundMethodVery Deep Convolutional Networks for Large-Scale Image Recognition
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1409.1556 · Cited by 114,148 · cited 1×
“In VGG-Nets [27], the interlayer topology is rather simple, where multiple Conv layers are stacked head-to-tail in sequence.”
BackgroundImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155 · cited 3×
“1078 [20] A.”
BackgroundInfluentialHochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318 · cited 2×
“…we discuss recent advances in RNNs that aim to improve the execution efficiency by designing compact RNNs in the level of unit or network. a) Unit level: LSTM is designed to improve the long-term dependence over vanilla RNNs by alleviating the issue of gradient explosion or vanishing [18], [46].”
BackgroundMethodLeCun, Y., Bengio, Y., & Hinton, G. (2015) · Nature · doi:10.1038/nature14539 · Cited by 84,126
Densely Connected Convolutional Networks
Huang, G., Liu, Z., Van Der Maaten, L., & Weinberger, K. Q. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.243 · Cited by 48,566
Going deeper with convolutions
Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., & Rabinovich, A. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298594 · Cited by 48,562
(2014) · Cited by 33,419
Rethinking the Inception Architecture for Computer Vision
Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., & Wojna, Z. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.308 · Cited by 32,138
MobileNetV2: Inverted Residuals and Linear Bottlenecks
Sandler, M., Howard, A., Zhu, M., Zhmoginov, A., & Chen, L.-C. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00474 · Cited by 28,755 · cited 35×
“The feature propagation gradually improves the abstraction level from local to global, which finally enables the network decision such as image recognition. a) Spatial correlation to enlarge RF: Compared to MLP, the key feature of CNNs is the local connections with shared weights, which results in high spatial correlat…”
BackgroundMethodInfluentialLearning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation
Cho, K., van Merrienboer, B., Gulcehre, C., Bahdanau, D., Bougares, F., Schwenk, H., & Bengio, Y. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1179 · Cited by 25,545 · cited 2×
“With the above concerns, GRU [19] is proposed and widely used as a simplified version of LSTM.”
MethodXception: Deep Learning with Depthwise Separable Convolutions
Chollet, F. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.195 · Cited by 20,380 · cited 2×
“(f) Depthwise-separable Conv [34]. that are much easier to train.”
BackgroundMethodMobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
Howard, A. G., Zhu, M., Chen, B., Kalenichenko, D., Wang, W., Weyand, T., Andreetto, M., & Adam, H. (2017) · arXiv · doi:10.48550/arxiv.1704.04861 · Cited by 19,015 · cited 1×
“Compared to the standard Conv, MobileNet [44] shows that using depthwise-separable Conv only reduces accuracy by 1% on ImageNet but saves lots of parameters and MACs.”
MethodAggregated Residual Transformations for Deep Neural Networks
Xie, S., Girshick, R., Dollar, P., Tu, Z., & He, K. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.634 · Cited by 12,465 · cited 3×
“Another method to alleviate the Conv overhead is group Conv [33].”
BackgroundMethodInfluentialTensor Decompositions and Applications
Kolda, T. G., & Bader, B. W. (2009) · SIAM Review · doi:10.1137/07070111x · Cited by 10,301 · cited 3×
“When every r i equals a positive integer r C and the kernel tensor K presents like a superdiagonal tensor, which means all elements in K are 0 except K ( x 1 , x 2 , . . . , x d ) with x 1 = x 2 = . . . = x d , then (19) becomes CP decomposition [98].”
BackgroundMethodShuffleNet: An Extremely Efficient Convolutional Neural Network for Mobile Devices
Zhang, X., Zhou, X., Lin, M., & Sun, J. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00716 · Cited by 9,982
ShuffleNet: An Extremely Efficient Convolutional Neural Network for Mobile Devices
Zhang, X., Zhou, X., Lin, M., & Sun, J. (2017) · arXiv · doi:10.48550/arxiv.1707.01083 · Cited by 9,982 · cited 2×
“ShuffleNet [37] overcomes it via the channel shuffle operation, which rearranges FMs and links channels among different groups.”
BackgroundLearning Transferable Architectures for Scalable Image Recognition
Zoph, B., Vasudevan, V., Shlens, J., & Le, Q. V. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00907 · Cited by 7,831
Multi-scale context aggregation by dilated convolutions
Yu, F., & Koltun, V. (2015) · arXiv · doi:10.48550/arxiv.1511.07122 · Cited by 7,738 · cited 1×
“Another way to enlarge RFs is to use atrous Conv [24], also known as dilated Conv [25].”
MethodDeformable Convolutional Networks
Dai, J., Qi, H., Xiong, Y., Li, Y., Zhang, G., Hu, H., & Wei, Y. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.89 · Cited by 7,513 · cited 1×
“Deformable Conv [26] is a generalization of atrous Conv, which augments the spatial sampling locations of kernels via additional 2-”
BackgroundZagoruyko, S., & Komodakis, N. (2016) · Procedings of the British Machine Vision Conference 2016 · doi:10.5244/c.30.87 · Cited by 7,244
Adversarial examples in the physical world
Kurakin, A., Goodfellow, I. J., & Bengio, S. (2018) · Artificial Intelligence Safety and Security · doi:10.1201/9781351251389-8 · Cited by 5,375
Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
Wu, Y., Schuster, M., Chen, Z., Le, Q. V., Norouzi, M., Macherey, W., Krikun, M., Cao, Y., Gao, Q., Macherey, K., et al. (2016) · arXiv · doi:10.48550/arxiv.1609.08144 · Cited by 5,319 · cited 1×
“Inspired by ResNet [30], Wu et al. [54] added residual connections into their eight-layer LSTM for machine translation.”
MethodLearning to forget: Continual prediction with LSTM
Gers, F. A., Schmidhuber, J., & Cummins, F. (2000) · Neural Computation · doi:10.1162/089976600300015015 · Cited by 4,823 · cited 1×
“…we discuss recent advances in RNNs that aim to improve the execution efficiency by designing compact RNNs in the level of unit or network. a) Unit level: LSTM is designed to improve the long-term dependence over vanilla RNNs by alleviating the issue of gradient explosion or vanishing [18], [46].”
BackgroundHan, S., Mao, H., & Dally, W. J. (2015) · arXiv · doi:10.48550/arxiv.1510.00149 · Cited by 4,662 · cited 4×
“…grains/structures, and methods are embedded into both of them. a) Weight pruning: Inspired by the pioneering works on weight pruning by LeCun et al. [226] and Hassibi and Stork [227], deep compression [6], [185] evidences that DNNs can still work well even if plenty of edges are pruned.”
BackgroundMethodInfluentialSelf-citationIn-Datacenter Performance Analysis of a Tensor Processing Unit
Jouppi, N. P., Young, C., Patil, N., Patterson, D., Agrawal, G., Bajwa, R., Bates, S., Bhatia, S., Boden, N., Borchers, A., et al. (2017) · Proceedings of the 44th Annual International Symposium on Computer Architecture · doi:10.1145/3079856.3080246 · Cited by 4,537 · cited 5×
“Conventional DNN accelerators solely optimize the hardware architecture to improve the compute parallelism and reduce memory accesses [8], [9].”
BackgroundMethodInfluentialXNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networks
Rastegari, M., Ordonez, V., Redmon, J., & Farhadi, A. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46493-0_32 · Cited by 4,408 · cited 5×
“For instance, the original XNOR -Net [157] with binary W and A suffers from 12.4% accuracy loss on AlexNet over ImageNet.”
BackgroundMethodInfluentialEfficient Processing of Deep Neural Networks: A Tutorial and Survey
Sze, V., Chen, Y.-H., Yang, T.-J., & Emer, J. S. (2017) · Proceedings of the IEEE · doi:10.1109/jproc.2017.2761740 · Cited by 4,215 · cited 2×
“The weights and psums propagate across Adapted from [14], [257], and [263].”
BackgroundMethodA Multilinear Singular Value Decomposition
De Lathauwer, L., De Moor, B., & Vandewalle, J. (2000) · SIAM Journal on Matrix Analysis and Applications · doi:10.1137/s0895479896305696 · Cited by 4,106 · cited 1×
“It is also called multilinear SVD (MLSVD) [113], because it is a special form of Tucker decomposition that all the factor matrices F ( i ) in (19) are orthogonal and thus the kernel tensor K is called all-orthogonal.”
BackgroundNeural Architecture Search with Reinforcement Learning
Zoph, B., & Le, Q. V. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1611.01578 · Cited by 3,974 · cited 9×
“Then these basic elements are stacked sequentially [64] with identity connections [45].”
BackgroundMethodInfluentialDARTS: Differentiable architecture search
Liu, H., Simonyan, K., & Yang, Y. (2018) · arXiv · doi:10.48550/arxiv.1806.09055 · Cited by 3,248 · cited 3×
“Similarly, one-shot NAS methods [59], [60], [74] focus on training a single super-net, from which small subnetworks directly inherit weights without additional training cost.”
BackgroundMethodEyeriss: An Energy-Efficient Reconfigurable Accelerator for Deep Convolutional Neural Networks
Chen, Y.-H., Krishna, T., Emer, J. S., & Sze, V. (2017) · IEEE Journal of Solid-State Circuits · doi:10.1109/jssc.2016.2616357 · Cited by 3,247 · cited 5×
“[261] Y.”
BackgroundMethodInfluential(1989) · Cited by 3,214
Long short-term memory recurrent neural network architectures for large scale acoustic modeling
Sak, H., Senior, A., & Beaufays, F. (2014) · Interspeech 2014 · doi:10.21437/interspeech.2014-80 · Cited by 2,973 · cited 1×
“Sak et al. [52] introduced a linear recurrent projection layer to reduce the dimension of hidden states, that is, r t = W r h t where the length of r t is smaller than that of h t , and feed the shorter r t − 1 back to replace the longer h t − 1 .”
BackgroundPruning Filters for Efficient ConvNets
Li, H., Kadav, A., Durdanovic, I., Samet, H., & Graf, H. P. (2016) · arXiv · doi:10.48550/arxiv.1608.08710 · Cited by 2,928 · cited 1×
“[200] further demonstrated filter pruning in CNNs by simply leveraging the mentioned group-wise magnitude comparison.”
MethodLearning Efficient Convolutional Networks through Network Slimming
Liu, Z., Li, J., Shen, Z., Huang, G., Yan, S., & Zhang, C. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.298 · Cited by 2,761
Channel Pruning for Accelerating Very Deep Neural Networks
He, Y., Zhang, X., & Sun, J. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.155 · Cited by 2,647 · cited 2×
“He et al. [206] solved a similar problem through a two-step algorithm that iteratively fixes weights to solve the sparsification mask via L 1 -norm LASSO regression and fixes the sparsification mask to update weights via least square reconstruction.”
BackgroundMethodOseledets, I. V. (2011) · SIAM Journal on Scientific Computing · doi:10.1137/090752286 · Cited by 2,636 · cited 1×
“…et al. [129] showed neural network applications, but they are yet to answer how to obtain an optimal HT form. e) Tensor train (TT): According to [130], the TT format of a d th-order tensor A ∈ R n 1 × n 2 × ... × n d can be represented as A ( j 1 , j 2 , . . . , where G k [ j k ] is a factor…”
BackgroundSearching for activation functions
Ramachandran, P., Zoph, B., & Le, Q. V. (2017) · arXiv · doi:10.48550/arxiv.1710.05941 · Cited by 2,335 · cited 1×
“…[58], [62]: standard Conv (e.g., C1 × 1, C3 × 3, C5 × 5), asymmetric Conv (e.g., C1 × 3 and C3 × 1, C1 × 7 and C7 × 1), depthwise-separable Conv (e.g., C3 × 3, C5 × 5), dilated Conv (e.g., 3 × 3); 2) pool: average pooling (e.g., 3 × 3), max pooling (e.g., 3 × 3); and 3) activation function [63].”
BinaryConnect: Training Deep Neural Networks with binary weights during propagations
Courbariaux, M., Bengio, Y., & David, J.-P. (2015) · arXiv · doi:10.48550/arxiv.1511.00363 · Cited by 2,180 · cited 4×
“In a nutshell, the heuristic methods are able to achieve sufficiently good accuracy on small or medium data sets such as MNIST, SVHN, and CIFAR10/CIFAR100 [150], [151], [168], [182].”
BackgroundMethodInfluentialCourbariaux, M., Hubara, I., Soudry, D., El-Yaniv, R., & Bengio, Y. (2016) · arXiv · doi:10.48550/arxiv.1602.02830 · Cited by 2,106
An Empirical Exploration of Recurrent Network Architectures
Józefowicz, R., Zaremba, W., & Sutskever, I. (2015) · International Conference on Machine Learning · Cited by 2,051
EIE: efficient inference engine on compressed deep neural network
Han, S., Liu, X., Mao, H., Pu, J., Pedram, A., Horowitz, M. A., & Dally, W. J. (2016) · ACM SIGARCH Computer Architecture News · doi:10.1145/3007787.3001163 · Cited by 1,937 · cited 3×
“[286] S.”
BackgroundMethodSelf-citationProgressive Neural Architecture Search
Liu, C., Zoph, B., Neumann, M., Shlens, J., Hua, W., Li, L.-J., Fei-Fei, L., Yuille, A., Huang, J., & Murphy, K. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01246-5_2 · Cited by 1,833
ThiNet: A Filter Level Pruning Method for Deep Neural Network Compression
Luo, J.-H., Wu, J., & Lin, W. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.541 · Cited by 1,832 · cited 4×
“For example, the quantized networks can be viewed as dense models with shorter bit-width [179]; structurally pruned networks (e.g., vector-wise in RNNs and FM-wise in CNNs) [204], [205], [232], [234] or decomposed networks [139] can be reassembled into smaller dense models to speed up the running.”
BackgroundMethodInfluentialDoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients
Zhou, S., Wu, Y., Ni, Z., Zhou, X., Wen, H., & Zou, Y. (2016) · arXiv · doi:10.48550/arxiv.1606.06160 · Cited by 1,690 · cited 8×
“For instance, the quantization of E can use a different transformation [147] Q where σ k is a random noise.”
BackgroundMethodInfluentialThe singular value decomposition: Its computation and some applications
Klema, V., & Laub, A. (1980) · IEEE Transactions on Automatic Control · doi:10.1109/tac.1980.1102314 · Cited by 1,631 · cited 1×
“Nevertheless, such solution incurs approximation error especially when k is small. b) Singular value decomposition (SVD): SVD [75] may be the best low-rank approximation of a matrix because its most information can be described by the singular values.”
BackgroundRethinking the Value of Network Pruning
Liu, Z., Sun, M., Zhou, T., Huang, G., & Darrell, T. (2018) · arXiv · doi:10.48550/arxiv.1810.05270 · Cited by 1,625 · cited 1×
“Interestingly, a recent work [247] gives a surprising observation that contradicts common beliefs: training a pruned network with sparse structure from scratch mostly performs better than the fine-tuned models.”
Understanding the Effective Receptive Field in Deep Convolutional Neural Networks
Luo, W., Li, Y., Urtasun, R., & Zemel, R. (2017) · arXiv · doi:10.48550/arxiv.1701.04128 · Cited by 1,575 · cited 1×
“Thus, designing effective RFs [21] is crucial to the representation capability of Conv layers, which is jointly determined by the weight kernel size and weight kernel pattern.”
Method(1993) · CaltechAUTHORS (California Institute of Technology) · Cited by 1,498
DaDianNao: A Machine-Learning Supercomputer
Chen, Y., Luo, T., Liu, S., Zhang, S., He, L., Wang, J., Li, L., Chen, T., Xu, Z., Sun, N., et al. (2014) · 2014 47th Annual IEEE/ACM International Symposium on Microarchitecture · doi:10.1109/micro.2014.58 · Cited by 1,492 · cited 4×
“Conventional DNN accelerators solely optimize the hardware architecture to improve the compute parallelism and reduce memory accesses [8], [9].”
BackgroundMethodInfluential(2018) · International Conference on Machine Learning · Cited by 1,465
Quantum-chemical insights from deep tensor neural networks
Schütt, K. T., Arbabzadah, F., Chmiela, S., Müller, K. R., & Tkatchenko, A. (2017) · Nature Communications · doi:10.1038/ncomms13890 · Cited by 1,445 · cited 1×
“Schütt et al. [109] used tensorized neural networks with low rank to learn molecular energies.”
BackgroundVlsi Digital Signal Processing Systems: Design And Implementation
Parhi, K. K. (2007) · Cited by 1,401
AMC: AutoML for Model Compression and Acceleration on Mobile Devices
He, Y., Lin, J., Liu, Z., Wang, H., Li, L.-J., & Han, S. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01234-2_48 · Cited by 1,400 · cited 5×
“For example, the weight pruning on MobileNet can achieve only 50%–60% sparsity without paying significant accuracy loss [198], [214].”
BackgroundMethodInfluentialSelf-citationCichocki, A., Mandic, D., De Lathauwer, L., Zhou, G., Zhao, Q., Caiafa, C., & PHAN, H. A. (2015) · IEEE Signal Processing Magazine · doi:10.1109/msp.2013.2297439 · Cited by 1,384 · cited 3×
“Hence, CP is typically used for interpreting data components while Tucker is often used for data compression [99].”
BackgroundMethodPruning Convolutional Neural Networks for Resource Efficient Inference
Molchanov, P. A., Tyree, S., Karras, T., Aila, T., & Kautz, J. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1611.06440 · Cited by 1,239
Chi, P., Li, S., Xu, C., Zhang, T., Zhao, J., Liu, Y., Wang, Y., & Xie, Y. (2016) · ACM SIGARCH Computer Architecture News · doi:10.1145/3007787.3001140 · Cited by 1,231 · cited 2×
“Taking memristor (e.g., RRAM [268], PCRAM [252]) as an example, the MVM can be performed in the analog domain.”
Self-citationMicikevicius, P., Narang, S., Alben, J., Diamos, G., Elsen, E., Garcia, D., Ginsburg, B., Houston, M., Kuchaiev, O., Venkatesh, G., et al. (2017) · arXiv · doi:10.48550/arxiv.1710.03740 · Cited by 1,179 · cited 2×
“Aiming at the training quantization, some works push the multiplications to 16-bit floating-point numbers [177] or 16-bit fixed-point numbers [179].”
BackgroundMethodEquivalent-accuracy accelerated neural-network training using analogue memory
Ambrogio, S., Narayanan, P., Tsai, H., Shelby, R. M., Boybat, I., di Nolfo, C., Sidler, S., Giordano, M., Bodini, M., Farinha, N. C. P., et al. (2018) · Nature · doi:10.1038/s41586-018-0180-5 · Cited by 1,151 · cited 2×
“Taking memristor (e.g., RRAM [268], PCRAM [252]) as an example, the MVM can be performed in the analog domain.”
MethodDeep Gradient Compression: Reducing the Communication Bandwidth for Distributed Training
Lin, Y., Han, S., Mao, H., Wang, Y., & Dally, W. J. (2017) · arXiv · doi:10.48550/arxiv.1712.01887 · Cited by 1,134 · cited 1×
“Besides the weight itself, the weight gradient can also be pruned using a similar magnitude-aware pruning [203].”
MethodSelf-citationDeep Complex NetworksPreprint
Trabelsi, C., Bilaniuk, O., Zhang, Y., Serdyuk, D., Subramanian, S., Santos, J. F., Mehri, S., Rostamzadeh, N., Bengio, Y., & Pal, C. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1705.09792 · Cited by 1,133 · cited 1×
“Additionally, using complex number shows potential to improve the expression of neural networks in recent years [93], so the explanation of a complex neuron with multiple inputs and a single output based on the QR decomposition [94] deserves more attention.”
BackgroundEyeriss v2: A Flexible Accelerator for Emerging Deep Neural Networks on Mobile Devices
Chen, Y.-H., Yang, T.-J., Emer, J. S., & Sze, V. (2019) · IEEE Journal on Emerging and Selected Topics in Circuits and Systems · doi:10.1109/jetcas.2019.2910232 · Cited by 1,106 · cited 2×
“A variety of compression techniques have been considered in the design of DNN accelerators at the cost of certain accuracy loss [10]–[13].”
BackgroundTowards artificial general intelligence with hybrid Tianjic chip architecture
Pei, J., Deng, L., Song, S., Zhao, M., Zhang, Y., Wu, S., Wang, G., Zou, Z., Wu, Z., He, W., et al. (2019) · Nature · doi:10.1038/s41586-019-1424-8 · Cited by 1,082 · cited 2×
“The complete architecture is similar to the spatial architectures in [264]–[267] (see Fig.”
BackgroundSelf-citationSoft Filter Pruning for Accelerating Deep Convolutional Neural Networks
He, Y., Kang, G., Dong, X., Fu, Y., & Yang, Y. (2018) · Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2018/309 · Cited by 1,020 · cited 1×
“Different from the hard pruning, the pruned filters can also be invoked during gradient descent, termed soft filter pruning [212].”
MethodHAQ: Hardware-Aware Automated Quantization With Mixed Precision
Wang, K., Liu, Z., Lin, Y., Lin, J., & Han, S. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00881 · Cited by 978 · cited 5×
“In addition, RL can be applied in data quantization to explore flexible bitwidth configuration across layers for accuracy improvement under certain hardware resource constraints [176].”
BackgroundMethodInfluentialSelf-citationIndependently Recurrent Neural Network (IndRNN): Building A Longer and Deeper RNN
Li, S., Li, W., Cook, C., Zhu, C., & Gao, Y. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00572 · Cited by 972
An Analysis of Deep Neural Network Models for Practical Applications
Canziani, A., Paszke, A., & Culurciello, E. (2016) · arXiv · doi:10.48550/arxiv.1605.07678 · Cited by 969
cuDNN: Efficient Primitives for Deep Learning
Chetlur, S., Woolley, C., Vandermersch, P., Cohen, J., Tran, J. J., Catanzaro, B., & Shelhamer, E. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1410.0759 · Cited by 943 · cited 1×
“Note that the Conv computation can be commonly converted to the modality of general matrix multiplication (GEMM) by lowering the feature and weight tensors to matrices [196].”
MethodSrivastava, R. K., Greff, K., & Schmidhuber, J. (2015) · arXiv · doi:10.48550/arxiv.1505.00387 · Cited by 912 · cited 3×
“[28] R.”
BackgroundCondenseNet: An Efficient DenseNet Using Learned Group Convolutions
Huang, G., Liu, S., Maaten, L. V. D., & Weinberger, K. Q. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00291 · Cited by 904
SCNN: An Accelerator for Compressed-sparse Convolutional Neural Networks
Parashar, A., Rhu, M., Mukkara, A., Puglielli, A., Venkatesan, R., Khailany, B., Emer, J., Keckler, S. W., & Dally, W. J. (2017) · Proceedings of the 44th Annual International Symposium on Computer Architecture · doi:10.1145/3079856.3080254 · Cited by 896 · cited 2×
“Besides the output stationary dataflow [257], we can also see architectures with input stationary dataflow [262] or weight stationary dataflow [9], as shown in Fig.”
MethodZhu, C., Han, S., Mao, H., & Dally, W. J. (2016) · arXiv · doi:10.48550/arxiv.1612.01064 · Cited by 865 · cited 5×
“Whereas, on larger data sets such as ImageNet, the optimization methods often perform better [162], [165].”
BackgroundMethodInfluentialSelf-citationPredicting Parameters in Deep Learning
Denil, M., Shakibi, B., Dinh, L., Ranzato, M., & de Freitas, N. (2013) · arXiv · doi:10.48550/arxiv.1306.0543 · Cited by 853 · cited 1×
“Whereas, this naive approach cannot achieve satisfactory accuracy in practice as pointed out in [79].”
BackgroundOnce-for-All: Train One Network and Specialize it for Efficient Deployment
Cai, H., Gan, C., Wang, T., Zhang, Z., & Han, S. (2019) · arXiv · doi:10.48550/arxiv.1908.09791 · Cited by 845 · cited 3×
“This problem is addressed in [72] by introducing techniques to train a single network (i.e., OFA network), which can be used under diverse architectural configurations, thereby amortizing the total training cost.”
BackgroundMethodSelf-citationAccelerating Very Deep Convolutional Networks for Classification and Detection
Zhang, X., Zou, J., He, K., & Sun, J. (2016) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2015.2502579 · Cited by 832 · cited 1×
“For example, Zhang et al. [88], [89] proposed a new structure based on SVD that splits one Conv weight kernel into two subkernels which is widely known as filter groups.”
BackgroundCheng, Y., Wang, D., Zhou, P., & Zhang, T. (2018) · IEEE Signal Processing Magazine · doi:10.1109/msp.2017.2765695 · Cited by 817 · cited 1×
“…without compression [14], a certain category of compression approaches [15], or a rapid overview of multiple categories on only the algorithm side [16], [17], we present a more systematic summary on most existing methodologies, show more detailed descriptions and comparisons for the individual or…”
MethodTo prune, or not to prune: exploring the efficacy of pruning for model compression
Zhu, M., & Gupta, S. (2017) · arXiv · doi:10.48550/arxiv.1710.01878 · Cited by 805 · cited 4×
“With a similar pruning method, researchers prove that the pruned large sparse models including CNNs and RNNs consistently show higher accuracy than their small dense counterparts [198], [228] and the sparse ones trained from scratch [229] under the condition of identical memory footprint.”
BackgroundMethodInfluentialNISP: Pruning Networks Using Neuron Importance Score Propagation
Yu, R., Li, A., Chen, C.-F., Lai, J.-H., Morariu, V. I., Han, X., Gao, M., Lin, C.-Y., & Davis, L. S. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00958 · Cited by 794
Convolutional networks for fast, energy-efficient neuromorphic computing
Esser, S. K., Merolla, P. A., Arthur, J. V., Cassidy, A. S., Appuswamy, R., Andreopoulos, A., Berg, D. J., McKinstry, J. L., Melano, T., Barch, D. R., et al. (2016) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1604850113 · Cited by 781 · cited 5×
“Sometimes, the binary levels of {− 1 , 1 } can be replaced with { 0 , 1 } [159].”
BackgroundMethodInfluentialQuantizing deep convolutional networks for efficient inference: A whitepaper
Krishnamoorthi, R. (2018) · arXiv · doi:10.48550/arxiv.1806.08342 · Cited by 753 · cited 1×
“Moreover, Tensorflow provides a quantization library for quantization-aware training and inference [307].”
BackgroundLi, F., Liu, B., Wang, X., Zhang, B., & Yan, J. (2016) · arXiv · doi:10.48550/arxiv.1605.04711 · Cited by 732
Designing Energy-Efficient Convolutional Neural Networks Using Energy-Aware Pruning
Yang, T.-J., Chen, Y.-H., & Sze, V. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.643 · Cited by 732 · cited 2×
“Different from previous work considering only the compression ratio and the accuracy loss, the energy reduction is also incorporated to determine the specific pruning strategy [231].”
BackgroundMethodStructured Pruning of Deep Convolutional Neural Networks
Anwar, S., Hwang, K., & Sung, W. (2017) · ACM Journal on Emerging Technologies in Computing Systems · doi:10.1145/3005348 · Cited by 698 · cited 1×
“Anwar et al. [235] forced an identical nonzero map on all outgoing weight kernels of each FM and determine the maps the evolutionary particle filtering approach.”
BackgroundLQ-Nets: Learned Quantization for Highly Accurate and Compact Deep Neural Networks
Zhang, D., Yang, J., Ye, D., & Hua, G. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01237-3_23 · Cited by 693
Sparse Convolutional Neural Networks
Liu, B., Wang, M., Foroosh, H., Tappen, M., & Penksy, M. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298681 · Cited by 680 · cited 3×
“For example, Liu et al. [236] sparsified the parameters in a similar way based on tensor-decomposed models; Choi et al. [221] added a similar regularization onto each weight kernel in both spatial and Winograd domains; Lin et al. [222] introduced the kernel-level scaling factors (termed synaptic…”
BackgroundMethodA Real-Time Algorithm for Signal Analysis with the Help of the Wavelet Transform
Holschneider, M., Kronland-Martinet, R., Morlet, J., & Tchamitchian, P. (1989) · Inverse Problems and Theoretical Imaging · doi:10.1007/978-3-642-97177-8_28 · Cited by 633
TernGrad: Ternary Gradients to Reduce Communication in Distributed Deep Learning
Wen, W., Xu, C., Yan, F., Wu, C., Wang, Y., Chen, Y., & Li, H. H. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1705.07878 · Cited by 622 · cited 2×
“Thanks to SGD’s tolerance of noise, the extreme quantization of G is feasible for reducing the communication bandwidth in distributed training systems [167].”
BackgroundMethodCnvlutin: ineffectual-neuron-free deep neural network computing
Albericio, J., Judd, P., Hetherington, T., Aamodt, T., Jerger, N. E., & Moshovos, A. (2016) · ACM SIGARCH Computer Architecture News · doi:10.1145/3007787.3001138 · Cited by 614 · cited 1×
“Furthermore, it is possible to remain input activations in the compressed format and the activation indexes are used to access only the required weights for each PE [283], [285], as shown in Fig.”
Cambricon-X: An accelerator for sparse neural networks
Zhang, S., Du, Z., Zhang, L., Lan, H., Liu, S., Li, L., Guo, Q., Chen, T., & Chen, Y. (2016) · 2016 49th Annual IEEE/ACM International Symposium on Microarchitecture (MICRO) · doi:10.1109/micro.2016.7783723 · Cited by 603 · cited 1×
“29 gives an implementation example adapted from Cambricon-X [281].”
MethodLow-rank matrix factorization for Deep Neural Network training with high-dimensional output targets
Sainath, T. N., Kingsbury, B., Sindhwani, V., Arisoy, E., & Ramabhadran, B. (2013) · 2013 IEEE International Conference on Acoustics, Speech and Signal Processing · doi:10.1109/icassp.2013.6638949 · Cited by 593 · cited 1×
“Similarly, one can replace r with a smaller k , which is referred as truncated SVD. Full-rank decomposition is an effective method for FC layer-based networks, especially between two layers with largely different amounts of neurons [76] or a sparse weight matrix with intrinsic low rank [77].”
MethodESE: Efficient Speech Recognition Engine with Sparse LSTM on FPGA
Han, S., Kang, J., Mao, H., Hu, Y., Li, X., Li, Y., Xie, D., Luo, H., Yao, S., Wang, Y., et al. (2017) · Proceedings of the 2017 ACM/SIGDA International Symposium on Field-Programmable Gate Arrays · doi:10.1145/3020078.3021745 · Cited by 591 · cited 2×
“Quantization and sparsification are the most widely combined ones to simultaneously reduce the data precision and the number of operations for higher throughput and lower energy [13], [255], [284], [286]–[289].”
BackgroundMethodSelf-citationNovikov, A., Podoprikhin, D., Osokin, A., & Vetrov, D. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1509.06569 · Cited by 558 · cited 4×
“However, it is worth noting that for the tensor decomposition cases [4], [121], some abnormal operations are induced (e.”
BackgroundMethodInfluentialEfficient Architecture Search by Network Transformation
Cai, H., Chen, T., Zhang, W., Yu, Y., & Wang, J. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.11709 · Cited by 557 · cited 1×
“…NAS can also be improved by exploring the architecture space with network transformation operations, starting from an existing network and reusing the weights [68]–[70]. c) Quality evaluation: To guide the search process, NAS methods need to get the quality of sampled neural network architectures.”
BackgroundBit Fusion: Bit-Level Dynamically Composable Architecture for Accelerating Deep Neural Network
Sharma, H., Park, J., Suda, N., Lai, L., Chau, B., Kim, J. K., Chandra, V., & Esmaeilzadeh, H. (2018) · 2018 ACM/IEEE 45th Annual International Symposium on Computer Architecture (ISCA) · doi:10.1109/isca.2018.00069 · Cited by 543 · cited 1×
“Two ways are utilized in this context: bit-serial [12], [278] and bit-decomposed [279], which are depicted in Figs.”
MethodPractical Block-Wise Neural Network Architecture Generation
Zhong, Z., Yan, J., Wu, W., Shao, J., & Liu, C.-L. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00257 · Cited by 522 · cited 2×
“9(b)] that can be stacked sequentially to construct neural networks [58], [65], rather than to directly find the neural network architecture as a whole.”
BackgroundMethodEfficient Neural Architecture Search via Parameter Sharing
Pham, H., Guan, M. Y., Zoph, B., Le, Q. V., & Dean, J. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1802.03268 · Cited by 517 · cited 2×
“Similarly, one-shot NAS methods [59], [60], [74] focus on training a single super-net, from which small subnetworks directly inherit weights without additional training cost.”
BackgroundMethodHierarchical Singular Value Decomposition of Tensors
Grasedyck, L. (2010) · SIAM Journal on Matrix Analysis and Applications · doi:10.1137/090764189 · Cited by 508
Deep Learning with Low Precision by Half-Wave Gaussian Quantization
Cai, Z., He, X., Sun, J., & Vasconcelos, N. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.574 · Cited by 498 · cited 1×
“Cai et al. [160] maintained the binarization of W while extending the A quantization to multiple bits in the positive region and approximated the quantization function via modified ReLU to help the gradient-based BP. Li et al. [181] and Deng et al. [151] extrapolated the quantization of both W and A…”
MethodA Systematic DNN Weight Pruning Framework Using Alternating Direction Method of Multipliers
Zhang, T., Ye, S., Zhang, K., Tang, J., Wen, W., Fardad, M., & Wang, Y. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01237-3_12 · Cited by 473 · cited 2×
“Recently, the ADMM method mentioned in Section III-D has also been used for high-accuracy sparsification [7], [223]–[225].”
MethodPredicting Parameters in Deep Learning
Shakibi, B. (2014) · University of British Columbia · doi:10.14288/1.0165555 · Cited by 462
Towards Accurate Binary Convolutional Neural Network
Lin, X., Zhao, C., & Pan, W. (2017) · arXiv · doi:10.48550/arxiv.1711.11294 · Cited by 453 · cited 3×
“Besides the basic quantization form in (23), the quantizer itself can also be learned with adaptive dynamic range and quantization levels for different layers [142]–[146].”
BackgroundMinerva: enabling low-power, highly-accurate deep neural network accelerators
Reagen, B., Whatmough, P., Adolf, R., Rama, S., Lee, H., Lee, S. K., Hernández-Lobato, J. M., Wei, G.-Y., & Brooks, D. (2016) · ACM SIGARCH Computer Architecture News · doi:10.1145/3007787.3001165 · Cited by 446 · cited 2×
“The zero input detection can be aggressively extended to small inputs [284].”
BackgroundFINN- R: An End-to-End Deep-Learning Framework for Fast Exploration of Quantized Neural Networks
Blott, M., Preußer, T. B., Fraser, N. J., Gambardella, G., O’brien, K., Umuroglu, Y., Leeser, M., & Vissers, K. (2018) · ACM Transactions on Reconfigurable Technology and Systems · doi:10.1145/3242897 · Cited by 446
Incremental Network Quantization: Towards Lossless CNNs with Low-Precision Weights
Zhou, A., Yao, A., Guo, Y., Xu, L., & Chen, Y. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1702.03044 · Cited by 439 · cited 2×
“Zhou et al. [172] combined the proposed incremental weight quantization and prior dynamic surgery based pruning in [249], which increases the 6.4 × compression ratio of the sole quantization on AlexNet to 53 × without compromising accuracy.”
BackgroundMethodInfluentialPUMA: A Programmable Ultra-efficient Memristor-based Accelerator for Machine Learning Inference
Ankit, A., Hajj, I. E., Chalamalasetti, S. R., Ndu, G., Foltin, M., Williams, R. S., Faraboschi, P., Hwu, W.-M. W., Strachan, J. P., Roy, K., et al. (2019) · Proceedings of the Twenty-Fourth International Conference on Architectural Support for Programming Languages and Operating Systems · doi:10.1145/3297858.3304049 · Cited by 424 · cited 4×
“Besides the MVM operation, some other operations such as scalar and vector operations should be additionally supported by PEs since they are necessary for some neural network models such as RNNs but cannot be handled by the memristor array efficiently [258].”
MethodInfluentialAdaptive dropout for training deep neural networks
Ba, J. L., & Frey, B. J. (2013) · neural information processing systems · Cited by 416
SCNN: An Accelerator for Compressed-sparse Convolutional Neural Networks
Parashar, A., Rhu, M., Mukkara, A., Puglielli, A., Venkatesan, R., Khailany, B., Emer, J., Keckler, S. W., & Dally, W. J. (2017) · ACM SIGARCH Computer Architecture News · doi:10.1145/3140659.3080254 · Cited by 413
Simba: Scaling Deep-Learning Inference with Multi-Chip-Module-Based Architecture
Shao, Y. S., Clemons, J., Venkatesan, R., Zimmer, B., Fojtik, M., Jiang, N., Keller, B., Klinefelter, A., Pinckney, N., Raina, P., et al. (2019) · Proceedings of the 52nd Annual IEEE/ACM International Symposium on Microarchitecture · doi:10.1145/3352460.3358302 · Cited by 406 · cited 2×
“The complete architecture is similar to the spatial architectures in [264]–[267] (see Fig.”
BackgroundLearning Structured Sparsity in Deep Neural Networks
Wen, W. X., Wu, C., Wang, Y., Chen, Y., & Li, H. H. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1608.03665 · Cited by 402 · cited 6×
“In [139], the SSL weight pruning proposed in [201] is successfully conducted on the predecomposed model.”
BackgroundMethodInfluentialLearning to Quantize Deep Networks by Optimizing Quantization Intervals With Task Loss
Jung, S., Son, C., Lee, S., Son, J., Han, J.-J., Kwak, Y., Hwang, S. J., & Choi, C. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00448 · Cited by 385 · cited 6×
“Besides the basic quantization form in (23), the quantizer itself can also be learned with adaptive dynamic range and quantization levels for different layers [142]–[146].”
BackgroundMethodInfluentialShift: A Zero FLOP, Zero Parameter Alternative to Spatial Convolutions
Wu, B., Wan, A., Yue, X., Jin, P., Zhao, S., Golmant, N., Gholaminejad, A., Gonzalez, J., & Keutzer, K. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00951 · Cited by 381
Designing Neural Network Architectures using Reinforcement Learning
Baker, B., Gupta, O., Naik, N., & Raskar, R. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1611.02167 · Cited by 372 · cited 2×
“…generation process as a Markov decision process, treat the validation accuracy of the sampled architecture as the reward, and update the architecture generation model using RL algorithms, including Q -learning [64], [65], policy-based REINFORCE [45], and proximal policy optimization (PPO) [58].”
BackgroundMethodNeuFlow: A runtime reconfigurable dataflow processor for vision
Farabet, C., Martini, B., Corda, B., Akselrod, P., Culurciello, E., & LeCun, Y. (2011) · CVPR 2011 WORKSHOPS · doi:10.1109/cvprw.2011.5981829 · Cited by 369 · cited 2×
“Due to the limited space, we just review the recent accelerators that can support large-scale neural networks and ignore the early ones [259], [260].”
BackgroundNeural Cache: Bit-Serial In-Cache Acceleration of Deep Neural Networks
Eckert, C., Wang, X., Wang, J., Subramaniyan, A., Iyer, R., Sylvester, D., Blaaauw, D., & Das, R. (2018) · 2018 ACM/IEEE 45th Annual International Symposium on Computer Architecture (ISCA) · doi:10.1109/isca.2018.00040 · Cited by 369 · cited 1×
“Besides eNVM devices, traditional memories such as SRAM [270], DRAM [271], and Flash [272] can also be modified to support the PIM-fashion processing of neural networks.”
Minimal gated unit for recurrent neural networks
Zhou, G.-B., Wu, J., Zhang, C.-L., & Zhou, Z.-H. (2016) · International Journal of Automation and Computing · doi:10.1007/s11633-016-1006-2 · Cited by 357 · cited 1×
“Minimal gated unit (MGU) [40] couples the reset and update gates of GRU.”
BackgroundSMASH: One-Shot Model Architecture Search through HyperNetworks
Brock, A., Lim, T., Ritchie, J. M., & Weston, N. J. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1708.05344 · Cited by 351 · cited 1×
“Alternatively, the evaluation step can also be accelerated using Hypernet-work [73], which can directly generate weights of a neural network architecture without training it.”
MethodConvolutional neural networks with low-rank regularization
Tai, C., Xiao, T., Zhang, Y., Wang, X., & Weinan, E. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1511.06067 · Cited by 341 · cited 1×
“D matrix [78].”
A Real-Time Algorithm for Signal Analysis with the Help of the Wavelet Transform
Holschneider, M., Kronland-Martinet, R., Morlet, J., & Tchamitchian, P. (1990) · inverse problems and theoretical imaging · doi:10.1007/978-3-642-75988-8_28 · Cited by 339 · cited 1×
“Another way to enlarge RFs is to use atrous Conv [24], also known as dilated Conv [25].”
MethodUNPU: An Energy-Efficient Deep Neural Network Accelerator With Fully Variable Weight Bit Precision
Lee, J., Kim, C., Kang, S., Shin, D., Kim, S., & Yoo, H.-J. (2019) · IEEE Journal of Solid-State Circuits · doi:10.1109/jssc.2018.2865489 · Cited by 335 · cited 1×
“Two ways are utilized in this context: bit-serial [12], [278] and bit-decomposed [279], which are depicted in Figs.”
MethodStripes: Bit-serial deep neural network computing
Judd, P., Albericio, J., Hetherington, T., Aamodt, T. M., & Moshovos, A. (2016) · 2016 49th Annual IEEE/ACM International Symposium on Microarchitecture (MICRO) · doi:10.1109/micro.2016.7783722 · Cited by 333 · cited 1×
“Two ways are utilized in this context: bit-serial [12], [278] and bit-decomposed [279], which are depicted in Figs.”
MethodConvolutional Neural Networks using Logarithmic Data Representation
Miyashita, D., Lee, E. H., & Murmann, B. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1603.01025 · Cited by 329 · cited 1×
“The logarithmic distribution [148] is a typical nonuniform quantization with exponential variance on step length.”
BackgroundRevisiting the Nystrom Method for Improved Large-Scale Machine LearningPreprint
Gittens, A., & Mahoney, M. W. (2013) · arXiv (Cornell University) · doi:10.48550/arxiv.1303.1849 · Cited by 329 · cited 1×
“A number of research studies utilize the CUR decomposition [95] for machine-learning applications, especially a variant similar to CUR called Nyström method [96].”
MethodRethinking the Smaller-Norm-Less-Informative Assumption in Channel Pruning of Convolution Layers
Ye, J., Lu, X., Lin, Z., & Wang, J. Z. (2018) · arXiv · doi:10.48550/arxiv.1802.00124 · Cited by 328 · cited 2×
“Furthermore, the compression of very deep networks on large data sets (e.g., > 50 layers on ImageNet) has yet to be studied well (except for few works on quantization [149] and sparsification [209]).”
BackgroundMethodRegularized Evolution for Image Classifier Architecture Search
Real, E., Aggarwal, A., Huang, Y., & Le, Q. V. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.33014780 · Cited by 328
DRISA: a DRAM-based Reconfigurable In-Situ Accelerator
Li, S., Niu, D., Malladi, K. T., Zheng, H., Brennan, B., & Xie, Y. (2017) · Proceedings of the 50th Annual IEEE/ACM International Symposium on Microarchitecture · doi:10.1145/3123939.3123977 · Cited by 326
Self-citationInterleaved Group Convolutions
Zhang, T., Qi, G.-J., Xiao, B., & Wang, J. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.469 · Cited by 323 · cited 1×
“Group Conv is further studied in [39], where each Conv block is composed of multiple interleaved group convolutions (IGCs) and each of them has different permutation strategy and kernel size.”
Phased LSTM: Accelerating Recurrent Network Training for Long or Event-based Sequences
Neil, D., Pfeiffer, M., & Liu, S. (2016) · Zurich Open Repository and Archive (University of Zurich) · doi:10.48550/arxiv.1610.09513 · Cited by 319 · cited 2×
“LSTM [43]. separately realized.”
BackgroundEfficient multi-objective neural architecture search via Lamarckian evolution
Elsken, T., Metzen, J. H., & Hutter, F. (2018) · arXiv · doi:10.48550/arxiv.1804.09081 · Cited by 311 · cited 1×
“…NAS can also be improved by exploring the architecture space with network transformation operations, starting from an existing network and reusing the weights [68]–[70]. c) Quality evaluation: To guide the search process, NAS methods need to get the quality of sampled neural network architectures.”
Background14.2 DNPU: An 8.1TOPS/W reconfigurable CNN-RNN processor for general-purpose deep neural networks
Shin, D., Lee, J., Lee, J., & Yoo, H.-J. (2017) · 2017 IEEE International Solid-State Circuits Conference (ISSCC) · doi:10.1109/isscc.2017.7870350 · Cited by 307
Training and Inference with Integers in Deep Neural Networks
Wu, S., Li, G., Chen, F., & Shi, L. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1802.04680 · Cited by 306 · cited 5×
“In a nutshell, the heuristic methods are able to achieve sufficiently good accuracy on small or medium data sets such as MNIST, SVHN, and CIFAR10/CIFAR100 [150], [151], [168], [182].”
BackgroundMethodInfluentialSelf-citationOn the importance of single directions for generalization
Morcos, A. S., Barrett, D. G. T., Rabinowitz, N. C., & Botvinick, M. (2018) · arXiv · doi:10.48550/arxiv.1803.06959 · Cited by 305 · cited 1×
“…algorithm [218] or L 0 -norm constrained gradient descent [219]. b) Neuron pruning: Neurons are also prunable by using the simple magnitude-aware rule (e.g., for A [237] or for E [202], [233]), according to the neuronal selectivity [238] or even based on random [239]/probabilistic [240] selection.”
MethodQuasi-Recurrent Neural Networks
Bradbury, J., Merity, S., Xiong, C., & Socher, R. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1611.01576 · Cited by 299 · cited 1×
“Quasi-RNN (QRNN) [49] replaces the previous hidden state h t − 1 with the previous input x t − 1 in the gate calculation, taking the forget gate as an example In this way, the heaviest calculation can be performed in parallel along both the temporal and mini-batch dimensions.”
MethodAxiom: A Householder-Parameterized Pure Unitary RNN for Long-Range Sequence Modeling
Chaudhary, S. (2026) · Zenodo (CERN European Organization for Nuclear Research) · doi:10.5281/zenodo.20108465 · Cited by 293
Aimar, A., Mostafa, H., Calabrese, E., Rios-Navarro, A., Tapiador-Morales, R., Lungu, I.-A., Milde, M. B., Corradi, F., Linares-Barranco, A., Liu, S.-C., et al. (2019) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2018.2852335 · Cited by 292
Ge, L., Liang, H., Yuan, J., & Thalmann, D. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.602 · Cited by 291 · cited 1×
“As the dimension of data increases continuously, Conv kernels might have higher orders such as in 3DCNN [140].”
BackgroundSimple Recurrent Units for Highly Parallelizable Recurrence
Lei, T., Zhang, Y., Wang, S. I., Dai, H., & Artzi, Y. (2018) · Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-1477 · Cited by 289 · cited 1×
“Inspired by QRNN, simple recurrent unit (SRU) [50] replaces Wh t − 1 with a lightweight Hadamard product of a vector-format parameter and the previous cellular state, that is, V (cid:2) c t − 1 , then the computation of c t becomes independent and parallelizable.”
MethodZhao, Q., Zhou, G., Xie, S., Zhang, L., & Cichocki, A. S. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1606.05535 · Cited by 284 · cited 1×
“Although TC seems to have a greater potential to obtain better quality than TT [118], [137], more research studies are needed to evidence this point in neural networks. h) TT in neural networks: In tensor network decomposition, the TT format is widely applied in neural networks.”
BackgroundSpeeding-up Convolutional Neural Networks Using Fine-tuned CP-Decomposition
Lebedev, V., Ganin, Y., Rakhuba, M., Oseledets, I. V., & Lempitsky, V. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1412.6553 · Cited by 283 · cited 1×
“Lebedev et al. [105] and Astrid and Lee [106] leveraged CP decomposition to compress Conv weight kernels into several subkernels to reduce the number of parameters and speed up the running.”
BackgroundJia, X., Song, S., He, W., Wang, Y., Rong, H., Zhou, F., Xie, L., Guo, Z., Yang, Y., Yu, L.-W., et al. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1807.11205 · Cited by 281 · cited 1×
“This is the reason that most quantization methods remain the master weight and its update in high precision [177]–[179], [193].”
BackgroundTensor-Train Recurrent Neural Networks for Video ClassificationPreprint
Yang, Y., Krompaß, D., & Tresp, V. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1707.01786 · Cited by 279 · cited 2×
“Note that in [124], only WWW x is compressed and an extreme compression ratio is demonstrated along with miraculous accuracy improvement.”
BackgroundMethodSo, D. R., Liang, C., & Le, Q. V. (2019) · arXiv · doi:10.48550/arxiv.1901.11117 · Cited by 277 · cited 1×
“Note that the automated NAS techniques can also be employed to design RNN [45], [59], [60] and transformer [61] models.”
MethodExtremely Low Bit Neural Network: Squeeze the Last Bit Out With ADMM
Leng, C., Dou, Z., Li, H., Zhu, S., & Jin, R. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.11713 · Cited by 275
Scalpel: Customizing DNN Pruning to the Underlying Hardware Parallelism
Yu, J., Lukefahr, A., Palframan, D., Dasika, G., Das, R., & Mahlke, S. (2017) · Proceedings of the 44th Annual International Symposium on Computer Architecture · doi:10.1145/3079856.3080215 · Cited by 272 · cited 5×
“For example, the block-wise weight sparsity is leveraged to optimize the running performance on general-purpose processors [13], [220], [234], and a more aggressive diagonal weight matrix is utilized in the accelerator design [282].”
InfluentialKalchbrenner, N., Danihelka, I., & Graves, A. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1507.01526 · Cited by 267 · cited 1×
“In grid LSTM [56], the LSTM cells are arranged in a 3-”
BackgroundProxylessNAS: Direct Neural Architecture Search on Target Task and Hardware
Cai, H., Zhu, L., & Han, S. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1812.00332 · Cited by 267 · cited 3×
“Similarly, one-shot NAS methods [59], [60], [74] focus on training a single super-net, from which small subnetworks directly inherit weights without additional training cost.”
BackgroundMethodSelf-citationEfficient and accurate approximations of nonlinear convolutional networks
Zhang, X., Zou, J., Ming, X., He, K., & Sun, J. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298809 · Cited by 262 · cited 1×
“For example, Zhang et al. [88], [89] proposed a new structure based on SVD that splits one Conv weight kernel into two subkernels which is widely known as filter groups.”
BackgroundShip Detection in Spaceborne Optical Image With SVD Networks
Zou, Z., & Shi, Z. (2016) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2016.2572736 · Cited by 257 · cited 1×
“The model accuracy sometimes even increases after removing the data redundancy [86] because the singular values and orthogonal bases may highlight potential inner relationships within the data [87].”
BackgroundYodaNN: An Architecture for Ultralow Power Binary-Weight CNN Acceleration
Andri, R., Cavigelli, L., Rossi, D., & Benini, L. (2018) · IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems · doi:10.1109/tcad.2017.2682138 · Cited by 247 · cited 1×
“If only the weights are binarized/ternarized, the costly MAC operations can be reduced to simpler accumulations [274].”
BackgroundGenerating trees and the Catalan and Schröder numbers
West, J. (1995) · Discrete Mathematics · doi:10.1016/0012-365x(94)00067-1 · Cited by 245 · cited 1×
“Although the HT format has flexible organization forms (because the count of different dimension trees with a fixed number of nodes is distributed as Catalan numbers [125]), it suffers from an intractable obstacle to obtain an optimal specific form.”
BackgroundFixed-point feedforward deep neural network design using weights +1, 0, and −1
Hwang, K., & Sung, W. (2014) · 2014 IEEE Workshop on Signal Processing Systems (SiPS) · doi:10.1109/sips.2014.6986082 · Cited by 244 · cited 1×
“The extensive rise of quantization for neural networks owes to the early binarization [5], [150], [156], [157], [159] and ternarization [161], [162], [187], [188].”
BackgroundHow to Train a Compact Binary Neural Network with High Accuracy?
Tang, W., Hua, G., & Wang, L. (2017) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v31i1.10862 · Cited by 241 · cited 3×
“Therefore, a small learning rate (e.g., 10 − 3 ) is widely adopted in quantized training, especially for extreme quantization such as binarization and ternarization [147], [158], [162].”
BackgroundMethodUltimate tensorization: compressing convolutional and FC layers alikePreprint
Garipov, T., Podoprikhin, D., Novikov, A., & Vetrov, D. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1611.03214 · Cited by 239 · cited 4×
“Table 5 enumerates recent TT compression on CNNs. Note that only Garipov et al. [121] extended tensorizing to Conv.”
BackgroundMethodInfluentialNeST: A Neural Network Synthesis Tool Based on a Grow-and-Prune Paradigm
Dai, X., Yin, H., & Jha, N. K. (2019) · IEEE Transactions on Computers · doi:10.1109/tc.2019.2914438 · Cited by 233 · cited 1×
“Given a small seed network, NeST [217] combines the gradient-aware growing and magnitude-aware pruning, which can be used on both weights and neurons.”
MethodWRPN: Wide Reduced-Precision Networks
Mishra, A., Nurvitadhi, E., Cook, J. J., & Marr, D. (2017) · arXiv · doi:10.48550/arxiv.1709.01134 · Cited by 227 · cited 1×
“Although more bits [144], [147], [169], [182] and wider layers [170], [174] can help to improve accuracy, they will increase parameters and operations as well.”
BackgroundXiao, L., Bahri, Y., Sohl‐Dickstein, J., Schoenholz, S. S., & Pennington, J. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1806.05393 · Cited by 227
AutoPruner: An end-to-end trainable filter pruning method for efficient deep model inference
Luo, J.-H., & Wu, J. (2020) · Pattern Recognition · doi:10.1016/j.patcog.2020.107461 · Cited by 221
SMASH: One-Shot Model Architecture Search through HyperNetworks
Brock, A., Lim, T., Ritchie, J. M., & Weston, N. J. (2018) · International Conference on Learning Representations · Cited by 217
A linear speedup analysis of distributed deep learning with sparse and quantized communication
Jiang, P., & Agrawal, G. (2018) · neural information processing systems · Cited by 216
Zhou, X., Du, Z., Guo, Q., Liu, S., Liu, C., Wang, C., Zhou, X., Li, L., Chen, T., & Chen, Y. (2018) · 2018 51st Annual IEEE/ACM International Symposium on Microarchitecture (MICRO) · doi:10.1109/micro.2018.00011 · Cited by 214 · cited 4×
“A variety of compression techniques have been considered in the design of DNN accelerators at the cost of certain accuracy loss [10]–[13].”
BackgroundMethodInfluentialYin, S., Ouyang, P., Tang, S., Tu, F., Li, X., Zheng, S., Lu, T., Gu, J., Liu, L., & Wei, S. (2018) · IEEE Journal of Solid-State Circuits · doi:10.1109/jssc.2017.2778281 · Cited by 212 · cited 6×
“Adapted from [14], [257], and [263].”
BackgroundMethodInfluentialModel compression via distillation and quantization
Polino, A., Pascanu, R., & Alistarh, D. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1802.05668 · Cited by 212 · cited 3×
“For example, the dynamic range and quantization levels can be parameterized in different ways and trained using iterative optimization [143], [144] or gradient descent [142], [145], [146].”
BackgroundUnitary Evolution Recurrent Neural Networks
Arjovsky, M., Shah, A., & Bengio, Y. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1511.06464 · Cited by 210 · cited 1×
“Unitary RNN (uRNN) [51] explores the use of orthogonal and unitary weight matrices in memory-restricted conditions. b) Network level: Another path to explore compact RNNs is the simplification of stacked layers, which in many ways follows the similar inspirations as the channel correlation for CNNs…”
BackgroundA VLSI architecture for high-performance, low-cost, on-chip learning
Hammerstrom, D. (1990) · 1990 IJCNN International Joint Conference on Neural Networks · doi:10.1109/ijcnn.1990.137621 · Cited by 209 · cited 1×
“Due to the limited space, we just review the recent accelerators that can support large-scale neural networks and ignore the early ones [259], [260].”
BackgroundEnergy-Efficient Neural Network Accelerator Based on Outlier-Aware Low-Precision Computation
Park, E., Kim, D., & Yoo, S. (2018) · 2018 ACM/IEEE 45th Annual International Symposium on Computer Architecture (ISCA) · doi:10.1109/isca.2018.00063 · Cited by 201 · cited 1×
“Quantization and sparsification are the most widely combined ones to simultaneously reduce the data precision and the number of operations for higher throughput and lower energy [13], [255], [284], [286]–[289].”
BackgroundTianjic: A Unified and Scalable Chip Bridging Spike-Based and Continuous Neural Computation
Deng, L., Wang, G., Li, G., Li, S., Liang, L., Zhu, M., Wu, Y., Yang, Z., Zou, Z., Pei, J., et al. (2020) · IEEE Journal of Solid-State Circuits · doi:10.1109/jssc.2020.2970709 · Cited by 201 · cited 1×
“Combining the weight distribution [8] with the inter-PE data passing [9], [257], Tianjic [264], [265] and Simba [266], [267] adopt a scalable many-core/many-chip architecture where all cores (i.e., PEs) work in a decentralized manner without the global off-chip memory.”
Self-citationLiterature survey on low rank approximation of matrices
Kishore Kumar, N., & Schneider, J. (2017) · Linear and Multilinear Algebra · doi:10.1080/03081087.2016.1267104 · Cited by 195 · cited 1×
“…into a low-dimensional space that enables fast decentralized gradient aggregation (e.g., ring all-reduce) in the compressed domain. c) Other matrix decomposition: Except for the full-rank decomposition and SVD, there are also some other matrix decomposition methods such as QR and CUR [92].”
MethodHierarchical Representations for Efficient Architecture Search
Liu, H., Simonyan, K., Vinyals, O., Fernando, C., & Kavukcuoglu, K. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1711.00436 · Cited by 194 · cited 3×
“9(c)] or supporting hierarchical topology [67] [see Fig.”
BackgroundMethodScalable Methods for 8-bit Training of Neural Networks
Banner, R., Hubara, I., Hoffer, E., & Soudry, D. (2018) · arXiv · doi:10.48550/arxiv.1805.11046 · Cited by 193 · cited 3×
“Therefore, reducing BN’s nonlinearity and quantizing BN operations also matter [180], [194], [195].”
BackgroundGuo, X., Bayat, F. M., Bavandpour, M., Klachko, M., Mahmoodi, M. R., Prezioso, M., Likharev, K. K., & Strukov, D. B. (2017) · 2017 IEEE International Electron Devices Meeting (IEDM) · doi:10.1109/iedm.2017.8268341 · Cited by 192 · cited 2×
“[272] X.”
BackgroundUCNN: Exploiting Computational Reuse in Deep Neural Networks via Weight Repetition
Hegde, K., Yu, J., Agrawal, R., Yan, M., Pellauer, M., & Fletcher, C. (2018) · 2018 ACM/IEEE 45th Annual International Symposium on Computer Architecture (ISCA) · doi:10.1109/isca.2018.00062 · Cited by 191 · cited 1×
“[288] K.”
Training Deep Neural Networks with 8-bit Floating Point Numbers
Wang, N., Choi, J., Brand, D., Chen, C., & Gopalakrishnan, K. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1812.08011 · Cited by 189 · cited 1×
“Recently, Wang et al. [184] further pushed the multiplications, accumulations, and updated in training to 8, 16, and 16-bit floating-point numbers, respectively. b) RNN quantization: Table 9 lists most existing quantization works for RNNs.”
BackgroundA Survey on Methods and Theories of Quantized Neural Networks
Guo, Y. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1808.04752 · Cited by 188 · cited 1×
“Compared with recent surveys on accelerators without compression [14], a certain category of compression approaches [15], or a rapid overview of multiple categories on only the algorithm side [16], [17], we present a more systematic summary on most existing methodologies, show more detailed…”
MethodFrequency-Domain Dynamic Pruning for Convolutional Neural Networks
Liu, Z., Xu, J., Peng, X., & Xiong, R. (2018) · neural information processing systems · Cited by 188
Compressing DMA Engine: Leveraging Activation Sparsity for Training Deep Neural Networks
Rhu, M., O'Connor, M., Chatterjee, N., Pool, J., Kwon, Y., & Keckler, S. W. (2018) · 2018 IEEE International Symposium on High Performance Computer Architecture (HPCA) · doi:10.1109/hpca.2018.00017 · Cited by 181
Deng, L., Jiao, P., Pei, J., Wu, Z., & Li, G. (2018) · Neural Networks · doi:10.1016/j.neunet.2018.01.010 · Cited by 178 · cited 5×
“…in the positive region and approximated the quantization function via modified ReLU to help the gradient-based BP. Li et al. [181] and Deng et al. [151] extrapolated the quantization of both W and A to multiple bits and always constrain W in a discrete space to remove the high-precision master…”
BackgroundMethodInfluentialSelf-citationLearning Compact Recurrent Neural Networks with Block-Term Tensor Decomposition
Ye, J., Wang, L., Li, G., Chen, D., Zhe, S., Chu, X., & Xu, Z. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00977 · Cited by 177
Self-citationBeating the Perils of Non-Convexity: Guaranteed Training of Neural Networks using Tensor MethodsPreprint
Janzamin, M., Sedghi, H., & Anandkumar, A. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1506.08473 · Cited by 176
Binary convolutional neural network on RRAM
Tang, T., Xia, L., Li, B., Wang, Y., & Yang, H. (2017) · 2017 22nd Asia and South Pacific Design Automation Conference (ASP-DAC) · doi:10.1109/aspdac.2017.7858419 · Cited by 176 · cited 1×
“In addition, binary neural networks are beneficial for the deployment on emerging memory architectures with only two stable levels or supporting only simple logic operations [292]–[295].”
BackgroundApproximation of $2^d\times2^d$ Matrices Using Tensor Decomposition
Oseledets, I. V. (2010) · SIAM Journal on Matrix Analysis and Applications · doi:10.1137/090757861 · Cited by 174 · cited 2×
“By tensoriz-ing the weight matrices first, the TT format with small core tensors can be easily achieved, which is known as quantized TT (QTT) [135] with lower ranks and higher compression ratios. g) TT variant: Tensor chain (TC): The TC format [136] (also called tensor ring [118]) is a variant of TT, which has a compar…”
BackgroundSnaPEA: Predictive Early Activation for Reducing Computation in Deep Convolutional Neural Networks
Akhlaghi, V., Yazdanbakhsh, A., Samadi, K., Gupta, R. K., & Esmaeilzadeh, H. (2018) · 2018 ACM/IEEE 45th Annual International Symposium on Computer Architecture (ISCA) · doi:10.1109/isca.2018.00061 · Cited by 173 · cited 4×
“32(a) monitors the sign of the accumulated activation in each PE and terminates its computation once a negative value is detected [242].”
BackgroundInfluentialYin, S., Sun, X., Yu, S., & Seo, J.-S. (2020) · IEEE Transactions on Electron Devices · doi:10.1109/ted.2020.3015178 · Cited by 171 · cited 1×
“For the current to voltage (I2V) converters, they can be implemented either explicitly [258], [268] or implicitly [269] in different designs.”
Progressive Neural Architecture Search
Liu, C., Zoph, B., Neumann, M., Shlens, J., Hua, W., Li, L.-J., Fei-Fei, L., Yuille, A., Huang, J., & Murphy, K. (2017) · arXiv · doi:10.48550/arxiv.1712.00559 · Cited by 171 · cited 2×
“One possible way of speeding up the evaluation step is to build an accuracy predictor [66], [72].”
BackgroundXNOR Neural Engine: A Hardware Accelerator IP for 21.6-fJ/op Binary Neural Network Inference
Conti, F., Schiavone, P. D., & Benini, L. (2018) · IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems · doi:10.1109/tcad.2018.2857019 · Cited by 170 · cited 1×
“[276] F.”
Makhzani, A., & Frey, B. J. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1409.2752 · Cited by 167
$L1$ -Norm Batch Normalization for Efficient Training of Deep Neural Networks
Wu, S., Li, G., Deng, L., Liu, L., Wu, D., Xie, Y., & Shi, L. (2019) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2018.2876179 · Cited by 166 · cited 2×
“Therefore, reducing BN’s nonlinearity and quantizing BN operations also matter [180], [194], [195].”
BackgroundSelf-citationFlexpoint: An Adaptive Numerical Format for Efficient Training of Deep Neural Networks
Köster, U., Webb, T. J., Wang, X., Nassar, M., Bansal, A. K., Constable, W., Elibol, O. H., Gray, S., Hall, S., Hornof, L., et al. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1711.02213 · Cited by 164
Ando, K., Ueyoshi, K., Orimo, K., Yonekawa, H., Sato, S., Nakahara, H., Takamaeda-Yamazaki, S., Ikebe, M., Asai, T., Kuroda, T., et al. (2018) · IEEE Journal of Solid-State Circuits · doi:10.1109/jssc.2017.2778702 · Cited by 160 · cited 1×
“If both the weights and activations are binarized/ternarized, the MAC operations can be implemented by quite simple XNOR and pop-count logic operations [273], [275]–[277], which is illustrated in Fig.”
BackgroundXue, J., Li, J., Yu, D., Seltzer, M., & Gong, Y. (2014) · 2014 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2014.6854828 · Cited by 157 · cited 1×
“Li and Park [80] and Xue et al. [81] utilized a simplified SVD to replace the original weight matrix, both of which have reduced the spatial complexity.”
BackgroundBankman, D., Yang, L., Moons, B., Verhelst, M., & Murmann, B. (2018) · 2018 IEEE International Solid - State Circuits Conference - (ISSCC) · doi:10.1109/isscc.2018.8310264 · Cited by 157
Yang, T.-H., Cheng, H.-Y., Yang, C.-L., Tseng, I.-C., Hu, H.-W., Chang, H.-S., & Li, H.-P. (2019) · Proceedings of the 46th International Symposium on Computer Architecture · doi:10.1145/3307650.3322271 · Cited by 153 · cited 2×
“Yang et al. [296] argued that the entire crossbar cannot be activated at once if considering the computation accuracy on practical eNVM devices.”
BackgroundLong-term Forecasting using Higher Order Tensor RNNsPreprint
Yu, R., Zheng, S., Anandkumar, A., & Yue, Y. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1711.00073 · Cited by 152
Scalpel: Customizing DNN Pruning to the Underlying Hardware Parallelism
Yu, J., Lukefahr, A., Palframan, D., Dasika, G., Das, R., & Mahlke, S. (2017) · ACM SIGARCH Computer Architecture News · doi:10.1145/3140659.3080215 · Cited by 152
Ren, A., Zhang, T., Ye, S., Li, J., Xu, W., Qian, X., Lin, X., & Wang, Y. (2019) · Proceedings of the Twenty-Fourth International Conference on Architectural Support for Programming Languages and Operating Systems · doi:10.1145/3297858.3304076 · Cited by 150 · cited 5×
“The joint-way compression across multiple techniques also emerges to pursue extreme compression [7].”
BackgroundMethodInfluentialKung, H. T., McDanel, B., & Zhang, S. Q. (2019) · Proceedings of the Twenty-Fourth International Conference on Architectural Support for Programming Languages and Operating Systems · doi:10.1145/3297858.3304028 · Cited by 146 · cited 2×
“The weights and psums propagate across Adapted from [14], [257], and [263].”
BackgroundConvolutional Rectifier Networks as Generalized Tensor DecompositionsPreprint
Cohen, N., & Shashua, A. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1603.00162 · Cited by 142 · cited 2×
“As a result, just a few practices of HT can be found [126]–[128].”
BackgroundNeural Networks with Few Multiplications
Lin, Z., Courbariaux, M., Memisevic, R., & Bengio, Y. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1510.03009 · Cited by 141 · cited 3×
“The extensive rise of quantization for neural networks owes to the early binarization [5], [150], [156], [157], [159] and ternarization [161], [162], [187], [188].”
BackgroundMethodInfluentialGist: Efficient Data Encoding for Deep Neural Network Training
Jain, A., Phanishayee, A., Mars, J., Tang, L., & Pekhimenko, G. (2018) · 2018 ACM/IEEE 45th Annual International Symposium on Computer Architecture (ISCA) · doi:10.1109/isca.2018.00070 · Cited by 140 · cited 2×
“Although the use of on-GPU compression such as [201], [233], [250], and [251] can improve the performance, there is still a big gap far from our expectation due to the redundant design of general-purpose processors for flexible programmability and general applicability.”
BackgroundWu, J., Wang, Y., Wu, Z., Wang, Z., Veeraraghavan, A., & Lin, Y. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1806.09228 · Cited by 137 · cited 1×
“Note that there also exist methods with data clustering rather than directly quantizing them to lower bitwidth [185], [186].”
MethodRxNN: A Framework for Evaluating Deep Neural Networks on Resistive Crossbars
Jain, S., Sengupta, A., Roy, K., & Raghunathan, A. (2021) · IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems · doi:10.1109/tcad.2020.3000185 · Cited by 137 · cited 1×
“Considering the crossbar mapping and the low-bit limitation during the training of models can Adapted from [296]. help recover the application accuracy [290], [291].”
Background
Research works printed but not linked: 2
- lin, 2013, Network in network, arXiv 1312 4400
- han, 2015, Learning both weights and connections for efficient neural network, Proc Adv Neural Inf Process Syst, 1135
Cited by
The Role of AI in Hospitals and Clinics: Transforming Healthcare in the 21st Century
Maleki Varnosfaderani, S., & Forouzanfar, M. (2024) · Bioengineering · doi:10.3390/bioengineering11040337 · Cited by 931
“This is where hardware acceleration steps in, acting as a powerful engine that fuels AI-powered diagnostics [111].”
BackgroundBeyond Transmitting Bits: Context, Semantics, and Task-Oriented Communications
Gunduz, D., Qin, Z., Aguerri, I. E., Dhillon, H. S., Yang, Z., Yener, A., Wong, K. K., & Chae, C.-B. (2023) · IEEE Journal on Selected Areas in Communications · doi:10.1109/jsac.2022.3223408 · Cited by 776
“We refer the reader to [149] for a more comprehensive survey of recent advances in model compression techniques for DNNs.”
Cellular, Wide-Area, and Non-Terrestrial IoT: A Survey on 5G Advances and the Road Toward 6G
Vaezi, M., Azari, A., Khosravirad, S. R., Shirvanimoghaddam, M., Azari, M. M., Chasaki, D., & Popovski, P. (2022) · IEEE Communications Surveys & Tutorials · doi:10.1109/comst.2022.3151028 · Cited by 631
Efficient Acceleration of Deep Learning Inference on Resource-Constrained Edge Devices: A Review
Shuvo, M. M. H., Islam, S. K., Cheng, J., & Morshed, B. I. (2023) · Proceedings of the IEEE · doi:10.1109/jproc.2022.3226481 · Cited by 436
“There are available reviews of different DL techniques [5], DNNs on edge [2], [16], DL compression techniques for hardware acceleration [10], DL inference on FPGA-based platforms [17], edge intelligence [18], and edge computing [19].”
BackgroundMethodHoefler, T., Alistarh, D., Ben-Nun, T., Dryden, N., & Peste, A. (2021) · ETH Zurich · doi:10.3929/ethz-b-000519202 · Cited by 423
Metadata
- Authors
- Deng, B. L., Li, G., Han, S., Shi, L., & Xie, Y.
- Affiliations
- Tsinghua University (CN)
- Massachusetts Institute of Technology (US)
- University of California, Santa Barbara (US)
- Venue
- Proceedings of the IEEE
- Published
- 2020-04
- Volume
- 108
- Issue
- 4
- Pages
- 485-532
- ISSN
- 0018-9219 (ISSN-L)
- 1558-2256
- OpenAlex
- W3012561096
- Cited by
- 996
- References
- 334
- Citations by year
- 2026: 131 · 2025: 171 · 2024: 193 · 2023: 174 · 2022: 152 · 2021: 142 · 2020: 31
- FWCI
- 46.83
- Type
- journal article
- Language
- en
- Open access
- closed
- Licence
- Locations
- Proceedings of the IEEE (publishedVersion)
- Rare & Special e-Zone (The Hong Kong University of Science and Technology) (submittedVersion)
- Funders
- National Science Foundation (1725447)
- Beijing Academy of Artificial Intelligence (BAAI), Tsinghua University Initiative Scientific Research Program
- Institute for Guo Qiang, Tsinghua University
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Computer Science › Hardware and Architecture › Parallel Computing and Optimization Techniques
- Physical Sciences › Engineering › Electrical and Electronic Engineering › Advanced Memory and Neural Computing
- Keywords
- neural network compression, neural network accelerators, deep neural network, data quantization, network sparsification, tensor decomposition, compact modeling, memory efficiency, embedded systems, hardware architecture
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.