Model Compression and Hardware Acceleration for Neural Networks: A Comprehensive Survey
The DOI matches its Crossref record; 317 of the 334 references it cites were found in a registry or an index, and 1 are not checked yet.
Abstract
Domain-specific hardware is becoming a promising topic in the backdrop of improvement slow down for general-purpose processors due to the foreseeable end of Moore's Law. Machine learning, especially deep neural networks (DNNs), has become the most dazzling domain witnessing successful applications in a wide spectrum of artificial intelligence (AI) tasks. The incomparable accuracy of DNNs is achieved by paying the cost of hungry memory consumption and high computational complexity, which greatly impedes their deployment in embedded systems. Therefore, the DNN compression concept was naturally proposed and widely used for memory saving and compute acceleration. In the past few years, a tremendous number of compression techniques have sprung up to pursue a satisfactory tradeoff between processing efficiency and application accuracy. Recently, this wave has spread to the design of neural network accelerators for gaining extremely high performance. However, the amount of related works is incredibly huge and the reported approaches are quite divergent. This research chaos motivates us to provide a comprehensive survey on the recent advances toward the goal of efficient compression and execution of DNNs without significantly compromising accuracy, involving both the high-level algorithms and their applications in hardware design. In this article, we review the mainstream compression approaches such as compact model, tensor decomposition, data quantization, and network sparsification…
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“but there is no review discussing all three categories which in future will be integrated seamlessly to design effective and efficient DNNs in the new edge intelligence era.”
From Bringing ai to edge: from deep learning’s perspective (2022) · cites this paper as [28]
Citations
Cites
Items 201–318 of 318
Dynamic Network Surgery for Efficient DNNs
Guo, Y., Yao, A., & Chen, Y. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1608.04493 · Cited by 137 · cited 2×
“[172] combined the proposed incremental weight quantization and prior dynamic surgery based pruning in [249], which increases the 6.”
BackgroundPath-Level Network Transformation for Efficient Architecture Search
Cai, H., Yang, J., Zhang, W., Han, S., & Yu, Y. H. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1806.02639 · Cited by 135
Self-citationSCOPE: A Stochastic Computing Engine for DRAM-Based In-Situ Accelerator
Li, S., Glova, A. O., Hu, X., Gu, P., Niu, D., Malladi, K. T., Zheng, H., Brennan, B. A., & Xie, Y. (2018) · IEEE/ACM International Symposium on Microarchitecture (MICRO) · doi:10.1109/micro.2018.00062 · Cited by 131 · cited 1×
“Besides eNVM devices, traditional memories such as SRAM [270], DRAM [271], and Flash [272] can also be modified to support the PIM-fashion processing of neural networks.”
Self-citationTraining Quantized Nets: A Deeper Understanding
Li, H., De, S., Xu, Z., Studer, C., Samet, H., & Goldstein, T. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1706.02379 · Cited by 130 · cited 2×
“In a nutshell, the heuristic methods are able to achieve sufficiently good accuracy on small or medium data sets such as MNIST, SVHN, and CIFAR10/CIFAR100 [150], [151], [168], [182].”
BackgroundMethodTensor Regression NetworksPreprint
Kossaifi, J., Lipton, Z. C., Kolbeinsson, A., Khanna, A., Furlanello, T., & Anandkumar, A. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1707.08308 · Cited by 129 · cited 1×
“Instead of directly replacing weights with decomposed tensors, Chien and Bao [101] considered the entire decomposition process of (19) as a connection, similar to [102] and [103].”
MethodScalable and Sustainable Deep Learning via Randomized Hashing
Spring, R., & Shrivastava, A. (2017) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3097983.3098035 · Cited by 128 · cited 1×
“The preselection of unimportant neurons can be finished by hashing search [241], negative-activation prediction [242], low-precision estimation [243], [244], prechecking neighboring pixels [245], or dimension-reduction estimation [246].”
BackgroundOn the Expressive Power of Deep Learning: A Tensor Analysis
Cohen, N., Sharir, O., & Shashua, A. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1509.05009 · Cited by 127 · cited 1×
“Among them, only Cohen and Shashua [128] and Cohen et al. [129] showed neural network applications, but they are yet to answer how to obtain an optimal HT form. e) Tensor train (TT): According to [130], the TT format of a d th-order tensor A ∈ R n 1 × n 2 × ... × n d can be represented as A ( j 1 ,…”
BackgroundExploring Sparsity in Recurrent Neural Networks
Narang, S., Elsen, E., Diamos, G., & Sengupta, S. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1704.05119 · Cited by 126 · cited 2×
“With a similar pruning method, researchers prove that the pruned large sparse models including CNNs and RNNs consistently show higher accuracy than their small dense counterparts [198], [228] and the sparse ones trained from scratch [229] under the condition of identical memory footprint.”
BackgroundMethodQuantized CNN: A Unified Approach to Accelerate and Compress Convolutional Networks
Cheng, J., Wu, J., Leng, C., Wang, Y., & Hu, Q. (2018) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2017.2774288 · Cited by 126 · cited 1×
“When quantizing weights, besides minimizing the weight approximation error, the problem can be formulated as to minimize the response error of each layer [189].”
BackgroundCoordinating Filters for Faster Deep Neural Networks
Wen, W., Xu, C., Wu, C., Wang, Y., Chen, Y., & Li, H. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.78 · Cited by 123
PermDNN: Efficient Compressed DNN Architecture with Permuted Diagonal Matrices
Deng, C., Liao, S., Xie, Y., Parhi, K. K., Qian, X., & Yuan, B. (2018) · IEEE/ACM International Symposium on Microarchitecture (MICRO) · doi:10.1109/micro.2018.00024 · Cited by 120 · cited 1×
“For example, the block-wise weight sparsity is leveraged to optimize the running performance on general-purpose processors [13], [220], [234], and a more aggressive diagonal weight matrix is utilized in the accelerator design [282].”
MethodSelf-citationMixed Precision Training of Convolutional Neural Networks using Integer Operations
Das, D., Mellempudi, N., Mudigere, D., Kalamkar, D. D., Avancha, S., Banerjee, K., Sridharan, S., Vaidyanathan, K., Kaul, B., Georganas, E., et al. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1802.00930 · Cited by 118 · cited 3×
“For example, the quantized networks can be viewed as dense models with shorter bit-width [179]; structurally pruned networks (e.g., vector-wise in RNNs and FM-wise in CNNs) [204], [205], [232], [234] or decomposed networks [139] can be reassembled into smaller dense models to speed up the running.”
BackgroundMethodInvestigating gated recurrent networks for speech synthesis
Wu, Z., & King, S. (2016) · 2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2016.7472657 · Cited by 118
Fundamental tensor operations for large-scale data analysis using tensor network formats
Lee, N., & Cichocki, A. S. (2017) · Multidimensional Systems and Signal Processing · doi:10.1007/s11045-017-0481-0 · Cited by 117 · cited 3×
“Here, we first introduce an operation between a tensor and a matrix called mode-i contracted product [97].”
BackgroundMethodCP-decomposition with Tensor Power Method for Convolutional Neural Networks compression
Astrid, M., & Lee, S. (2017) · IEEE International Conference on Big Data and Smart Computing (BigComp) · doi:10.1109/bigcomp.2017.7881725 · Cited by 114 · cited 1×
“Lebedev et al. [105] and Astrid and Lee [106] leveraged CP decomposition to compress Conv weight kernels into several subkernels to reduce the number of parameters and speed up the running.”
BackgroundGrow and Prune Compact, Fast, and Accurate LSTMs
Dai, X., Yin, H., & Jha, N. K. (2019) · IEEE Transactions on Computers · doi:10.1109/tc.2019.2954495 · Cited by 114 · cited 1×
“Recently, NeST has been extended to LSTM by introducing the DNN structure into the gates and then using similar growing and pruning techniques to produce sparse weights, which is able to reduce the layers required to stack [230].”
MethodTBN: Convolutional Neural Network with Ternary Inputs and Binary Weights
Wan, D., Shen, F., Liu, L., Zhu, F., Qin, J., Shao, L., & Shen, H. T. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-01216-8_20 · Cited by 113 · cited 2×
“For example, Wan et al. [166] combined (27) for weight binarization with scaling factors and (28) for activation ternarization without scaling factors; Achterhold et al. [163] leveraged Bayesian compression to ternarize weight parameters.”
BackgroundBalanced Quantization: An Effective and Efficient Approach to Quantized Neural Networks
Zhou, S., Wang, Y., Wen, H., He, Q., & Zou, Y. (2017) · Journal of Computer Science and Technology · doi:10.1007/s11390-017-1750-y · Cited by 111 · cited 4×
“Based on the naive DoReFa-Net, the same group further proposes balanced quantization to match the real data distribution better by introducing mean ( | XXX | ) for normalization [153] and histogram equalization [169] before Q k .”
BackgroundMethodResultInfluentialCompression of Deep Convolutional Neural Networks for Fast and Low Power Mobile Applications
Kim, Y., Park, E., Yoo, S., Choi, T., Lu, Y., & Shin, D. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1511.06530 · Cited by 111 · cited 1×
“Kim et al. [100] used Tucker-2 decomposition to compress Conv weight kernels and they further deploy the compressed CNNs in cellphone.”
MethodLearning Intrinsic Sparse Structures within Long Short-Term Memory
Wen, W., He, Y., Rajbhandari, S., Zhang, M., Wang, W., Liu, F., Hu, B., Chen, Y., & Li, H. H. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1709.05027 · Cited by 108 · cited 2×
“For example, the quantized networks can be viewed as dense models with shorter bit-width [179]; structurally pruned networks (e.g., vector-wise in RNNs and FM-wise in CNNs) [204], [205], [232], [234] or decomposed networks [139] can be reassembled into smaller dense models to speed up the running.”
BackgroundMethodAttacking Binarized Neural NetworksPreprint
Galloway, A., Taylor, G. W., & Moussa, M. A. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1711.00449 · Cited by 108 · cited 1×
“In addition to the research on sparsification techniques, Galloway et al. [305] empirically showed that stochastic quantization can also reduce the impact of attack based on experiments.”
UNIQ: Uniform Noise Injection for Non-Uniform Quantization of Neural Networks
Baskin, C., Liss, N., Schwartz, E., Zheltonozhskii, E., Giryes, R., Bronstein, A. M., & Mendelson, A. (2019) · ACM Transactions on Computer Systems · doi:10.1145/3444943 · Cited by 107
Parallelizing SRAM Arrays with Customized Bit-Cell for Binary Neural Networks
Liu, R., Peng, X., Sun, X., Khwa, W.-S., Si, X., Chen, J., Li, J., Chang, M., & Yu, S. (2018) · ACM/IEEE Design Automation Conference (DAC) · doi:10.1145/3195970.3196089 · Cited by 107 · cited 1×
“In addition, binary neural networks are beneficial for the deployment on emerging memory architectures with only two stable levels or supporting only simple logic operations [292]–[295].”
BackgroundmeProp: Sparsified Back Propagation for Accelerated Deep Learning with Reduced Overfitting
Sun, X., Ren, X., Ma, S., & Wang, H. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1706.06197 · Cited by 106 · cited 2×
“, for A [237] or for E [202], [233]), according to the neuronal selectivity [238] or even based on random [239]/probabilistic [240] selection.”
BackgroundArchitectural Complexity Measures of Recurrent Neural Networks
Zhang, S., Wu, Y., Che, T., Lin, Z., Memisevic, R., Salakhutdinov, R., & Bengio, Y. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1602.08210 · Cited by 105 · cited 1×
“A similar but different approach is the skip-connected RNN [55], where skip connections are explicitly added into the RNN graph for improving the performance by allowing direct information transmission between nonconsecutive timesteps.”
BackgroundTensor-factorized neural networks
Chien, J., & Bao, Y.-T. (2017) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2017.2690379 · Cited by 104 · cited 1×
“Instead of directly replacing weights with decomposed tensors, Chien and Bao [101] considered the entire decomposition process of (19) as a connection, similar to [102] and [103].”
MethodCompressing recurrent neural network with tensor train
Tjandra, A., Sakti, S., & Nakamura, S. (2017) · 2017 International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn.2017.7966420 · Cited by 104 · cited 1×
“Compared to CNNs, the works on RNNs [122]–[124], [138] are fewer perhaps due to the complicated gate structure of LSTM/GRU and the less popularity and standardization of RNNs in current AI tasks when compared to CNNs. Table 6 enumerates recent works.”
MethodStructADMM: A Systematic, High-Efficiency Framework of Structured Weight Pruning for DNNsPreprint
Zhang, T., Ye, S., Zhang, K., Ma, X., Liu, N., Zhang, L., Tang, J., Ma, K., Lin, X., Fardad, M., et al. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1807.11091 · Cited by 103 · cited 2×
“Via different grouping schemes in the constraint condition, various sparsification structures can be produced [224].”
BackgroundMethodPrediction Based Execution on Deep Neural Networks
Song, M., Zhao, J., Hu, Y., Zhang, J., & Li, T. (2018) · Proceedings - International Symposium on Computer Architecture/Proceedings · doi:10.1109/isca.2018.00068 · Cited by 100 · cited 4×
“33; while for the temporally decoupled speculation and execution, the PE array only processes the computation of important neurons according to the predicted indexes [244], as illustrated in Fig.”
BackgroundMethodInfluentialFactorization tricks for LSTM networks
Kuchaiev, O., & Ginsburg, B. (2017) · arXiv · doi:10.48550/arxiv.1703.10722 · Cited by 99 · cited 1×
“Factorized LSTM [53] fac-torizes each weight matrix of an LSTM into two smaller matrices and feeds them with two subvectors.”
BackgroundA novel channel pruning method for deep neural network compressionPreprint
Hu, Y., Sun, S., Li, J., Wang, X., & Gu, Q. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1805.11394 · Cited by 97 · cited 1×
“For weight kernel pruning, an optimization problem similar to the following (41) has been formulated, which can be solved by the genetic algorithm [218] or L 0 -norm constrained gradient descent [219]. b) Neuron pruning: Neurons are also prunable by using the simple magnitude-aware rule (e.g., for…”
MethodWang, P., Ji, Y., Hong, C., Lyu, Y., Wang, D., & Xie, Y. (2018) · ACM/IEEE Design Automation Conference (DAC) · doi:10.1145/3195970.3196116 · Cited by 94 · cited 2×
“SNrram [297] extracts small dense submatrices from a large sparse matrix and then map them onto crossbars.”
BackgroundSelf-citationNorm matters: efficient and accurate normalization schemes in deep networks
Hoffer, E., Banner, R., Golan, I., & Soudry, D. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1803.01814 · Cited by 90 · cited 1×
“Therefore, reducing BN’s nonlinearity and quantizing BN operations also matter [180], [194], [195].”
BackgroundTIE: energy-efficient tensor train-based inference engine for deep neural network
Deng, C., Sun, F., Qian, X., Lin, J. T., Wang, Z., & Yuan, B. (2019) · Proceedings - International Symposium on Computer Architecture/Proceedings · doi:10.1145/3307650.3322258 · Cited by 89 · cited 4×
“Recently, TIE [11] accelerates the TT decomposed neural networks, where the reshape is implemented by partitioning the working SRAM into multiple groups with a well-designed data selection mechanism.”
BackgroundMethodInfluentialYu, M., Lin, Z., Narra, K. G., Li, S., Li, Y., Kim, N. S., Schwing, A. G., Annavaram, M., & Avestimehr, S. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1811.03617 · Cited by 89 · cited 1×
“In the scenario of distributed training, Yu et al. [91] utilized principal component analysis (PCA) to linearly project the weight gradients into a low-dimensional space that enables fast decentralized gradient aggregation (e.g., ring all-reduce) in the compressed domain. c) Other matrix…”
MethodAn Introduction to Hierarchical (H-) Rank and TT-Rank of Tensors with Examples
Grasedyck, L., & Hackbusch, W. (2011) · Computational Methods in Applied Mathematics · doi:10.2478/cmam-2011-0016 · Cited by 87 · cited 1×
“[132] L.”
Alternating Multi-bit Quantization for Recurrent Neural Networks
Xu, C., Yao, J., Lin, Z., Ou, W., Cao, Y., Wang, Z., & Zha, H. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1802.00150 · Cited by 82 · cited 5×
“For example, the dynamic range and quantization levels can be parameterized in different ways and trained using iterative optimization [143], [144] or gradient descent [142], [145], [146].”
BackgroundMethodResultInfluentialDefensive dropout for hardening deep neural networks under adversarial attacks
Wang, S., Wang, X., Zhao, P., Wen, W., Kaeli, D. R., Chin, P. S., & Lin, X. (2018) · doi:10.1145/3240765.3264699 · Cited by 82 · cited 1×
“…both experimental and theoretical perspectives on linear classifiers and discuss the possibility of extending such technique to DNNs. Wang et al. [303] used defensive dropout to strengthen DNNs. Ji et al. [304] proposed a method to defend the adversarial examples by removing the unnecessary…”
BackgroundNetwork Sketching: Exploiting Binary Structure in Deep CNNs
Guo, Y., Yao, A., Zhao, H., & Chen, Y. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.430 · Cited by 81 · cited 3×
“To improve accuracy, the final scaling factors α and binary vectors Q B can be further refined [144], [171].”
MethodChannelNets: Compact and Efficient Convolutional Neural Networks via Channel-Wise Convolutions
Gao, H., Wang, Z., Cai, L., & Ji, S. (2020) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2020.2975796 · Cited by 81
ReCom: An efficient resistive accelerator for compressed deep neural networks
Ji, H., Song, L., Jiang, L., Li, H., & Chen, Y. (2018) · Design, Automation, and Test in Europe · doi:10.23919/date.2018.8342009 · Cited by 81 · cited 2×
“[299] H.”
MethodExpressive power of recurrent neural networksPreprint
Khrulkov, V., Novikov, A., & Oseledets, I. V. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1711.00811 · Cited by 80 · cited 1×
“[133] V.”
Scalable high-performance architecture for convolutional ternary neural networks on FPGA
Prost-Boucle, A., Bourge, A., Petrot, F., Alemdar, H., Caldwell, N., & Leroy, V. (2017) · 2017 27th International Conference on Field Programmable Logic and Applications (FPL) · doi:10.23919/fpl.2017.8056850 · Cited by 78 · cited 1×
“[300] A.”
Learning Efficient Tensor Representations with Ring-structured Networks
Zhao, Q., Sugiyama, M., Yuan, L., & Cichocki, A. (2019) · ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2019.8682231 · Cited by 78 · cited 3×
“Although TC seems to have a greater potential to obtain better quality than TT [118], [137], more research studies are needed to evidence this point in neural networks. h) TT in neural networks: In tensor network decomposition, the TT format is widely applied in neural networks.”
BackgroundLin, J., Zhu, Z., Wang, Y., & Xie, Y. (2019) · Asia and South Pacific Design Automation Conference (ASP-DAC) · doi:10.1145/3287624.3287715 · Cited by 77 · cited 2×
“Lin et al. [298] partitioned the weight matrix into blocks and further reorder the columns in each block to cluster zero columns onto the same crossbar for coarse-grain pruning.”
BackgroundSelf-citationProxQuant: Quantized Neural Networks via Proximal Operators
Bai, Y., Wang, Y.-X., & Liberty, E. (2018) · arXiv · doi:10.48550/arxiv.1810.00861 · Cited by 76 · cited 1×
“Bai et al. [192] adopted an analogous idea.”
BackgroundSparsely-Connected Neural Networks: Towards Efficient VLSI Implementation of Deep Neural NetworksPreprint
Ardakani, A., Condo, C., & Gross, W. J. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1611.01427 · Cited by 75 · cited 1×
“Although most pruning methods are deterministic (according to weights and thresholds), probabilistic pruning also exists [197].”
MethodTIME: A Training-in-Memory Architecture for RRAM-Based Deep Neural Networks
Cheng, M., Xia, L., Zhu, Z., Cai, Y., Xie, Y., Wang, Y., & Yang, H. (2018) · IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems · doi:10.1109/tcad.2018.2824304 · Cited by 75 · cited 1×
“[253] M.”
Self-citationBinaryRelax: A Relaxation Approach for Training Deep Neural Networks with Quantized Weights
Yin, P., Zhang, S., Lyu, J., Osher, S., Qi, Y., & Xin, J. (2018) · SIAM Journal on Imaging Sciences · doi:10.1137/18m1166134 · Cited by 75
Discovering Low-Precision Networks Close to Full-Precision Networks for Efficient Embedded Inference
McKinstry, J. L., Esser, S. K., Appuswamy, R., Bablani, D., Arthur, J. V., Yildiz, I. B., & Modha, D. S. (2018) · arXiv · doi:10.48550/arxiv.1809.04191 · Cited by 74 · cited 5×
“For CNN inference (W & A), ≥ 8-bit quantization is able to achieve lossless or even better accuracy, whereas ≤ 4-bit quantization will produce obvious accuracy degradation, especially on very deep networks [146], [149].”
BackgroundResultInfluentialLow Bit-Width Convolutional Neural Network on RRAM
Cai, Y., Tang, T., Xia, L., Li, B., Wang, Y., & Yang, H. (2019) · IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems · doi:10.1109/tcad.2019.2917852 · Cited by 71 · cited 2×
“This precision limitation lies in weights (i.e., memory cell defects), input activations (i.e., DAC resolution), and (partial) output activations (i.e., ADC resolution) [290].”
BackgroundDeepCloak: Masking Deep Neural Network Models for Robustness Against Adversarial Samples
Gao, J., Wang, B., Lin, Z., Xu, W., & Qi, Y. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1702.06763 · Cited by 69 · cited 1×
“…experimental and theoretical perspectives on linear classifiers and discuss the possibility of extending such technique to DNNs. Wang et al. [303] used defensive dropout to strengthen DNNs. Ji et al. [304] proposed a method to defend the adversarial examples by removing the unnecessary features.”
BackgroundWider and Deeper, Cheaper and Faster: Tensorized LSTMs for Sequence LearningPreprint
He, Z., Gao, S., Xiao, L., Liu, D., He, H., & Barber, D. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1711.01577 · Cited by 68 · cited 1×
“Tensorized RNN and LSTM [57] represent the hidden states as tensors and update them via a cross-layer convolution.”
BackgroundTensor rank learning in CP decomposition via convolutional neural network
Zhou, M., Liu, Y., Long, Z., Chen, L., & Zhu, C. (2018) · Signal Processing Image Communication · doi:10.1016/j.image.2018.03.017 · Cited by 68 · cited 1×
“Zhou et al. [110] conversely used neural networks to learn an appropriate CP rank.”
BackgroundThe unreasonable effectiveness of the forget gate
van der Westhuizen, J., & Lasenby, J. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1804.04849 · Cited by 68 · cited 1×
“Furthermore, S-LSTM [41] and JANET [42] contain only one forget gate.”
BackgroundNestedNet: Learning Nested Sparse Structures in Deep Neural Networks
Kim, E., Ahn, C., & Oh, S. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00904 · Cited by 66 · cited 1×
“Multiple sparse networks with different sparsity level (lower level has higher sparsity) can be simultaneously trained from scratch using a merged object function, wherein the weights in the lower level are shared by the high-level networks [215].”
BackgroundTensor Decomposition for Compressing Recurrent Neural Network
Tjandra, A., Sakti, S., & Nakamura, S. (2018) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn.2018.8489213 · Cited by 65
Tensor Contraction Layers for Parsimonious Deep Nets
Kossaifi, J., Khanna, A., Lipton, Z. C., Furlanello, T., & Anandkumar, A. (2017) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw.2017.243 · Cited by 64 · cited 1×
“Instead of directly replacing weights with decomposed tensors, Chien and Bao [101] considered the entire decomposition process of (19) as a connection, similar to [102] and [103].”
MethodTensor Completion in Hierarchical Tensor Representations
Rauhut, H., Schneider, R., & Stojanac, Ž. (2015) · Applied and numerical harmonic analysis · doi:10.1007/978-3-319-16042-9_14 · Cited by 62
Aizenberg, I. N., Luchetta, A., & Manetti, S. (2011) · Soft Computing · doi:10.1007/s00500-011-0755-7 · Cited by 61 · cited 1×
“Additionally, using complex number shows potential to improve the expression of neural networks in recent years [93], so the explanation of a complex neuron with multiple inputs and a single output based on the QR decomposition [94] deserves more attention.”
BackgroundVariational Network Quantization
Achterhold, J., Koehler, J. M., Schmeink, A., & Genewein, T. (2018) · International Conference on Learning Representations · Cited by 60
XNORBIN: A 95 TOp/s/W hardware accelerator for binary convolutional neural networks
Bahou, A. A., Karunaratne, G., Andri, R., Cavigelli, L., & Benini, L. (2018) · IEEE Symposium in Low-Power and High-Speed Chips (COOL CHIPS) · doi:10.1109/coolchips.2018.8373076 · Cited by 59
PredictiveNet: An energy-efficient convolutional neural network via zero prediction
Lin, Y., Sakr, C., Kim, Y., & Shanbhag, N. R. (2017) · IEEE International Symposium on Circuits and Systems proceedings · doi:10.1109/iscas.2017.8050797 · Cited by 58 · cited 1×
“The preselection of unimportant neurons can be finished by hashing search [241], negative-activation prediction [242], low-precision estimation [243], [244], prechecking neighboring pixels [245], or dimension-reduction estimation [246].”
BackgroundEffective Quantization Methods for Recurrent Neural Networks
He, Q., Wen, H., Zhou, S., Wu, Y., Yao, C., Zhou, X., & Zou, Y. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1611.10176 · Cited by 56 · cited 4×
“Usually, the straight-through estimator (STE) is used to circumvent this problem [147], [152], [153].”
MethodResultInfluentialUnderstanding Straight-Through Estimator in Training Activation Quantized Neural Nets
Yin, P., Lyu, J., Zhang, S., Osher, S., Qi, Y., & Xin, J. X. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1903.05662 · Cited by 56 · cited 1×
“Using pulse train to approximate the derivative of discontinuous points [151], designing proper STE formats [154], or gradually rather than immediately closing the gradient BP path to the high-precision data [155] also works.”
MethodDomain-Adaptive Deep Network Compression
Masana, M., van de Weijer, J., Herranz, L., Bagdanov, A. D., & Alvarez, J. M. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.460 · Cited by 55
Sharing Residual Units Through Collective Tensor Factorization To Improve Deep Neural Networks
Chen, Y., Jin, X., Kang, B., Feng, J., & Yan, S. (2018) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2018/88 · Cited by 55 · cited 2×
“Chen et al. [107] introduced a collective residual unit (CRU) based on block term decomposition (BTD), which is a combination of Tucker and CP, to enhance the utilization of parameters in residual CNNs. Tran et al. [108] proposed a new tensor operation based on the mode-i contracted product to…”
BackgroundMethodCai, Y., Lin, Y., Xia, L., Chen, X., Han, S., Wang, Y., & Yang, H. (2018) · ACM/IEEE Design Automation Conference (DAC) · doi:10.1145/3195970.3196071 · Cited by 53 · cited 2×
“Although we can find a few ones for the training phase [252]–[254], RNN models [255] or both CNNs and RNNs [256]–[258], they are still not the main-stream.”
BackgroundMethodSelf-citation2PFPCE: Two-Phase Filter Pruning Based on Conditional EntropyPreprint
Min, C., Wang, A., Chen, Y., Xu, W., & Chen, X. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1809.02220 · Cited by 53 · cited 1×
“By contrast, from a statistical perspective, Min et al. [211] estimated the FM’s importance according to its conditional entropy to minimize the global loss entropy error.”
BackgroundCrossbar-Aware Neural Network Pruning
Liang, L., Deng, L., Zeng, Y., Hu, X., Ji, Y., Ma, X., Li, G., & Xie, Y. (2018) · IEEE Access · doi:10.1109/access.2018.2874823 · Cited by 52 · cited 2×
“…similar to the following (41) has been formulated, which can be solved by the genetic algorithm [218] or L 0 -norm constrained gradient descent [219]. b) Neuron pruning: Neurons are also prunable by using the simple magnitude-aware rule (e.g., for A [237] or for E [202], [233]), according to…”
BackgroundSelf-citationProgressive Weight Pruning of Deep Neural Networks using ADMMPreprint
Ye, S., Zhang, T., Zhang, K., Li, J., Xu, K., Yang, Y., Yu, F., Tang, J., Fardad, M., Liu, S., et al. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1810.07378 · Cited by 52 · cited 2×
“Recently, the ADMM method mentioned in Section III-D has also been used for high-accuracy sparsification [7], [223]–[225].”
BackgroundMethodZimmer, B., Venkatesan, R., Shao, Y. S., Clemons, J., Fojtik, M., Jiang, N., Keller, B., Klinefelter, A., Pinckney, N., Raina, P., et al. (2019) · 2019 Symposium on VLSI Circuits · doi:10.23919/vlsic.2019.8778056 · Cited by 52 · cited 2×
“Combining the weight distribution [8] with the inter-PE data passing [9], [257], Tianjic [264], [265] and Simba [266], [267] adopt a scalable many-core/many-chip architecture where all cores (i.”
BackgroundLayer-compensated Pruning for Resource-constrained Convolutional Neural NetworksPreprint
Chin, T.-W., Zhang, C., & Marculescu, D. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1810.00518 · Cited by 51 · cited 1×
“EAs have been introduced to compensate the loss error caused by the naive greedy pruning that in each iteration the top-k unimportant filters are heuristically pruned [213].”
BackgroundSVD-Softmax: Fast Softmax Approximation on Large Vocabulary Neural Networks
Shim, K., Lee, M., Choi, I., Boo, Y., & Sung, W. (2017) · neural information processing systems · Cited by 50
HitNet: Hybrid Ternary Recurrent Neural Network
Wang, P., Xie, X., Deng, L., Li, G., Wang, D., & Xie, Y. (2018) · neural information processing systems · Cited by 49
Self-citationXNORBIN: A 95 TOp/s/W Hardware Accelerator for Binary Convolutional Neural Networks
Bahou, A. A., Karunaratne, G., Andri, R., Cavigelli, L., & Benini, L. (2018) · Repository for Publications and Research Data (ETH Zurich) · doi:10.3929/ethz-b-000329926 · Cited by 49
Dynamic Sparse Graph for Efficient Deep LearningPreprint
Liu, L., Deng, L., Hu, X., Zhu, M., Li, G., Ding, Y., & Xie, Y. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1810.00859 · Cited by 47 · cited 1×
“The preselection of unimportant neurons can be finished by hashing search [241], negative-activation prediction [242], low-precision estimation [243], [244], prechecking neighboring pixels [245], or dimension-reduction estimation [246].”
BackgroundSelf-citationSparse Hierarchical Tucker Factorization and Its Application to Healthcare
Perros, I., Chen, R. E., Vuduc, R. W., & Sun, J. (2015) · Proceedings · doi:10.1109/icdm.2015.29 · Cited by 45 · cited 1×
“[127] I.”
Improving efficiency in convolutional neural networks with multilinear filters
Tran, D. T., Iosifidis, A., & Gabbouj, M. (2018) · Neural Networks · doi:10.1016/j.neunet.2018.05.017 · Cited by 45 · cited 1×
“[108] proposed a new tensor operation based on the mode-i contracted product to replace the normal Conv to reduce both the spatial and computational complexity.”
MethodHRTF personalization modeling based on RBF neural network
Li, L., & Huang, Q. (2013) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2013.6638350 · Cited by 44 · cited 1×
“Thus, the applications in neural networks are rare, only for pilot [114] or small-scale [115] ones. c) Tensorizing and curse of dimensionality: Since high-order tensors can usually help achieve better decomposition results, an intuitive idea comes out to convert the matrix to a high-order tensor…”
BackgroundTensor Train decomposition on TensorFlow (T3F)Preprint
Novikov, A., Izmailov, P., Khrulkov, V., Figurnov, M., & Oseledets, I. V. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1801.01928 · Cited by 44 · cited 1×
“For example, Novikov et al. [306] proposed a Tensorflow library for TT decomposition which makes machine learning relying on tensor decomposition easier to implement.”
BackgroundRecurrent Neural Networks With Limited Numerical Precision
Ott, J., Lin, Z., Zhang, Y., Liu, S., & Bengio, Y. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1608.06902 · Cited by 43 · cited 5×
“For example, hysteresis parameters can be introduced to control the level projection by shifting the transition region [151], [159]; pow2-ternarization is also used to ternarize W by introducing the power-of-two quantization for the fractional part [188].”
BackgroundMethodInfluentialHuang, H., Ni, L., Wang, K., Wang, Y., & Yu, H. (2017) · IEEE Transactions on Nanotechnology · doi:10.1109/tnano.2017.2732698 · Cited by 40 · cited 1×
“Except for few works such as accelerating tensorized neural networks on the eNVM-based crossbar architecture [119], most works that study neural network compression on emerging memory devices discuss about data quantization and network sparsification.”
BackgroundPer-Tensor Fixed-Point Quantization of the Back-Propagation AlgorithmPreprint
Sakr, C., & Shanbhag, N. R. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1812.11732 · Cited by 39 · cited 1×
“A recent work [183] proposes a systematic methodology for precision assignment across all layers to achieve nearly optimal full quantization (W/A/E/G/U).”
BackgroundLearning Low-precision Neural Networks without Straight-Through Estimator (STE)
Liu, Z.-G., & Mattina, M. (2019) · Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2019/425 · Cited by 37
Recurrent Neural Networks With Limited Numerical PrecisionPreprint
Ott, J., Lin, Z., Zhang, Y., Liu, S., & Bengio, Y. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1611.07065 · Cited by 35
A note on tensor chain approximation
Espig, M., Kumar, N. K., & Schneider, J. (2012) · Computing and Visualization in Science · doi:10.1007/s00791-014-0218-7 · Cited by 32 · cited 1×
“…with small core tensors can be easily achieved, which is known as quantized TT (QTT) [135] with lower ranks and higher compression ratios. g) TT variant: Tensor chain (TC): The TC format [136] (also called tensor ring [118]) is a variant of TT, which has a comparable approximation capability.”
BackgroundSparsity-based Defense Against Adversarial Attacks on Linear Classifiers
Marzi, Z., Gopalakrishnan, S., Madhow, U., & Pedarsani, R. (2018) · IEEE International Symposium on Information Theory (ISIT) · doi:10.1109/isit.2018.8437638 · Cited by 32
Hybrid Pruning: Thinner Sparse Networks for Fast Inference on Edge DevicesPreprint
Xu, X., Park, M. S., & Brick, C. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1811.00482 · Cited by 31 · cited 1×
“The element-wise irregular pruning can be stacked on the filter-wise structured pruning to boost the compression ratio [216].”
MethodTensorial Neural Networks: Generalization of Neural Networks and Application to Model CompressionPreprint
Su, J., Li, J., Bhattacharjee, B., & Huang, F. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1805.10352 · Cited by 31 · cited 1×
“Therefore, most research studies pay attention to compress FC layers, whereas the Conv layers remain unchanged [4], [118]–[120].”
BackgroundLow Precision RNNs: Quantizing RNNs Without Losing AccuracyPreprint
Kapur, S., Mishra, A., & Marr, D. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1710.07706 · Cited by 30
Learning Low Precision Deep Neural Networks through Regularization.Preprint
Choi, Y., El‐Khamy, M., & Lee, J. (2018) · arXiv (Cornell University) · Cited by 29
Deterministic CUR for Improved Large-Scale Data Analysis: An Empirical Study
Thurau, C., Kersting, K., & Bauckhage, C. (2012) · SIAM International Conference on Data Mining · doi:10.1137/1.9781611972825.59 · Cited by 28 · cited 1×
“A number of research studies utilize the CUR decomposition [95] for machine-learning applications, especially a variant similar to CUR called Nyström method [96].”
MethodSynaptic Strength For Convolutional Neural NetworkPreprint
Chen, L., Zhong, Z., Wu, W. L., & Yan, J. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1811.02454 · Cited by 28 · cited 1×
“…tensor-decomposed models; Choi et al. [221] added a similar regularization onto each weight kernel in both spatial and Winograd domains; Lin et al. [222] introduced the kernel-level scaling factors (termed synaptic strength), which equals the products of the BN scaling factor from each input FM…”
BackgroundTensor Networks for Dimensionality Reduction, Big Data and Deep Learning
Cichocki, A. S. (2017) · Studies in computational intelligence · doi:10.1007/978-3-319-67946-4_1 · Cited by 27 · cited 1×
“Fortunately, a tensor network [97], [99], [116] that represents a tensor with a link of matrices or low-order tensors with contracted products, is promising to avoid the curse of dimensionality by eliminating the high-order kernel tensor.”
BackgroundCompression of convolutional neural networks: A short survey
Pilipović, R., Bulić, P., & Risojević, V. (2018) · International Symposium INFOTEH-JAHORINA (INFOTEH) · doi:10.1109/infoteh.2018.8345545 · Cited by 27 · cited 1×
“…compression [14], a certain category of compression approaches [15], or a rapid overview of multiple categories on only the algorithm side [16], [17], we present a more systematic summary on most existing methodologies, show more detailed descriptions and comparisons for the individual or joint…”
MethodTETRIS: TilE-matching the TRemendous Irregular Sparsity
Ji, Y., Liang, L., Deng, L., Zhang, Y., Zhang, Y., & Xie, Y. (2018) · Rare & Special e-Zone (The Hong Kong University of Science and Technology) · Cited by 27
Self-citationTraining Behavior of Sparse Neural Network Topologies
Alford, S., Robinett, R. A., Milechin, L., & Kepner, J. (2019) · IEEE High Performance Extreme Computing Conference · doi:10.1109/hpec.2019.8916385 · Cited by 26 · cited 2×
“With a similar pruning method, researchers prove that the pruned large sparse models including CNNs and RNNs consistently show higher accuracy than their small dense counterparts [198], [228] and the sparse ones trained from scratch [229] under the condition of identical memory footprint.”
MethodResultAccuracy to Throughput Trade-Offs for Reduced Precision Neural Networks on Reconfigurable Logic
Su, J., Fraser, N. J., Gambardella, G., Blott, M., Durelli, G. C., Thomas, D. B., Leong, P. H. W., & Cheung, P. Y. K. (2018) · Lecture notes in computer science · doi:10.1007/978-3-319-78890-6_3 · Cited by 25 · cited 1×
“Su et al. [280] profiled this tradeoff in inference via building a throughput estimation model.”
BackgroundUNIQ: Uniform Noise Injection for Non-Uniform Quantization of Neural NetworksPreprint
Baskin, C., Schwartz, E., Zheltonozhskii, E., et al. (2018) · arXiv · doi:10.48550/arxiv.1804.10969 · Cited by 25
Singular value Decomposition applied to Associative Memory of Hopfield Neural Network
Deb, T., Ghosh, A. K., & Mukherjee, A. (2018) · Materials Today Proceedings · doi:10.1016/j.matpr.2017.09.222 · Cited by 24 · cited 2×
“For QR, it usually performs faster than SVD but with more accuracy loss [85].”
MethodHierarchical tucker tensor regression: Application to brain imaging data analysis
Hou, M., & Chaib-draa, B. (2015) · Proceedings - International Conference on Image Processing · doi:10.1109/icip.2015.7351019 · Cited by 23 · cited 1×
“As a result, just a few practices of HT can be found [126]–[128].”
BackgroundLee, N., & Cichocki, A. S. (2016) · SIAM Journal on Matrix Analysis and Applications · doi:10.1137/15m1028479 · Cited by 21 · cited 1×
“13(a) shows the structure of TT in a tensor network graph, where each node represents a core tensor and each edge is a mode of its connected tensors. f) HT versus TT: It is widely accepted that TT is a special form of HT [131]–[134].”
BackgroundOn tensor-product model based representation of neural networks
Rövid, A., Szeidl, L., & Várlaki, P. (2011) · IEEE International Conference on Intelligent Engineering Systems (INES) · doi:10.1109/ines.2011.5954721 · Cited by 20 · cited 1×
“Thus, the applications in neural networks are rare, only for pilot [114] or small-scale [115] ones. c) Tensorizing and curse of dimensionality: Since high-order tensors can usually help achieve better decomposition results, an intuitive idea comes out to convert the matrix to a high-order tensor…”
BackgroundNeural Network for Text Classification Based on Singular Value Decomposition
Li, C. H., & Park, S. C. (2007) · IEEE International Conference on Computer and Information Technology (CIT) · doi:10.1109/cit.2007.52 · Cited by 19 · cited 1×
“Li and Park [80] and Xue et al. [81] utilized a simplified SVD to replace the original weight matrix, both of which have reduced the spatial complexity.”
BackgroundBiNMAC: Binarized neural Network Manycore ACcelerator
Jafari, A., Hosseini, M., Kulkarni, A., Patel, C., & Mohsenin, T. (2018) · ACM Great Lakes Symposium on VLSI (GLSVLSI) · doi:10.1145/3194554.3194634 · Cited by 19 · cited 1×
“If both the weights and activations are binarized/ternarized, the MAC operations can be implemented by quite simple XNOR and pop-count logic operations [273], [275]–[277], which is illustrated in Fig.”
BackgroundKim, C., Shin, D., Kim, B., & Park, J. (2018) · IEEE Journal on Emerging and Selected Topics in Circuits and Systems · doi:10.1109/jetcas.2018.2865006 · Cited by 19 · cited 2×
“The preselection of unimportant neurons can be finished by hashing search [241], negative-activation prediction [242], low-precision estimation [243], [244], prechecking neighboring pixels [245], or dimension-reduction estimation [246].”
BackgroundLee, J., Shin, D., & Yoo, H. (2017) · IEEE Asian Solid-State Circuits Conference (A-SSCC) · doi:10.1109/asscc.2017.8240260 · Cited by 18
Training Single Hidden Layer Feedforward Neural Networks by Singular Value Decomposition
Huynh, H. T., & Won, Y. (2009) · International Conference on Computer Sciences and Convergence Information Technology · doi:10.1109/iccit.2009.170 · Cited by 17 · cited 1×
“Huynh and Won [82] proposed a new training method to achieve acceleration, which is compatible with the SVD format for networks with a single hidden layer.”
MethodTraining deep neural networks with discrete state transition
Li, G., Deng, L., Tian, L., Cui, H., Han, W., Pei, J., & Shi, L. (2017) · Neurocomputing · doi:10.1016/j.neucom.2017.06.058 · Cited by 16 · cited 2×
“…to multiple bits in the positive region and approximated the quantization function via modified ReLU to help the gradient-based BP. Li et al. [181] and Deng et al. [151] extrapolated the quantization of both W and A to multiple bits and always constrain W in a discrete space to remove the…”
BackgroundMethodSelf-citationEnd-to-end Learning of a Convolutional Neural Network via Deep Tensor DecompositionPreprint
Oymak, S., & Soltanolkotabi, M. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1805.06523 · Cited by 15 · cited 1×
“Recently, Oymak and Soltanolkotabi [111] rewrote the nonoverlapping Conv kernel and its input into the rank-1 CP format to improve the training efficiency.”
MethodStructurally Sparsified Backward Propagation for Faster Long Short-Term Memory TrainingPreprint
Zhu, M., Clemons, J., Pool, J., Rhu, M., Keckler, S. W., & Xie, Y. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1806.00512 · Cited by 13 · cited 3×
“Both element-wise (see [202] and [239]) and structured sparsity (see [233]) have been reported.”
BackgroundMethodSelf-citationTraining Compact Neural Networks with Binary Weights and Low Precision Activations
Zhuang, B., Shen, C., & Reid, I. (2018) · arXiv · doi:10.48550/arxiv.1808.02631 · Cited by 10 · cited 2×
“Although more bits [144], [147], [169], [182] and wider layers [170], [174] can help to improve accuracy, they will increase parameters and operations as well.”
BackgroundMethodSingular-Value-Decomposition Analysis of Associative Memory in a Neural Network
Kumamoto, T., Suzuki, M., & Matsueda, H. (2017) · Journal of the Physical Society of Japan · doi:10.7566/jpsj.86.024005 · Cited by 10 · cited 1×
“Furthermore, SVD can also be used to compress the input data to simplify and accelerate the training circuitously [84]–[86].”
MethodXue, S., Jiang, H., Dai, L.-R., & Liu, Q. (2015) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2015.7178833 · Cited by 9 · cited 2×
“Furthermore, SVD can also be used to compress the input data to simplify and accelerate the training circuitously [84]–[86].”
BackgroundMethodCompression of Deep Convolutional Neural Networks under Joint Sparsity ConstraintsPreprint
Choi, Y., El‐Khamy, M., & Lee, J. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1805.08303 · Cited by 8 · cited 3×
“Choi et al. [173], [221] used two steps to compress the models: 1) they first add magnitude-aware pruning regularization (cid”
BackgroundMethodL_1-Norm Low-Rank Matrix Decomposition by Neural Networks and Mollifiers
Liu, Y., Yang, S., Wu, P., Li, C., & Yang, M. (2015) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2015.2496964 · Cited by 7 · cited 1×
“Full-rank decomposition is an effective method for FC layer-based networks, especially between two layers with largely different amounts of neurons [76] or a sparse weight matrix with intrinsic low rank [77].”
MethodQuantized Neural Networks: Training Neural Networks with Low Precision Weights and Activations
Hubara, I., Courbariaux, M., Soudry, D., El-Yaniv, R., & Bengio, Y. (2016) · arXiv · doi:10.48550/arxiv.1609.07061 · cited 1×
“Usually, the straight-through estimator (STE) is used to circumvent this problem [147], [152], [153].”
Method
Research works printed but not linked: 2
- lin, 2013, Network in network, arXiv 1312 4400
- han, 2015, Learning both weights and connections for efficient neural network, Proc Adv Neural Inf Process Syst, 1135
Cited by
The Role of AI in Hospitals and Clinics: Transforming Healthcare in the 21st Century
Maleki Varnosfaderani, S., & Forouzanfar, M. (2024) · Bioengineering · doi:10.3390/bioengineering11040337 · Cited by 931
“This is where hardware acceleration steps in, acting as a powerful engine that fuels AI-powered diagnostics [111].”
BackgroundBeyond Transmitting Bits: Context, Semantics, and Task-Oriented Communications
Gunduz, D., Qin, Z., Aguerri, I. E., Dhillon, H. S., Yang, Z., Yener, A., Wong, K. K., & Chae, C.-B. (2023) · IEEE Journal on Selected Areas in Communications · doi:10.1109/jsac.2022.3223408 · Cited by 776
“We refer the reader to [149] for a more comprehensive survey of recent advances in model compression techniques for DNNs.”
Cellular, Wide-Area, and Non-Terrestrial IoT: A Survey on 5G Advances and the Road Toward 6G
Vaezi, M., Azari, A., Khosravirad, S. R., Shirvanimoghaddam, M., Azari, M. M., Chasaki, D., & Popovski, P. (2022) · IEEE Communications Surveys & Tutorials · doi:10.1109/comst.2022.3151028 · Cited by 631
Efficient Acceleration of Deep Learning Inference on Resource-Constrained Edge Devices: A Review
Shuvo, M. M. H., Islam, S. K., Cheng, J., & Morshed, B. I. (2023) · Proceedings of the IEEE · doi:10.1109/jproc.2022.3226481 · Cited by 436
“There are available reviews of different DL techniques [5], DNNs on edge [2], [16], DL compression techniques for hardware acceleration [10], DL inference on FPGA-based platforms [17], edge intelligence [18], and edge computing [19].”
BackgroundMethodHoefler, T., Alistarh, D., Ben-Nun, T., Dryden, N., & Peste, A. (2021) · ETH Zurich · doi:10.3929/ethz-b-000519202 · Cited by 423
Metadata
- Authors
- Deng, B. L., Li, G., Han, S., Shi, L., & Xie, Y.
- Affiliations
- Tsinghua University (CN)
- Massachusetts Institute of Technology (US)
- University of California, Santa Barbara (US)
- Venue
- Proceedings of the IEEE
- Published
- 2020-04
- Volume
- 108
- Issue
- 4
- Pages
- 485-532
- ISSN
- 0018-9219 (ISSN-L)
- 1558-2256
- OpenAlex
- W3012561096
- Cited by
- 996
- References
- 334
- Citations by year
- 2026: 131 · 2025: 171 · 2024: 193 · 2023: 174 · 2022: 152 · 2021: 142 · 2020: 31
- FWCI
- 46.83
- Type
- journal article
- Language
- en
- Open access
- closed
- Licence
- Locations
- Proceedings of the IEEE (publishedVersion)
- Rare & Special e-Zone (The Hong Kong University of Science and Technology) (submittedVersion)
- Funders
- National Science Foundation (1725447)
- Beijing Academy of Artificial Intelligence (BAAI), Tsinghua University Initiative Scientific Research Program
- Institute for Guo Qiang, Tsinghua University
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Computer Science › Hardware and Architecture › Parallel Computing and Optimization Techniques
- Physical Sciences › Engineering › Electrical and Electronic Engineering › Advanced Memory and Neural Computing
- Keywords
- neural network compression, neural network accelerators, deep neural network, data quantization, network sparsification, tensor decomposition, compact modeling, memory efficiency, embedded systems, hardware architecture
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.