Efficient Processing of Deep Neural Networks: A Tutorial and Survey
The DOI matches its Crossref record; 144 of the 157 references it cites were found in a registry or an index.
Abstract
Deep neural networks (DNNs) are currently widely used for many artificial intelligence (AI) applications including computer vision, speech recognition, and robotics. While DNNs deliver state-of-the-art accuracy on many AI tasks, it comes at the cost of high computational complexity. Accordingly, techniques that enable efficient processing of DNNs to improve energy efficiency and throughput without sacrificing application accuracy or increasing hardware cost are critical to the wide deployment of DNNs in AI systems. This article aims to provide a comprehensive tutorial and survey about the recent advances toward the goal of enabling efficient processing of DNNs. Specifically, it will provide an overview of DNNs, discuss various hardware platforms and architectures that support DNNs, and highlight key trends in reducing the computation cost of DNNs either solely via hardware design changes or via joint hardware design and DNN algorithm changes. It will also summarize various development resources that enable researchers and practitioners to quickly get started in this field, and highlight important benchmarking metrics and design considerations that should be used for evaluating the rapidly growing number of DNN hardware designs, optionally including algorithmic codesigns, being proposed in academia and industry. The reader will take away the following concepts from this article: understand the key design considerations for DNNs; be able to evaluate different DNN hardware implementations with benchmarks and comparison metrics; understand the tradeoffs between various hardware architectures and platforms; be able to evaluate the utility of various DNN design techniques for efficient processing; and understand recent implementation trends and opportunities.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Classical arithmetic operations, such as MAC, constitute the bulk of the computational requirements for models used in artificial intelligence [13].”
From Brain-inspired computing systems: a systematic literature review (2024) · cites this paper as [13]
In the citing paper
“To clarify, consider the multiply-and-accumulate (MAC) operation in digital processors, where the operation energy consumption is significantly dependent on the bit width of the input operands [13, 235] or equivalently to the signal-to-noise ratio (SNR) in analog computing [236].”
From Brain-inspired computing systems: a systematic literature review (2024) · cites this paper as [13]
In the citing paper
“Altogether, processing one 224 × 224 input image needs a VGG-16 model of 15.5 G multiply-and-accumulates and 138 M weights [40].”
From Face Recognition Based on Deep Learning and FPGA for Ethnicity Identification (2022) · cites this paper as [40]
In the citing paper
“The energy consumption of executing a DNN model is highly dictated by the computation architecture and methods (e.g., ultra-low power compute-in-memory AI accelerator) at the edge computation nodes [249], the architecture of DNN models [250], and the wireless transmission for data exchange during the model inference procedure.”
From Edge artificial intelligence for 6g: vision, enabling technologies, and applications (2022) · cites this paper as [250]
In the citing paper
“but there is no review discussing all three categories which in future will be integrated seamlessly to design effective and efficient DNNs in the new edge intelligence era.”
From Bringing ai to edge: from deep learning’s perspective (2022) · cites this paper as [25]
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · cited 15×
“In 2015, the ImageNet winning entry, ResNet [15], exceeded human-level accuracy with a top-5 error rate4 below 5%.”
InfluentialVery Deep Convolutional Networks for Large-Scale Image Recognition
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1409.1556 · Cited by 114,148 · cited 10×
“Used in VGG-16. decompose 5x5 filter 5x1 filter 1x5 filter Apply sequentially (b) Constructing a 5×5 support from 1×5 and 5×1 filter.”
InfluentialImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155 · cited 10×
“this, local response normalization (LRN) [3] was used, which”
BackgroundMethodInfluentialHochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318 · cited 1×
“In contrast, recurrent neural networks (RNNs), of which Long Short-Term Memory networks (LSTMs) [38] are a popular variant, have internal memory to allow long-term dependencies to affect the output.”
LeCun, Y., Bengio, Y., & Hinton, G. (2015) · Nature · doi:10.1038/nature14539 · Cited by 84,126
Gradient-based learning applied to document recognition
Lecun, Y., Bottou, L., Bengio, Y., & Haffner, P. (1998) · Proceedings of the IEEE · doi:10.1109/5.726791 · Cited by 58,805 · cited 4×
“Table 2 Summary of Popular DNNs [3], [15], [48], [50], [51]”
InfluentialGoing deeper with convolutions
Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., & Rabinovich, A. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298594 · Cited by 48,562
ImageNet Large Scale Visual Recognition Challenge
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al. (2015) · International Journal of Computer Vision · doi:10.1007/s11263-015-0816-y · Cited by 39,380 · cited 8×
“†ACCURACY IS MEASURED BASED ON TOP-5 ERROR ON IMAGENET [14].”
InfluentialFully convolutional networks for semantic segmentation
Long, J., Shelhamer, E., & Darrell, T. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298965 · Cited by 38,834
Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation
Girshick, R., Donahue, J., Darrell, T., & Malik, J. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2014.81 · Cited by 32,684
Rethinking the Inception Architecture for Computer Vision
Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., & Wojna, Z. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.308 · Cited by 32,138
Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
Ioffe, S., & Szegedy, C. (2015) · arXiv · doi:10.48550/arxiv.1502.03167 · Cited by 27,818 · cited 2×
“(2), where the parameters (γ, β) are learned from training [47].”
Xception: Deep Learning with Depthwise Separable Convolutions
Chollet, F. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.195 · Cited by 20,380 · cited 1×
“, applied only on one of the input channels) followed by 1 × 1 3-D convolutions as demonstrated in Xception [155] and MobileNets [156].”
MethodDelving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification
He, K., Zhang, X., Ren, S., & Sun, J. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.123 · Cited by 19,741 · cited 1×
“Variations of ReLU, such as leaky ReLU [41], parametric ReLU [42], and exponential LU [43] have also been explored for improved accuracy.”
BackgroundMobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
Howard, A. G., Zhu, M., Chen, B., Kalenichenko, D., Wang, W., Weyand, T., Andreetto, M., & Adam, H. (2017) · arXiv · doi:10.48550/arxiv.1704.04861 · Cited by 19,015 · cited 1×
“, applied only on one of the input channels) followed by 1 × 1 3-D convolutions as demonstrated in Xception [155] and MobileNets [156].”
MethodMastering the game of Go with deep neural networks and tree search
Silver, D., Huang, A., Maddison, C. J., Guez, A., Sifre, L., van den Driessche, G., Schrittwieser, J., Antonoglou, I., Panneershelvam, V., Lanctot, M., et al. (2016) · Nature · doi:10.1038/nature16961 · Cited by 15,612 · cited 3×
“CNN are widely used in a variety of applications including image understanding [3], speech recognition [39], game play [6], robotics [32], etc.”
Dermatologist-level classification of skin cancer with deep neural networks
Esteva, A., Kuprel, B., Novoa, R. A., Ko, J., Swetter, S. M., Blau, H. M., & Thrun, S. (2017) · Nature · doi:10.1038/nature21056 · Cited by 14,362 · cited 2×
“They have also been used in medical imaging to detect skin cancer [5], brain cancer [28], and breast cancer [29].”
BackgroundDistilling the Knowledge in a Neural Network
Hinton, G., Vinyals, O., & Dean, J. (2015) · arXiv · doi:10.48550/arxiv.1503.02531 · Cited by 14,204
(2010) · International Conference on Machine Learning · Cited by 13,995
Caffe: Convolutional Architecture for Fast Feature Embedding
Jia, Y., Shelhamer, E., Donahue, J., Karayev, S., Long, J., Girshick, R., Guadarrama, S., & Darrell, T. (2014) · Proceedings of the 22nd ACM international conference on Multimedia · doi:10.1145/2647868.2654889 · Cited by 11,077 · cited 3×
“nfigured based on the size of 9 3 5 10 32 2 1 3 21 9 2 6 11 7 2x2 pooling, stride 2 32 5 6 21 Max pooling Average pooling 18 3 3 12 Fig. 9. Various forms of pooling (Figure adopted from Caffe Tutorial [43]). its receptive field (e.g., 2 2) and the type of pooling (e.g., max or average), as shown in Fig. 9. Typically the…”
MethodInfluentialMemristor—The Missing Circuit Element
Chua, L. (1971) · IEEE Transactions on Circuit Theory · doi:10.1109/tct.1971.1083337 · Cited by 10,268 · cited 1×
“The multiply and accumulate operation can also be directly integrated into advanced non-volatile high density memories by using them as programmable resistive elements, commonly referred to as memristors [105].”
Hinton, G., Deng, L., Yu, D., Dahl, G., Mohamed, A.-R., Jaitly, N., Senior, A., Vanhoucke, V., Nguyen, P., Sainath, T., et al. (2012) · IEEE Signal Processing Magazine · doi:10.1109/msp.2012.2205597 · Cited by 10,260 · cited 1×
“• Speech and Language DNNs have significantly improved the accuracy of speech recognition [21] as well as many related tasks such as machine translation [2], natural language processing [22], and audio generation [23].”
Playing Atari with Deep Reinforcement Learning
Mnih, V., Kavukcuoglu, K., Silver, D., Graves, A., Antonoglou, I., Wierstra, D., & Riedmiller, M. (2013) · arXiv · doi:10.48550/arxiv.1312.5602 · Cited by 9,471 · cited 1×
“DNNs have surpassed human level accuracy in playing Atari [31] as well as Go [6], where an exhaustive search of all possibilities is not feasible due to the unimaginably huge number of possible moves.”
Reinforcement Learning: A Survey
Kaelbling, L. P., Littman, M. L., & Moore, A. W. (1996) · Journal of Artificial Intelligence Research · doi:10.1613/jair.301 · Cited by 8,972 · cited 1×
“y Recently, many of the grand AI challenges involving game play have been overcome using DNNs. These successes also required innovations in training techniques and many rely on reinforcement learning [30]. DNNs have surpassed human level accuracy in playing Atari [31] as well as Go [6], where an exhaustive search of a…”
BackgroundDomain-Adversarial Training of Neural Networks
Ganin, Y., Ustinova, E., Ajakan, H., Germain, P., Larochelle, H., Laviolette, F., Marchand, M., & Lempitsky, V. (2017) · Advances in Computer Vision and Pattern Recognition · doi:10.1007/978-3-319-58347-1_10 · Cited by 8,318
Two-Stream Convolutional Networks for Action Recognition in Videos
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1406.2199 · Cited by 6,756 · cited 1×
“DNNs have significantly improved the accuracy of many computer vision tasks such as image classification [14], object localization and detection [18], image segmentation [19], and action recognition [20].”
SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <0.5MB model size
Iandola, F. N., Han, S., Moskewicz, M. W., Ashraf, K., Dally, W. J., & Keutzer, K. (2016) · arXiv · doi:10.48550/arxiv.1602.07360 · Cited by 6,201
Natural Language Processing (almost) from Scratch
Collobert, R., Weston, J., Bottou, L., Karlen, M., Kavukcuoglu, K., & Kuksa, P. P. (2011) · arXiv (Cornell University) · doi:10.48550/arxiv.1103.0398 · Cited by 4,962 · cited 1×
“• Speech and Language DNNs have significantly improved the accuracy of speech recognition [21] as well as many related tasks such as machine translation [2], natural language processing [22], and audio generation [23].”
WaveNet: a generative model for raw audio
Oord, A. V. D., Dieleman, S., Zen, H., Simonyan, K., Vinyals, O., Graves, A., Kalchbrenner, N., Senior, A., & Kavukcuoglu, K. (2016) · arXiv · doi:10.48550/arxiv.1609.03499 · Cited by 4,911 · cited 1×
“• Speech and Language DNNs have significantly improved the accuracy of speech recognition [21] as well as many related tasks such as machine translation [2], natural language processing [22], and audio generation [23].”
Han, S., Mao, H., & Dally, W. J. (2015) · arXiv · doi:10.48550/arxiv.1510.00149 · Cited by 4,662
Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learning
Szegedy, C., Ioffe, S., Vanhoucke, V., & Alemi, A. (2017) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v31i1.11231 · Cited by 4,636 · cited 1×
“v4 uses residual connections [51], described the next section, for at 0.”
Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)
Clevert, D.-A., Unterthiner, T., & Hochreiter, S. (2015) · arXiv · doi:10.48550/arxiv.1511.07289 · Cited by 4,513 · cited 1×
“Variations of ReLU, such as leaky ReLU [41], parametric ReLU [42], and exponential LU [43] have also been explored for improved accuracy.”
XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networks
Rastegari, M., Ordonez, V., Redmon, J., & Farhadi, A. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46493-0_32 · Cited by 4,408
A million spiking-neuron integrated circuit with a scalable communication network and interface
Merolla, P. A., Arthur, J. V., Alvarez-Icaza, R., Cassidy, A. S., Sawada, J., Akopyan, F., Jackson, B. L., Imam, N., Guo, C., Nakamura, Y., et al. (2014) · Science · doi:10.1126/science.1254642 · Cited by 4,147 · cited 1×
“An example of a project that was inspired by the spiking of the brain is the IBM TrueNorth [8].”
WIDROW, B., & HOFF, M. E. (1960) · doi:10.21236/ad0241531 · Cited by 4,009
Eyeriss: An Energy-Efficient Reconfigurable Accelerator for Deep Convolutional Neural Networks
Chen, Y.-H., Krishna, T., Emer, J. S., & Sze, V. (2017) · IEEE Journal of Solid-State Circuits · doi:10.1109/jssc.2016.2616357 · Cited by 3,247
Self-citation(1989) · Cited by 3,214
Training and operation of an integrated neuromorphic network based on metal-oxide memristors
Prezioso, M., Merrikh-Bayat, F., Hoskins, B. D., Adam, G. C., Likharev, K. K., & Strukov, D. B. (2015) · Nature · doi:10.1038/nature14441 · Cited by 3,001
Predicting the sequence specificities of DNA- and RNA-binding proteins by deep learning
Alipanahi, B., Delong, A., Weirauch, M. T., & Frey, B. J. (2015) · Nature Biotechnology · doi:10.1038/nbt.3300 · Cited by 2,873
FitNets: Hints for Thin Deep Nets
Romero, A., Ballas, N., Kahou, S. E., Chassang, A., Gatta, C., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1412.6550 · Cited by 2,859 · cited 1×
“the teacher DNN can also be incorporated as the extra hints to train the student DNN [161].”
Convergence Results for Neural Networks via Electrodynamics
Panigrahy, R., Rahimi, A., Sachdeva, S., & Zhang, Q. (2018) · Leibniz international proceedings in informatics · doi:10.4230/lipics.itcs.2018.22 · Cited by 2,761
OverFeat: Integrated Recognition, Localization and Detection using Convolutional Networks
Sermanet, P., Eigen, D., Zhang, X., Mathieu, M., Fergus, R., & LeCun, Y. (2013) · arXiv · doi:10.48550/arxiv.1312.6229 · Cited by 2,680 · cited 1×
“Overfeat [49] has a very similar architecture to AlexNet with five CONV layers and three FC layers.”
Two-Dimensional Signal and Image Processing
Lim, J. S. (1989) · Medical Entomology and Zoology · Cited by 2,561
Predicting effects of noncoding variants with deep learning–based sequence model
Zhou, J., & Troyanskaya, O. G. (2015) · Nature Methods · doi:10.1038/nmeth.3547 · Cited by 2,338
Convolutional networks and applications in vision
LeCun, Y., Kavukcuoglu, K., & Farabet, C. (2010) · Proceedings of 2010 IEEE International Symposium on Circuits and Systems · doi:10.1109/iscas.2010.5537907 · Cited by 2,213 · cited 1×
“a form of structured sparsity is orthogonal to the sparsity that occurs from network pruning as described in Section VII-B2. Accordingly, a convolutional neural network (CNN) is a popular form of DNN [35]. 1) Convolutional Neural Networks (CNNs): CNNs are composed of multiple convolutional layers (CONV), as shown in F…”
BackgroundBinaryConnect: Training Deep Neural Networks with binary weights during propagations
Courbariaux, M., Bengio, Y., & David, J.-P. (2015) · arXiv · doi:10.48550/arxiv.1511.00363 · Cited by 2,180 · cited 2×
“Reduce Weight BinaryConnect [127] 1 32 (float) 19.”
Optimizing FPGA-based Accelerator Design for Deep Convolutional Neural Networks
Zhang, C., Li, P., Sun, G., Guan, Y., Xiao, B., & Cong, J. (2015) · Proceedings of the 2015 ACM/SIGDA International Symposium on Field-Programmable Gate Arrays · doi:10.1145/2684746.2689060 · Cited by 2,158 · cited 1×
“In an example of the no local reuse dataflow from UCLA [91], the filter weights and input activations are read from the global buffer, processed by the MAC units with custom adder trees that can complete the accumulation in a single cycle, and the resulting partial sums or output activations are then put back to the g…”
Courbariaux, M., Hubara, I., Soudry, D., El-Yaniv, R., & Bengio, Y. (2016) · arXiv · doi:10.48550/arxiv.1602.02830 · Cited by 2,106 · cited 4×
“However, BC and BNN have an accuracy loss of 19% and 29.8%, respectively [129].”
Influential1.1 Computing's energy problem (and what we can do about it)
Horowitz, M. (2014) · 2014 IEEE International Solid-State Circuits Conference Digest of Technical Papers (ISSCC) · doi:10.1109/isscc.2014.6757323 · Cited by 2,069
Buciluǎ, C., Caruana, R., & Niculescu-Mizil, A. (2006) · Proceedings of the 12th ACM SIGKDD international conference on Knowledge discovery and data mining · doi:10.1145/1150402.1150464 · Cited by 2,068
(2013) · Cited by 2,056
EIE: efficient inference engine on compressed deep neural network
Han, S., Liu, X., Mao, H., Pu, J., Pedram, A., Horowitz, M. A., & Dally, W. J. (2016) · ACM SIGARCH Computer Architecture News · doi:10.1145/3007787.3001163 · Cited by 1,937
80 Million Tiny Images: A Large Data Set for Nonparametric Object and Scene Recognition
Torralba, A., Fergus, R., & Freeman, W. T. (2008) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2008.128 · Cited by 1,921
DeepDriving: Learning Affordance for Direct Perception in Autonomous Driving
Chen, C., Seff, A., Kornhauser, A., & Xiao, J. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.312 · Cited by 1,876 · cited 2×
“These DNNs are employed in a myriad of applications from self-driving cars [4], to detecting cancer [5] to playing complex games [6].”
DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients
Zhou, S., Wu, Y., Ni, Z., Zhou, X., Wen, H., & Zou, Y. (2016) · arXiv · doi:10.48550/arxiv.1606.06160 · Cited by 1,690
ISAAC: a convolutional neural network accelerator with in-situ analog arithmetic in crossbars
Shafiee, A., Nag, A., Muralimanohar, N., Balasubramonian, R., Strachan, J. P., Hu, M., Williams, R. S., & Srikumar, V. (2016) · ACM SIGARCH Computer Architecture News · doi:10.1145/3007787.3001139 · Cited by 1,536 · cited 2×
“(b) Gi is conductance of resistive memory (Figure from [106]).”
BackgroundMethodDaDianNao: A Machine-Learning Supercomputer
Chen, Y., Luo, T., Liu, S., Zhang, S., He, L., Wang, J., Li, L., Chen, T., Xu, Z., Sun, N., et al. (2014) · 2014 47th Annual IEEE/ACM International Symposium on Microarchitecture · doi:10.1109/micro.2014.58 · Cited by 1,492 · cited 5×
“Another example of no local reuse dataflow is found in [93].”
InfluentialSelf-citationLearning both Weights and Connections for Efficient Neural Networks
Han, S., Pool, J., Tran, J., & Dally, W. J. (2015) · arXiv · doi:10.48550/arxiv.1506.02626 · Cited by 1,481
Exploiting Linear Structure Within Convolutional Networks for Efficient Evaluation
Denton, R., Zaremba, W., Bruna, J., LeCun, Y., & Fergus, R. (2014) · arXiv · doi:10.48550/arxiv.1404.0736 · Cited by 1,414
Deep Learning with Limited Numerical Precision
Gupta, S., Agrawal, A., Gopalakrishnan, K., & Narayanan, P. (2015) · arXiv · doi:10.48550/arxiv.1502.02551 · Cited by 1,364 · cited 4×
“Other examples of output stationary are found in [88, 90].”
InfluentialEnd-to-End Training of Deep Visuomotor Policies
Levine, S., Finn, C., Darrell, T. J., & Abbeel, P. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1504.00702 · Cited by 1,344 · cited 2×
“CNN are widely used in a variety of applications including image understanding [3], speech recognition [39], game play [6], robotics [32], etc.”
Do Deep Nets Really Need to be Deep?
Ba, J. L., & Caruana, R. (2013) · arXiv (Cornell University) · doi:10.48550/arxiv.1312.6184 · Cited by 1,335
DianNao: a small-footprint high-throughput accelerator for ubiquitous machine-learning
Chen, T., Du, Z., Sun, N., Wang, J., Wu, C., Chen, Y., & Temam, O. (2014) · Proceedings of the 19th international conference on Architectural support for programming languages and operating systems · doi:10.1145/2541940.2541967 · Cited by 1,297 · cited 6×
“DNN Timeline • 1940s - Neural networks were proposed • 1960s - Deep neural networks were proposed • 1989 - Neural networks for recognizing digits (LeNet) • 1990s - Hardware for shallow neural nets (Intel ETANN) • 2011 - Breakthrough DNN-based speech recognition (Microsoft) • 2012 - DNNs for vision start supplanting ha…”
InfluentialSelf-citationChi, P., Li, S., Xu, C., Zhang, T., Zhao, J., Liu, Y., Wang, Y., & Xie, Y. (2016) · ACM SIGARCH Computer Architecture News · doi:10.1145/3007787.3001140 · Cited by 1,231 · cited 2×
“PRIME [109] also replaces the DRAM main memory with memristors; specifically, it uses 256×256 memristor arrays that can be configured for 4-bit multi-level cell computation or 1-bit single level cell storage.”
The human splicing code reveals new insights into the genetic determinants of disease
Xiong, H. Y., Alipanahi, B., Lee, L. J., Bretschneider, H., Merico, D., Yuen, R. K. C., Hua, Y., Gueroussov, S., Najafabadi, H. S., Hughes, T. R., et al. (2015) · Science · doi:10.1126/science.1254806 · Cited by 1,220 · cited 1×
“• Medical DNNs: They have played an important role in genomics to gain insight into the genetics of diseases such as autism, cancers, and spinal muscular atrophy [24]–[27].”
Quantized Convolutional Neural Networks for Mobile Devices
Wu, J., Leng, C., Wang, Y., Hu, Q., & Cheng, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.521 · Cited by 1,212
Eyeriss: a spatial architecture for energy-efficient dataflow for convolutional neural networks
Chen, Y.-H., Emer, J., & Sze, V. (2016) · ACM SIGARCH Computer Architecture News · doi:10.1145/3007787.3001177 · Cited by 1,182
Self-citationDeep convolutional neural networks for LVCSR
Sainath, T. N., Mohamed, A.-R., Kingsbury, B., & Ramabhadran, B. (2013) · 2013 IEEE International Conference on Acoustics, Speech and Signal Processing · doi:10.1109/icassp.2013.6639347 · Cited by 1,053 · cited 1×
“CNN are widely used in a variety of applications including image understanding [3], speech recognition [39], game play [6], robotics [32], etc.”
Compressing Neural Networks with the Hashing Trick
Chen, W., Wilson, J. T., Tyree, S., Weinberger, K. Q., & Chen, Y. (2015) · arXiv · doi:10.48550/arxiv.1504.04788 · Cited by 974
Self-citationcuDNN: Efficient Primitives for Deep Learning
Chetlur, S., Woolley, C., Vandermersch, P., Cohen, J., Tran, J. J., Catanzaro, B., & Shelhamer, E. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1410.0759 · Cited by 943
(2014) · arXiv (Cornell University) · Cited by 942
ShiDianNao: shifting vision processing closer to the sensor
Du, Z., Fasthuber, R., Chen, T., Ienne, P., Li, L., Luo, T., Feng, X., Chen, Y., & Temam, O. (2015) · Proceedings of the 42nd Annual International Symposium on Computer Architecture · doi:10.1145/2749469.2750389 · Cited by 933
Self-citationCompression of Deep Convolutional Neural Networks for Fast and Low Power Mobile ApplicationsPreprint
Kim, Y., Park, E., Yoo, S., Choi, T., Lu, Y., & Shin, D. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1511.06530 · Cited by 920
Fast Algorithms for Convolutional Neural Networks
Lavin, A., & Gray, S. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.435 · Cited by 915 · cited 3×
“Other approaches include Strassen [75] and Winograd [76], which rearrange the computation such that the number of multiplications reduce from O(N(3)) to O(N) and by 2.”
SCNN: An Accelerator for Compressed-sparse Convolutional Neural Networks
Parashar, A., Rhu, M., Mukkara, A., Puglielli, A., Venkatesan, R., Khailany, B., Emer, J., Keckler, S. W., & Dally, W. J. (2017) · Proceedings of the 44th Annual International Symposium on Computer Architecture · doi:10.1145/3079856.3080254 · Cited by 896 · cited 2×
“SCNN [146] supports processing of convolutional”
Self-citationZhu, C., Han, S., Mao, H., & Dally, W. J. (2016) · arXiv · doi:10.48550/arxiv.1612.01064 · Cited by 865
Recent advances in deep learning for speech research at Microsoft
Deng, L., Li, J., Huang, J.-T., Yao, K., Yu, D., Seide, F., Seltzer, M., Zweig, G., He, X., Williams, J., et al. (2013) · 2013 IEEE International Conference on Acoustics, Speech and Signal Processing · doi:10.1109/icassp.2013.6639345 · Cited by 801 · cited 3×
“Since the breakthrough application of DNNs to speech recognition [2] and image recognition [3], the number of applications that use DNNs has exploded.”
Convolutional networks for fast, energy-efficient neuromorphic computing
Esser, S. K., Merolla, P. A., Arthur, J. V., Cassidy, A. S., Appuswamy, R., Andreopoulos, A., Berg, D. J., McKinstry, J. L., Melano, T., Barch, D. R., et al. (2016) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1604850113 · Cited by 781 · cited 2×
“1Note: Recent work using TrueNorth in a stylized fashion allows it to be used to compute reduced precision neural networks [9].”
MethodDeep Learning with Limited Numerical Precision
Gupta, S., Agrawal, A., Gopalakrishnan, K., & Narayanan, P. (2015) · Cited by 757
Deep Learning for Identifying Metastatic Breast Cancer
Wang, D., Khosla, A., Gargeya, R., Irshad, H., & Beck, A. H. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1606.05718 · Cited by 756 · cited 1×
“They have also been used in medical imaging to detect skin cancer [5], brain cancer [28] and breast cancer [29].”
Li, F., Liu, B., Wang, X., Zhang, B., & Yan, J. (2016) · arXiv · doi:10.48550/arxiv.1605.04711 · Cited by 732
Designing Energy-Efficient Convolutional Neural Networks Using Energy-Aware Pruning
Yang, T.-J., Chen, Y.-H., & Sze, V. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.643 · Cited by 732 · cited 4×
“Table 4 shows another example of these DNN model metrics, by comparing sparse DNNs pruned using [145] to dense DNNs.”
MethodInfluentialSelf-citation14.5 Eyeriss: An energy-efficient reconfigurable accelerator for deep convolutional neural networks
Chen, Y.-H., Krishna, T., Emer, J., & Sze, V. (2016) · 2016 IEEE International Solid-State Circuits Conference (ISSCC) · doi:10.1109/isscc.2016.7418007 · Cited by 713
Self-citationStructured Pruning of Deep Convolutional Neural Networks
Anwar, S., Hwang, K., & Sung, W. (2017) · ACM Journal on Emerging Technologies in Computing Systems · doi:10.1145/3005348 · Cited by 698
Alwani, M., Chen, H., Ferdman, M., & Milder, P. (2016) · 2016 49th Annual IEEE/ACM International Symposium on Microarchitecture (MICRO) · doi:10.1109/micro.2016.7783725 · Cited by 662
Cnvlutin: ineffectual-neuron-free deep neural network computing
Albericio, J., Judd, P., Hetherington, T., Aamodt, T., Jerger, N. E., & Moshovos, A. (2016) · ACM SIGARCH Computer Architecture News · doi:10.1145/3007787.3001138 · Cited by 614
Convolutional neural network architectures for predicting DNA–protein binding
Zeng, H., Edwards, M. D., Liu, G., & Gifford, D. K. (2016) · Bioinformatics · doi:10.1093/bioinformatics/btw255 · Cited by 511 · cited 1×
“• Medical DNNs: They have played an important role in genomics to gain insight into the genetics of diseases such as autism, cancers, and spinal muscular atrophy [24]–[27].”
BackgroundSafe, Multi-Agent, Reinforcement Learning for Autonomous Driving
Shalev-Shwartz, S., Shammah, S., & Shashua, A. (2016) · arXiv · doi:10.48550/arxiv.1610.03295 · Cited by 509
Cognitive Mapping and Planning for Visual Navigation
Gupta, S., Davidson, J., Levine, S., Sukthankar, R., & Malik, J. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.769 · Cited by 500
Deep Learning with Low Precision by Half-Wave Gaussian Quantization
Cai, Z., He, X., Sun, J., & Vasconcelos, N. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.574 · Cited by 498
Handwritten digit recognition: applications of neural network chips and automatic learning
Le Cun, Y., Jackel, L. D., Boser, B., Denker, J. S., Graf, H. P., Guyon, I., Henderson, D., Howard, R. E., & Hubbard, W. (1989) · IEEE Communications Magazine · doi:10.1109/35.41400 · Cited by 480 · cited 2×
“ts the accuracy if only a single crop was used (i.e., the DNN is run only once), which is more consistent with what would likely be deployed in real-time and/or energy-constrained applications. LeNet [11] was one of the first CNN approaches introduced in 1989. It was designed for the task of digit classification in gray…”
BackgroundMethodLearning deep control policies for autonomous aerial vehicles with MPC-guided policy search
Zhang, T., Kahn, G., Levine, S., & Abbeel, P. (2016) · 2016 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra.2016.7487175 · Cited by 452 · cited 1×
“• Robotics DNNs have been successful in the domain of robotic tasks such as grasping with a robotic arm [32], motion planning for ground robots [33], visual navigation [4, 34], control to stabilize a quadcopter [35] and driving strategies for autonomous vehicles [36].”
High Efficiency Video Coding (HEVC): Algorithms and Architectures
(2014) · Integrated Circuits and Systems · doi:10.1007/978-3-319-06895-4 · Cited by 448
Self-citationMinerva: enabling low-power, highly-accurate deep neural network accelerators
Reagen, B., Whatmough, P., Adolf, R., Rama, S., Lee, H., Lee, S. K., Hernández-Lobato, J. M., Wei, G.-Y., & Brooks, D. (2016) · ACM SIGARCH Computer Architecture News · doi:10.1145/3007787.3001165 · Cited by 446
Hybrid memory cube new DRAM architecture increases density and performance
Jeddeloh, J., & Keeth, B. (2012) · 2012 Symposium on VLSI Technology (VLSIT) · doi:10.1109/vlsit.2012.6242474 · Cited by 444 · cited 1×
“le of using eDRAM for DNN processing can be found in DaDianNao [89], which uses 36MB of eDRAM to store the weights for the fully-connected layers. 3-D memory, such as Micron’s Hyper Memory Cube (HMC) [94], stacks the DRAM on top of the logic using through silicon vias (TSV); this results in an order of magnitude highe…”
MethodIncremental Network Quantization: Towards Lossless CNNs with Low-Precision Weights
Zhou, A., Yao, A., Guo, Y., Xu, L., & Chen, Y. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1702.03044 · Cited by 439
Self-citationTETRIS: Scalable and Efficient Neural Network Acceleration with 3D Memory
Gao, M., Pu, J., Yang, X., Horowitz, M., & Kozyrakis, C. (2017) · Proceedings of the Twenty-Second International Conference on Architectural Support for Programming Languages and Operating Systems · doi:10.1145/3037697.3037702 · Cited by 421
Pfeiffer, M., Schaeuble, M., Nieto, J., Siegwart, R., & Cadena, C. (2017) · 2017 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra.2017.7989182 · Cited by 413 · cited 1×
“• Robotics DNNs have been successful in the domain of robotics tasks such as grasping with a robotic arm [28], motion planning for ground robots [29], visual navigation [4, 30], control to stabilize a quadcopter [31] and driving strategies for autonomous vehicles [32].”
Learning Structured Sparsity in Deep Neural Networks
Wen, W. X., Wu, C., Wang, Y., Chen, Y., & Li, H. H. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1608.03665 · Cited by 402
Self-citationMinimizing Computation in Convolutional Neural Networks
Cong, J., & Xiao, B. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-11179-7_36 · Cited by 372 · cited 2×
“Other approaches include Strassen [75] and Winograd [76], which rearrange the computation such that the number of multiplications reduce from O(N(3)) to O(N) and by 2.”
Towards End-to-End Speech Recognition with Deep Convolutional Neural Networks
Zhang, Y., Pezeshki, M., Brakel, P., Zhang, S., Laurent, C., Bengio, Y., & Courville, A. (2016) · Interspeech 2016 · doi:10.21437/interspeech.2016-1446 · Cited by 352 · cited 1×
“Finally, a non-linearity called maxout, which takes the max value of two intersecting linear functions, has shown to be effective in speech recognition tasks [44, 45].”
Fast Training of Convolutional Networks through FFTs
Mathieu, M., Henaff, M., & LeCun, Y. (2014) · International Conference on Learning Representations · Cited by 347
A dynamically configurable coprocessor for convolutional neural networks
Chakradhar, S., Sankaradas, M., Jakkula, V., & Cadambi, S. (2010) · Proceedings of the 37th annual international symposium on Computer architecture · doi:10.1145/1815961.1815993 · Cited by 343 · cited 1×
“Other weight stationary examples are found in [84]–[86], [88], [89].”
BackgroundNeurocube: a programmable digital neuromorphic architecture with high-density 3D memory
Kim, D., Kung, J., Chai, S., Yalamanchili, S., & Mukhopadhyay, S. (2016) · ACM SIGARCH Computer Architecture News · doi:10.1145/3007787.3001178 · Cited by 342
Graham, B. (2014) · arXiv · doi:10.48550/arxiv.1412.6071 · Cited by 337
Stripes: Bit-serial deep neural network computing
Judd, P., Albericio, J., Hetherington, T., Aamodt, T. M., & Moshovos, A. (2016) · 2016 49th Annual IEEE/ACM International Symposium on Microarchitecture (MICRO) · doi:10.1109/micro.2016.7783722 · Cited by 333
Memory-centric accelerator design for Convolutional Neural Networks
Peemen, M., Setio, A. A. A., Mesman, B., & Corporaal, H. (2013) · 2013 IEEE 31st International Conference on Computer Design (ICCD) · doi:10.1109/iccd.2013.6657019 · Cited by 332
Convolutional Neural Networks using Logarithmic Data Representation
Miyashita, D., Lee, E. H., & Murmann, B. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1603.01025 · Cited by 329 · cited 3×
“Furthermore, when weights are quantized to powers of two, the multiplication can be replaced with a bitshift [122, 135].”
InfluentialA 240 G-ops/s Mobile Coprocessor for Deep Neural Networks
Gokhale, V., Jin, J., Dundar, A., Martini, B., & Culurciello, E. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition Workshops · doi:10.1109/cvprw.2014.106 · Cited by 300 · cited 1×
“One example of previous work that implement weight stationary dataflow is nn-X, or neuFlow [85], which uses eight 2-D convolution engines for processing a 10×10 filter.”
Hardware-oriented Approximation of Convolutional Neural Networks
Gysel, P., Motamedi, M., & Ghiasi, S. (2016) · arXiv · doi:10.48550/arxiv.1604.03168 · Cited by 287
Speeding-up Convolutional Neural Networks Using Fine-tuned CP-Decomposition
Lebedev, V., Ganin, Y., Rakhuba, M., Oseledets, I. V., & Lempitsky, V. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1412.6553 · Cited by 283
Improving deep neural network acoustic models using generalized maxout networks
Zhang, X., Trmal, J., Povey, D., & Khudanpur, S. (2014) · 2014 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2014.6853589 · Cited by 273 · cited 1×
“Finally, a non-linearity called maxout, which takes the max value of two intersecting linear functions, has shown to be effective in speech recognition tasks [44, 45].”
Scalpel: Customizing DNN Pruning to the Underlying Hardware Parallelism
Yu, J., Lukefahr, A., Palframan, D., Dasika, G., Das, R., & Mahlke, S. (2017) · Proceedings of the 44th Annual International Symposium on Computer Architecture · doi:10.1145/3079856.3080215 · Cited by 272 · cited 1×
“, SIMD) found in existing general purpose hardware, which results in more efficient processing [149]; (2) it amortizes the overhead cost required to signal the location of the non-zero weights across a group of weights, which improves compression and thus reduces storage cost.”
Do Deep Convolutional Nets Really Need to be Deep and Convolutional?Preprint
Urban, G., Geras, K. J., Kahou, S. E., Aslan, Ö., Wang, S., Caruana, R., Mohamed, A., Philipose, M., & Richardson, M. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1603.05691 · Cited by 247
A Massively Parallel Coprocessor for Convolutional Neural Networks
Sankaradas, M., Jakkula, V., Cadambi, S., Chakradhar, S., Durdanovic, I., Cosatto, E., & Graf, H. P. (2009) · 2009 20th IEEE International Conference on Application-specific Systems, Architectures and Processors · doi:10.1109/asap.2009.25 · Cited by 224
Exploring the Regularity of Sparse Structure in Convolutional Neural Networks
Mao, H., Han, S., Pool, J., Li, W., Liu, X., Wang, Y., & Dally, W. J. (2017) · arXiv · doi:10.48550/arxiv.1705.08922 · Cited by 219 · cited 1×
“These groups of weights can include a pair of neighboring weights, an entire row or column of a filter, an entire channel of a filter or the entire filter itself; using larger groups tends to result in higher loss in accuracy [150].”
YodaNN: An Ultra-Low Power Convolutional Neural Network Accelerator Based on Binary Weights
Andri, R., Cavigelli, L., Rossi, D., & Benini, L. (2016) · 2016 IEEE Computer Society Annual Symposium on VLSI (ISVLSI) · doi:10.1109/isvlsi.2016.111 · Cited by 211 · cited 1×
“YodaNN [133] uses binary weights, while BRein [134] uses binary weights and activations.”
LogNet: Energy-efficient neural networks using logarithmic computation
Lee, E. H., Miyashita, D., Chai, E., Murmann, B., & Wong, S. S. (2017) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2017.7953288 · Cited by 193
RedEye: analog ConvNet image sensor architecture for continuous mobile vision
LiKamWa, R., Hou, Y., Gao, J., Polansky, M., & Zhong, L. (2016) · ACM SIGARCH Computer Architecture News · doi:10.1145/3007787.3001164 · Cited by 189
Origami: A Convolutional Network Accelerator
Cavigelli, L., Gschwend, D., Mayer, C., Willi, S., Muheim, B., & Benini, L. (2015) · ACM Great Lakes Symposium on VLSI (GLSVLSI) · doi:10.1145/2742060.2743766 · Cited by 180
Using Dataflow to Optimize Energy Efficiency of Deep Neural Network Accelerators
Chen, Y., Emer, J., & Sze, V. (2017) · IEEE Micro · doi:10.1109/mm.2017.54 · Cited by 171
Self-citationA 0.3–2.6 TOPS/W precision-scalable processor for real-time large-scale ConvNets
Moons, B., & Verhelst, M. (2016) · 2016 IEEE Symposium on VLSI Circuits (VLSI-Circuits) · doi:10.1109/vlsic.2016.7573525 · Cited by 161
Scalable and modularized RTL compilation of Convolutional Neural Networks onto FPGA
Ma, Y., Suda, N., Cao, Y., Seo, J.-S., & Vrudhula, S. (2016) · 2016 26th International Conference on Field Programmable Logic and Applications (FPL) · doi:10.1109/fpl.2016.7577356 · Cited by 161
Scalpel: Customizing DNN Pruning to the Underlying Hardware Parallelism
Yu, J., Lukefahr, A., Palframan, D. J., Dasika, G. S., Das, R., & Mahlke, S. A. (2017) · ACM SIGARCH Computer Architecture News · doi:10.1145/3140659.3080215 · Cited by 152
Exact Acceleration of Linear Object Detectors
Dubout, C., & Fleuret, F. (2012) · Lecture notes in computer science · doi:10.1007/978-3-642-33712-3_22 · Cited by 150
A machine-learning classifier implemented in a standard 6T SRAM array
Zhang, J., Wang, Z., & Verma, N. (2016) · Symposium on VLSI Circuits · doi:10.1109/vlsic.2016.7573556 · Cited by 149 · cited 3×
“For instance, the multiply and accumulate operation can be directly integrated into the bit-cells of an SRAM array [102], as shown in Fig.”
BackgroundMethodInfluentialPark, S., Bong, K., Shin, D., Lee, J., Choi, S., & Yoo, H. (2015) · Digest of technical papers/Digest of technical papers - IEEE International Solid-State Circuits Conference · doi:10.1109/isscc.2015.7062935 · Cited by 127
Handwritten Digit Recognition: Applications of Neural Net Chips and Automatic Learning
LeCun, Y., Jackel, L. D., Boser, B. E., Denker, J. S., Graf, H. P., Guyon, I., Henderson, D., Howard, R. E., & Hubbard, W. (1990) · Springer eBooks · doi:10.1007/978-3-642-76153-9_35 · Cited by 105
Jermyn, M., Desroches, J., Mercier, J., Tremblay, M.-A., St‐Arnaud, K., Guiot, M., Petrecca, K., & Leblond, F. (2016) · Journal of Biomedical Optics · doi:10.1117/1.jbo.21.9.094002 · Cited by 104 · cited 1×
“They have also been used in medical imaging to detect skin cancer [5], brain cancer [28] and breast cancer [29].”
A scalable sparse matrix-vector multiplication kernel for energy-efficient sparse-blas on FPGAs
Dorrance, R., Ren, F., & Markovic, D. M. (2014) · ACM/SIGDA International Symposium on Field-Programmable Gate Arrays (FPGA) · doi:10.1145/2554688.2554785 · Cited by 89 · cited 2×
“storage formats (figure from [147]).”
BackgroundAndo, K., Ueyoshi, K., Orimo, K., Yonekawa, H., Sato, S., Nakahara, H., Ikebe, M., Asai, T., Takamaeda-Yamazaki, S., Kuroda, T., et al. (2017) · Symposium on VLSI Circuits · doi:10.23919/vlsic.2017.8008533 · Cited by 87
Thinking about thinking: the discovery of the LMS algorithm
Widrow, B. C. (2005) · IEEE Signal Processing Magazine · doi:10.1109/msp.2005.1407720 · Cited by 82 · cited 1×
“3In the early 1960s, single analog neuron systems were used for adaptive filtering [12, 13].”
Using Strassen's algorithm to accelerate the solution of linear systems
Bailey, D. H., Lee, K., & Simon, H. D. (1991) · The Journal of Supercomputing · doi:10.1007/bf00129836 · Cited by 71
Embedded DRAM development: Technology, physical design, and application issues
Keitel-Schulz, D., & Wehn, N. (2001) · IEEE Design & Test of Computers · doi:10.1109/54.922799 · Cited by 70
24.2 A 2.5GHz 7.7TOPS/W switched-capacitor matrix multiplier with co-designed local memory in 40nm
Lee, E. H., & Wong, S. S. (2016) · Digest of technical papers/Digest of technical papers - IEEE International Solid-State Circuits Conference · doi:10.1109/isscc.2016.7418085 · Cited by 69
Chen, H., Jayasuriya, S., Yang, J., Stephen, J., Sivaramakrishnan, S., Veeraraghavan, A., & Molnar, A. C. (2016) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2016.104 · Cited by 66 · cited 1×
“In addition, since the first layer of the DNN often outputs a gradient-like feature map, it maybe possible to skip the computations in the first layer, which further reduces energy consumption as discussed in [115, 116].”
Zhang, J., Wang, Z., & Verma, N. (2015) · Digest of technical papers/Digest of technical papers - IEEE International Solid-State Circuits Conference · doi:10.1109/isscc.2015.7063061 · Cited by 54
Neuromorphic architectures with electronic synapses
Eryilmaz, S. B., Joshi, S., Neftci, E., Wan, W., Cauwenberghs, G., & Wong, H. ‐. P. (2016) · International Symposium on Quality Electronic Design (ISQED) · doi:10.1109/isqed.2016.7479186 · Cited by 42 · cited 1×
“Processing with non-volatile resistive memories has several drawbacks as described in [108].”
Yoo, H., Park, S., Bong, K., Shin, D., Lee, J., & Choi, S. (2015) · Cited by 40
Towards an embedded biologically-inspired machine vision processor
Sriram, V., Cox, D., Tsoi, K. H., & Luk, W. W. (2010) · International Conference on Field-Programmable Technology (FPT) · doi:10.1109/fpt.2010.5681487 · Cited by 36
Energy-efficient HOG-based object detection at 1080HD 60 fps with multi-scale support
Suleiman, A., & Sze, V. (2014) · SiPS ... design and implementation - IEEE Workshop on Signal Processing Systems · doi:10.1109/sips.2014.6986096 · Cited by 36
Self-citationA 180nm CMOS image sensor with on-chip optoelectronic image compression
Wang, A. Z., Sivaramakrishnan, S., & Molnar, A. C. (2012) · IEEE Custom Integrated Circuits Conference (CICC) · doi:10.1109/cicc.2012.6330604 · Cited by 20
Wang, Z., Schapire, R. E., & Verma, N. (2014) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2014.6854329 · Cited by 14 · cited 1×
“device variations to form a strong classifier [105].”
BackgroundQuantized Neural Networks: Training Neural Networks with Low Precision Weights and Activations
Hubara, I., Courbariaux, M., Soudry, D., El-Yaniv, R., & Bengio, Y. (2016) · arXiv · doi:10.48550/arxiv.1609.07061 · cited 4×
“For instance, Quantized Neural Networks (QNN) [119], DoReFa-Net [120], and HWGQ-Net [130] allow the activations to have 2-bits, while the weights remain at 1-bit; in HWGQ-Net, this reduces the accuracy loss to 5.2%.”
Influential
Research works printed but not linked: 34
- 10.13140/RG.2.2.18893.74727 · DOI · Paper page
- Li, Stanford CS Class CS231n: Convolutional Neural Networks for Visual Recognition
- 2016, Complete Visual Networking Index (VNI) Forecast
- Woodhouse, 2016, Big, Big, Big Data: Higher and Higher Resolution Video Surveillance
- Maas, Rectifier nonlinearities improve neural network acoustic models, 1
- Caffe LeNet MNIST
- Caffe Model Zoo
- Matconvnet Pretrained Models
- TensorFlow-Slim Image Classification Library
- Deep Learning Frameworks
- Yann, The MNIST DATABASE of Handwritten Digits
- Krizhevsky, The CIFAR-10 Dataset
- Krizhevsky, 2010, Convolutional deep belief networks on cifar-10, Unpublished Manuscript, 40, 1
- Pascal VOC Data Sets
- Microsoft Common Objects in Context (COCO) Dataset
- Google Open Images
- YouTube-8M
- AudioSet
- 2017, Intel Architecture Instruction Set Extensions Programming Reference
- Condon, Facebook unveils Big Basin, new server geared for deep learning
- Intel Math Kernel Library
- 10.1145/1816038.1815993 · DOI
- 10.1145/2872887.2750389 · DOI
- 10.1145/2644865.2541967 · DOI · Paper page
- Standard, High bandwidth memory (HBM) DRAM
- 10.1145/3093336.3037702 · DOI
- Wilson, International technology roadmap for semiconductors (ITRS)
- 10.1109/icvd.2004.1260946 · doi:10.1109/icvd.2004.1260946
- Lee, A 2.5 GHz 7.7 TOPS/W switched-capacitor matrix multiplier with co-designed local memory in 40 nm, 418
- Lee, LogNet: Energy-efficient neural networks using logrithmic computations, 5900
- DNN Energy Estimation
- 10.1145/3140659.3080254 · DOI · Paper page
- Iandola, SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <1 MB model size
- Benchmarking DNN Processors
Cited by
A survey of the recent architectures of deep convolutional neural networks
Khan, A., Sohail, A., Zahoora, U., & Qureshi, A. S. (2020) · Artificial Intelligence Review · doi:10.1007/s10462-020-09825-6 · Cited by 3,137
“However, the training of deep and high capacity architectures is still a significant overhead on memory usage and computational resources (Lacey et al. 2016; Sze et al. 2017; Justus et al. 2019).”
BackgroundEdge Intelligence: Paving the Last Mile of Artificial Intelligence With Edge Computing
Zhou, Z., Chen, X., Li, E., Zeng, L., Luo, K., & Zhang, J. (2019) · Proceedings of the IEEE · doi:10.1109/jproc.2019.2918951 · Cited by 2,455
“Interested readers are encouraged to refer to the recent monograph [114] for more comprehensive discussions about recent advancements on hardware acceleration for DNN processing.”
BackgroundMemory devices and applications for in-memory computing
Sebastian, A., Le Gallo, M., Khaddam-Aljameh, R., & Eleftheriou, E. (2020) · Nature Nanotechnology · doi:10.1038/s41565-020-0655-z · Cited by 2,289
“NaTuRe NaNoTecHNology This non-volatile accumulative behaviour, in spite of its nonlinear and stochastic nature, can be exploited in several applications, such as training deep neural networks, where the conductance values need to be incrementally modified.”
BackgroundDeep Learning in Mobile and Wireless Networking: A Survey
Zhang, C., Patras, P., & Haddadi, H. (2019) · IEEE Communications Surveys & Tutorials · doi:10.1109/comst.2019.2904897 · Cited by 1,768
Photonics for artificial intelligence and neuromorphic computing
Shastri, B. J., Tait, A. N., Ferreira de Lima, T., Pernice, W. H. P., Bhaskaran, H., Wright, C. D., & Prucnal, P. R. (2021) · Nature Photonics · doi:10.1038/s41566-020-00754-y · Cited by 1,756
Metadata
- Authors
- Sze, V., Chen, Y.-H., Yang, T.-J., & Emer, J. S.
- Affiliations
- Massachusetts Institute of Technology (US)
- Nvidia (United States) (US)
- Venue
- Proceedings of the IEEE
- Published
- 2017-12
- Volume
- 105
- Issue
- 12
- Pages
- 2295-2329
- ISSN
- 0018-9219 (ISSN-L)
- 1558-2256
- OpenAlex
- W2604319603
- Cited by
- 4,215
- References
- 157
- Citations by year
- 2026: 504 · 2025: 523 · 2024: 474 · 2023: 466 · 2022: 508 · 2021: 568 · 2020: 551 · 2019: 392 · 2018: 199 · 2017: 28
- FWCI
- 132.75
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Proceedings of the IEEE (publishedVersion)
- Funders
- DARPA YFA
- MIT CICS
- gift from Nvidia
- gift from Intel
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Computer Science › Hardware and Architecture › Parallel Computing and Optimization Techniques
- Physical Sciences › Engineering › Electrical and Electronic Engineering › Advanced Memory and Neural Computing
- Keywords
- deep neural network, energy efficiency, throughput, hardware accelerators, computation cost reduction, benchmarking metrics
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.