Demystifying Parallel and Distributed Deep Learning: An In-depth Concurrency Analysis

Deep Residual Learning for Image Recognition

He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474

ImageNet classification with deep convolutional neural networks

Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155

Long Short-Term Memory

Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318

Adam: A Method for Stochastic Optimization

Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946

Deep learning

LeCun, Y., Bengio, Y., & Hinton, G. (2015) · Nature · doi:10.1038/nature14539 · Cited by 84,126

ImageNet: A large-scale hierarchical image database

Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333

Gradient-based learning applied to document recognition

Lecun, Y., Bottou, L., Bengio, Y., & Haffner, P. (1998) · Proceedings of the IEEE · doi:10.1109/5.726791 · Cited by 58,805

Densely Connected Convolutional Networks

Huang, G., Liu, Z., Van Der Maaten, L., & Weinberger, K. Q. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.243 · Cited by 48,566

Going deeper with convolutions

Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., & Rabinovich, A. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298594 · Cited by 48,562

Fully convolutional networks for semantic segmentation

Long, J., Shelhamer, E., & Darrell, T. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298965 · Cited by 38,834

Human-level control through deep reinforcement learning

Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., Graves, A., Riedmiller, M., Fidjeland, A. K., Ostrovski, G., et al. (2015) · Nature · doi:10.1038/nature14236 · Cited by 30,357

Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation

Cho, K., van Merrienboer, B., Gulcehre, C., Bahdanau, D., Bougares, F., Schwenk, H., & Bengio, Y. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1179 · Cited by 25,545

Learning Multiple Layers of Features from Tiny Images

Krizhevsky, A. (2024) · doi:10.57702/zp44cu3g · Cited by 23,395

A survey on transfer learning

Pan, S. J., & Yang, Q. (2010) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2009.191 · Cited by 23,387

Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification

He, K., Zhang, X., Ren, S., & Sun, J. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.123 · Cited by 19,741

MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications

Howard, A. G., Zhu, M., Chen, B., Kalenichenko, D., Wang, W., Weyand, T., Andreetto, M., & Adam, H. (2017) · arXiv · doi:10.48550/arxiv.1704.04861 · Cited by 19,015

MapReduce: simplified data processing on large clusters

Dean, J., & Ghemawat, S. (2008) · Communications of the ACM · doi:10.1145/1327452.1327492 · Cited by 18,476

Distributed Optimization and Statistical Learning via the Alternating Direction Method of Multipliers

Boyd, S., Parikh, N., Chu, E., Peleato, B., & Eckstein, J. (2011) · Foundations and Trends® in Machine Learning · doi:10.1561/2200000016 · Cited by 17,075

A fast learning algorithm for deep belief nets

Hinton, G. E., Osindero, S., & Teh, Y.-W. (2006) · Neural Computation · doi:10.1162/neco.2006.18.7.1527 · Cited by 16,336

Proceedings of IEEE Conference on Computer Vision and Pattern Recognition

(1993) · Proceedings of IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.1993.341010 · Cited by 14,265

Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift

Ioffe, S. (2024) · arXiv (Cornell University) · doi:10.57702/o9raffed · Cited by 14,214

Distilling the Knowledge in a Neural Network

Hinton, G., Vinyals, O., & Dean, J. (2015) · arXiv · doi:10.48550/arxiv.1503.02531 · Cited by 14,204

Untitled work

(2010) · Cited by 14,134

Numerical optimization

(2021) · A First Course in Statistical Programming with R · doi:10.1017/9781108993456.011 · Cited by 13,724

Distributed Optimization and Statistical Learning via the Alternating Direction Method of Multipliers

Boyd, S. (2010) · now publishers, Inc. eBooks · doi:10.1561/9781601984616 · Cited by 12,677

Backpropagation Applied to Handwritten Zip Code Recognition

LeCun, Y., Boser, B., Denker, J. S., Henderson, D., Howard, R. E., Hubbard, W., & Jackel, L. D. (1989) · Neural Computation · doi:10.1162/neco.1989.1.4.541 · Cited by 11,891

Proceedings of the 24th international conference on Machine learning

(2007) · doi:10.1145/1273496 · Cited by 11,630

Finding structure in time

Elman, J. L. (1990) · Cognitive Science · doi:10.1207/s15516709cog1402_1 · Cited by 11,212

Caffe: Convolutional Architecture for Fast Feature Embedding

Jia, Y., Shelhamer, E., Donahue, J., Karayev, S., Long, J., Girshick, R., Guadarrama, S., & Darrell, T. (2014) · Proceedings of the 22nd ACM international conference on Multimedia · doi:10.1145/2647868.2654889 · Cited by 11,077

A Stochastic Approximation Method

Robbins, H., & Monro, S. (1951) · The Annals of Mathematical Statistics · doi:10.1214/aoms/1177729586 · Cited by 9,643

TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems

Abadi, M., Agarwal, A., Barham, P., Brevdo, E., Chen, Z., Citro, C., Corrado, G. S., Davis, A., Dean, J., Devin, M., et al. (2016) · arXiv · doi:10.48550/arxiv.1603.04467 · Cited by 9,408

Mastering the game of Go without human knowledge

Silver, D., Schrittwieser, J., Simonyan, K., Antonoglou, I., Huang, A., Guez, A., Hubert, T., Baker, L., Lai, M., Bolton, A., et al. (2017) · Nature · doi:10.1038/nature24270 · Cited by 9,084

Advances in Neural Information Processing Systems 14: Proceedings of the 2001 Conference

(2002) · The MIT Press eBooks · doi:10.7551/mitpress/1120.001.0001 · Cited by 8,890

Learning long-term dependencies with gradient descent is difficult

Bengio, Y., Simard, P., & Frasconi, P. (1994) · IEEE Transactions on Neural Networks · doi:10.1109/72.279181 · Cited by 8,542

Ensemble methods in machine learning

Dietterich, T. G. (2000) · Lecture Notes in Computer Science · doi:10.1007/3-540-45014-9_1 · Cited by 8,408

Untitled work

(2010) · Cited by 8,261

Show, Attend and Tell: Neural Image Caption Generation with Visual Attention

Xu, K., Ba, J., Kiros, R., Cho, K., Courville, A., Salakhutdinov, R., Zemel, R., & Bengio, Y. (2015) · arXiv · doi:10.48550/arxiv.1502.03044 · Cited by 8,248

Multi-scale context aggregation by dilated convolutions

Yu, F., & Koltun, V. (2015) · arXiv · doi:10.48550/arxiv.1511.07122 · Cited by 7,738

Asynchronous Methods for Deep Reinforcement Learning

Mnih, V., Badia, A. P., Mirza, M., Graves, A., Lillicrap, T. P., Harley, T., Silver, D., & Kavukcuoglu, K. (2016) · arXiv · doi:10.48550/arxiv.1602.01783 · Cited by 7,132

Practical Bayesian Optimization of Machine Learning Algorithms

Snoek, J., Larochelle, H., & Adams, R. P. (2012) · arXiv · doi:10.48550/arxiv.1206.2944 · Cited by 6,691

Proceedings of the 25th international conference on Machine learning - ICML '08

(2008) · doi:10.1145/1390156 · Cited by 6,680

A Method for the Construction of Minimum-Redundancy Codes

Huffman, D. (1952) · Proceedings of the IRE · doi:10.1109/jrproc.1952.273898 · Cited by 6,334

Learning Transferable Architectures for Scalable Image Recognition

Zoph, B., Vasudevan, V., Shlens, J., & Le, Q. V. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00907 · Cited by 6,249

SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <0.5MB model size

Iandola, F. N., Han, S., Moskewicz, M. W., Ashraf, K., Dally, W. J., & Keutzer, K. (2016) · arXiv · doi:10.48550/arxiv.1602.07360 · Cited by 6,201

The Byzantine generals problem

Lamport, L., Shostak, R., & Pease, M. (1982) · ACM Transactions on Programming Languages and Systems · doi:10.1145/357172.357176 · Cited by 6,023

SGDR: Stochastic Gradient Descent with Warm Restarts

Loshchilov, I., & Hutter, F. (2016) · arXiv · doi:10.48550/arxiv.1608.03983 · Cited by 5,309

Backpropagation through time: what it does and how to do it

Werbos, P. J. (1990) · Proceedings of the IEEE · doi:10.1109/5.58337 · Cited by 4,879

Greedy layer-wise training of deep networks

Bengio, Y., Lamblin, P., Popovici, D., & Larochelle, H. (2007) · Advances in Neural Information Processing Systems 19 · doi:10.7551/mitpress/7503.003.0024 · Cited by 4,841

Progressive Growing of GANs for Improved Quality, Stability, and Variation

Karras, T., Aila, T., Laine, S., & Lehtinen, J. (2017) · arXiv · doi:10.48550/arxiv.1710.10196 · Cited by 4,727

In-Datacenter Performance Analysis of a Tensor Processing Unit

Jouppi, N. P., Young, C., Patil, N., Patterson, D., Agrawal, G., Bajwa, R., Bates, S., Bhatia, S., Boden, N., Borchers, A., et al. (2017) · Proceedings of the 44th Annual International Symposium on Computer Architecture · doi:10.1145/3079856.3080246 · Cited by 4,537

XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networks

Rastegari, M., Ordonez, V., Redmon, J., & Farhadi, A. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46493-0_32 · Cited by 4,408

Efficient Processing of Deep Neural Networks: A Tutorial and Survey

Sze, V., Chen, Y.-H., Yang, T.-J., & Emer, J. S. (2017) · Proceedings of the IEEE · doi:10.1109/jproc.2017.2761740 · Cited by 4,215 · cited 2×

Neural Architecture Search with Reinforcement Learning

Zoph, B., & Le, Q. V. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1611.01578 · Cited by 3,974

Simple Statistical Gradient-Following Algorithms for Connectionist Reinforcement Learning

Williams, R. J. (1992) · Machine Learning · doi:10.1007/bf00992696 · Cited by 3,416

DARTS: Differentiable architecture search

Liu, H., Simonyan, K., & Yang, Y. (2018) · arXiv · doi:10.48550/arxiv.1806.09055 · Cited by 3,248

Large Scale Distributed Deep Networks

Dean, J. B., Corrado, G. S., Monga, R., Chen, K., Devin, M., Mao, M., Ranzato, M., Senior, A., Tucker, P. A., Yang, K., et al. (2012) · Cited by 3,199

End to End Learning for Self-Driving Cars

Bojarski, M., Del Testa, D., Dworakowski, D., Firner, B., Flepp, B., Goyal, P., Jackel, L. D., Monfort, M., Muller, U., Zhang, J., et al. (2016) · arXiv · doi:10.48550/arxiv.1604.07316 · Cited by 3,198

Optimization Methods for Large-Scale Machine Learning

Bottou, L., Curtis, F. E., & Nocedal, J. (2018) · SIAM Review · doi:10.1137/16m1080173 · Cited by 3,128

Understanding Machine Learning: From Theory to Algorithms

Shalev-Shwartz, S., & Ben-David, S. (2014) · doi:10.1017/cbo9781107298019 · Cited by 3,115

Popular Ensemble Methods: An Empirical Study

Opitz, D., & Maclin, R. (1999) · Journal of Artificial Intelligence Research · doi:10.1613/jair.614 · Cited by 2,953

Untitled work

(2015) · Cited by 2,857

A method for solving the convex programming problem with convergence rate O(1/k^2)

Nesterov, Y. E. (1983) · Доклады Академии наук · Cited by 2,662

Proceedings of the 23rd international conference on Machine learning

Cohen, W. W., & Moore, A. (2006) · Cited by 2,647

Gaussian elimination is not optimal

Strassen, V. (1969) · Numerische Mathematik · doi:10.1007/bf02165411 · Cited by 2,509

Proceedings of the 16th international joint conference on Artificial Intelligence - IJCAI '99

Colton, S., Bundy, A., & Walsh, T. (1999) · Cited by 2,500

A Comparative Analysis of Selection Schemes Used in Genetic Algorithms

Goldberg, D. E., & Deb, K. (1991) · Foundations of Genetic Algorithms · doi:10.1016/b978-0-08-050684-5.50008-2 · Cited by 2,407

Privacy-preserving deep learning

Shokri, R., & Shmatikov, V. (2015) · Proceedings of the 22nd ACM SIGSAC Conference on Computer and Communications Security · doi:10.1145/2810103.2813687 · Cited by 2,356

On the momentum term in gradient descent learning algorithms

Qian, N. (1999) · Neural Networks · doi:10.1016/s0893-6080(98)00116-6 · Cited by 2,312

Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour

Goyal, P., Dollár, P., Girshick, R., Noordhuis, P., Wesolowski, L., Kyrola, A., Tulloch, A., Jia, Y., & He, K. (2017) · arXiv · doi:10.48550/arxiv.1706.02677 · Cited by 2,294

Listen, attend and spell: A neural network for large vocabulary conversational speech recognition

Chan, W., Jaitly, N., Le, Q., & Vinyals, O. (2016) · 2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2016.7472621 · Cited by 2,226

BinaryConnect: Training Deep Neural Networks with binary weights during propagations

Courbariaux, M., Bengio, Y., & David, J.-P. (2015) · arXiv · doi:10.48550/arxiv.1511.00363 · Cited by 2,180

Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing, 2007 (ICASSP 2007)

Livescu, K., Bezman, A., Borges, N., Yung, L.-Y., Çetin, Ö., Frankel, J., King, S., Magimai.-Doss, M., Chi, X., & Lavoie, L. (2007) · Edinburgh Research Explorer (University of Edinburgh) · Cited by 2,145

Proceedings of the 19th International Conference on Artificial Intelligence and Statistics 2016

Murray, I., & Graham, M. M. (2016) · Cited by 2,115

Deep Speech 2: End-to-End Speech Recognition in English and Mandarin

Amodei, D., Anubhai, R., Battenberg, E., Case, C. T., Casper, J., Catanzaro, B., Chen, J., Chrzanowski, M., Coates, A., Diamos, G., et al. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1512.02595 · Cited by 2,110

Binarized Neural Networks: Training Deep Neural Networks with Weights and Activations Constrained to +1 or -1

Courbariaux, M., Hubara, I., Soudry, D., El-Yaniv, R., & Bengio, Y. (2016) · arXiv · doi:10.48550/arxiv.1602.02830 · Cited by 2,106

Proceedings of The 32nd International Conference on Machine Learning

Germain, M., Gregor, K., Murray, I., & Larochelle, H. (2015) · Cited by 2,020

Distributed asynchronous deterministic and stochastic gradient optimization algorithms

Tsitsiklis, J., Bertsekas, D., & Athans, M. (1986) · IEEE Transactions on Automatic Control · doi:10.1109/tac.1986.1104412 · Cited by 1,967

Robust Stochastic Approximation Approach to Stochastic Programming

Nemirovski, A., Juditsky, A., Lan, G., & Shapiro, A. (2009) · SIAM Journal on Optimization · doi:10.1137/070704277 · Cited by 1,945

Proceedings IEEE International Conference on Acoustics Speech and Signal Processing

Zheng, F., McLaughlin, S., & Mulgrew, B. (1991) · Cited by 1,905

Accelerating Stochastic Gradient Descent using Predictive Variance Reduction

Johnson, R., & Zhang, T. (2013) · Cited by 1,889

Progressive Neural Architecture Search

Liu, C., Zoph, B., Neumann, M., Shlens, J., Hua, W., Li, L.-J., Fei-Fei, L., Yuille, A., Huang, J., & Murphy, K. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01246-5_2 · Cited by 1,833

Acceleration of Stochastic Approximation by Averaging

Polyak, B. T., & Juditsky, A. B. (1992) · SIAM Journal on Control and Optimization · doi:10.1137/0330046 · Cited by 1,827

DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients

Zhou, S., Wu, Y., Ni, Z., Zhou, X., Wen, H., & Zou, Y. (2016) · arXiv · doi:10.48550/arxiv.1606.06160 · Cited by 1,690

TrueNorth: Design and Tool Flow of a 65 mW 1 Million Neuron Programmable Neurosynaptic Chip

Akopyan, F., Sawada, J., Cassidy, A., Alvarez-Icaza, R., Arthur, J., Merolla, P., Imam, N., Nakamura, Y., Datta, P., Nam, G.-J., et al. (2015) · IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems · doi:10.1109/tcad.2015.2474396 · Cited by 1,687

Mitosis Detection in Breast Cancer Histology Images with Deep Neural Networks

Cireşan, D. C., Giusti, A., Gambardella, L. M., & Schmidhuber, J. (2013) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-40763-5_51 · Cited by 1,536

LogP: towards a realistic model of parallel computation

Culler, D., Karp, R., Patterson, D., Sahay, A., Schauser, K. E., Santos, E., Subramonian, R., & von Eicken, T. (1993) · Proceedings of the fourth ACM SIGPLAN symposium on Principles and practice of parallel programming - PPOPP '93 · doi:10.1145/155332.155333 · Cited by 1,496

Untitled work

(2017) · neural information processing systems · Cited by 1,413

Hybrid computing using a neural network with dynamic external memory

Graves, A., Wayne, G., Reynolds, M., Harley, T., Danihelka, I., Grabska-Barwińska, A., Colmenarejo, S. G., Grefenstette, E., Ramalho, T., Agapiou, J., et al. (2016) · Nature · doi:10.1038/nature20101 · Cited by 1,412

Deep Learning with Limited Numerical Precision

Gupta, S., Agrawal, A., Gopalakrishnan, K., & Narayanan, P. (2015) · arXiv · doi:10.48550/arxiv.1502.02551 · Cited by 1,364

Scheduling multithreaded computations by work stealing

Blumofe, R. D., & Leiserson, C. E. (1999) · Journal of the ACM · doi:10.1145/324133.324234 · Cited by 1,336

Do Deep Nets Really Need to be Deep?

Ba, J. L., & Caruana, R. (2013) · arXiv (Cornell University) · doi:10.48550/arxiv.1312.6184 · Cited by 1,335

Proceedings of the 26th International Conference on Neural Information Processing Systems

Burges, C. J. C., Bottou, L., Welling, M., Zoubin Ghahramani, A., & Weinberger, K. Q. (2013) · Cited by 1,323

Rounding errors in algebraic processes

(1964) · Journal of the Franklin Institute · doi:10.1016/0016-0032(64)90390-4 · Cited by 1,315

DianNao: a small-footprint high-throughput accelerator for ubiquitous machine-learning

Chen, T., Du, Z., Sun, N., Wang, J., Wu, C., Chen, Y., & Temam, O. (2014) · Proceedings of the 19th international conference on Architectural support for programming languages and operating systems · doi:10.1145/2541940.2541967 · Cited by 1,297

In-Datacenter Performance Analysis of a Tensor Processing Unit

Jouppi, N. P., Young, C., Patil, N., Patterson, D., Agrawal, G., Bajwa, R., Bates, S., Bhatia, S., Boden, N., Borchers, A., et al. (2017) · ACM SIGARCH Computer Architecture News · doi:10.1145/3140659.3080246 · Cited by 1,240

Map-Reduce for Machine Learning on Multicore

Chu, C.-T., Kim, S. K., Lin, Y.-A., Yu, Y., Bradski, G., Ng, A. Y., & Olukotun, K. (2007) · Advances in Neural Information Processing Systems 19 · doi:10.7551/mitpress/7503.003.0040 · Cited by 1,229

Proceedings of the 3rd International Conference on Neural Information Processing Systems

Lippmann, R. P., Moody, J. E., & Touretzky, D. S. (1990) · Cited by 1,211

Hogwild: A Lock-Free Approach to Parallelizing Stochastic Gradient Descent

Niu, F., Recht, B., Re, C., & Wright, S. J. (2011) · arXiv · doi:10.48550/arxiv.1106.5730 · Cited by 1,211

Torch7: A Matlab-like Environment for Machine Learning

Collobert, R., Kavukcuoglu, K., & Farabet, C. (2011) · Infoscience (Ecole Polytechnique Fédérale de Lausanne) · Cited by 1,186

GPipe: Efficient Training of Giant Neural Networks using Pipeline Parallelism

Huang, Y., Cheng, Y., Bapna, A., Firat, O., Chen, M. X., Chen, D., Lee, H., Ngiam, J., Le, Q. V., Wu, Y., et al. (2018) · arXiv · doi:10.48550/arxiv.1811.06965 · Cited by 1,180

A guide to convolution arithmetic for deep learning

Dumoulin, V., & Visin, F. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1603.07285 · Cited by 1,172

Deep Gradient Compression: Reducing the Communication Bandwidth for Distributed Training

Lin, Y., Han, S., Mao, H., Wang, Y., & Dally, W. J. (2017) · arXiv · doi:10.48550/arxiv.1712.01887 · Cited by 1,134

Parallelized Stochastic Gradient Descent

Zinkevich, M., Weimer, M., Li, L., & Smola, A. J. (2010) · Cited by 1,124

Scaling distributed machine learning with the parameter server

Li, M., Andersen, D. G., Park, J. W., Smola, A. J., Ahmed, A. A., Josifovski, V., Long, J. E., Shekita, E. J., & Su, B.-Y. (2014) · Operating Systems Design and Implementation · doi:10.5555/2685048.2685095 · Cited by 1,021

Scaling Distributed Machine Learning with the Parameter Server

Li, M. (2014) · Proceedings of the 2014 International Conference on Big Data Science and Computing · doi:10.1145/2640087.2644155 · Cited by 979

Untitled work

(2009) · neural information processing systems · Cited by 962

cuDNN: Efficient Primitives for Deep Learning

Chetlur, S., Woolley, C., Vandermersch, P., Cohen, J., Tran, J. J., Catanzaro, B., & Shelhamer, E. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1410.0759 · Cited by 943

Untitled work

(2014) · arXiv (Cornell University) · Cited by 942

One weird trick for parallelizing convolutional neural networks

Krizhevsky, A. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1404.5997 · Cited by 919

Fast Algorithms for Convolutional Neural Networks

Lavin, A., & Gray, S. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.435 · Cited by 915

Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networks

Salimans, T., & Kingma, D. P. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1602.07868 · Cited by 883

On optimization methods for deep learning

Ngiam, J., Coates, A., Lahiri, A., Prochnow, B., Le, Q. V., & Ng, A. Y. (2011) · International Conference on Machine Learning · Cited by 862

QSGD: Communication-Efficient SGD via Gradient Quantization and Encoding

Alistarh, D., Grubic, D., Li, J., Tomioka, R., & Vojnović, M. (2016) · London School of Economics and Political Science Research Online (London School of Economics and Political Science) · doi:10.48550/arxiv.1610.02132 · Cited by 859

Proceedings of the 22nd international conference on Machine learning - ICML '05

(2005) · doi:10.1145/1102351 · Cited by 840

The Parallel Evaluation of General Arithmetic Expressions

Brent, R. P. (1974) · Journal of the ACM · doi:10.1145/321812.321815 · Cited by 814

Deep Reinforcement Learning: An Overview

Li, Y. (2017) · arXiv · doi:10.48550/arxiv.1701.07274 · Cited by 762

Evolving Deep Neural Networks

Miikkulainen, R., Liang, J., Meyerson, E., Rawal, A., Fink, D., Francon, O., Raju, B., Shahrzad, H., Navruzyan, A., Duffy, N., et al. (2019) · Artificial Intelligence in the Age of Neural Networks and Brain Computing · doi:10.1016/b978-0-12-815480-9.00015-3 · Cited by 760

Large-scale deep unsupervised learning using graphics processors

Raina, R., Madhavan, A., & Ng, A. Y. (2009) · Proceedings of the 26th Annual International Conference on Machine Learning · doi:10.1145/1553374.1553486 · Cited by 742 · cited 1×

Ternary Weight Networks

Li, F., Liu, B., Wang, X., Zhang, B., & Yan, J. (2016) · arXiv · doi:10.48550/arxiv.1605.04711 · Cited by 732

Deep learning via Hessian-free optimization

Martens, J. (2010) · Cited by 722

Untitled work

(2011) · Cited by 713

Untitled work

(2013) · Cited by 651

Building high-level features using large scale unsupervised learning

Ranzato, M., Monga, R., Devin, M., Chen, K., Corrado, G. S., Dean, J., Le, Q. V., & Ng, A. Y. (2012) · International Conference on Machine Learning · Cited by 646

TernGrad: Ternary Gradients to Reduce Communication in Distributed Deep Learning

Wen, W., Xu, C., Yan, F., Wu, C., Wang, Y., Chen, Y., & Li, H. H. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1705.07878 · Cited by 622

Sparse Communication for Distributed Gradient Descent

Aji, A. F., & Heafield, K. (2017) · Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d17-1045 · Cited by 585

Training Deep Nets with Sublinear Memory Cost

Chen, T., Xu, B., Zhang, C., & Guestrin, C. (2016) · arXiv · doi:10.48550/arxiv.1604.06174 · Cited by 561

Don't Decay the Learning Rate, Increase the Batch Size

Smith, S. L., Kindermans, P.-J., Ying, C., & Le, Q. V. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1711.00489 · Cited by 559

Gossip algorithms: design, analysis and applications

Boyd, S., Ghosh, A., Prabbakar, B., & Shah, D. (2005) · Proceedings IEEE 24th Annual Joint Conference of the IEEE Computer and Communications Societies. · doi:10.1109/infcom.2005.1498447 · Cited by 558

Project Adam: building an efficient and scalable deep learning training system

Chilimbi, T. M., Suzue, Y., Apacible, J., & Kalyanaraman, K. (2014) · Operating Systems Design and Implementation · doi:10.5555/2685048.2685094 · Cited by 550

Scaling Neural Machine Translation

Ott, M., Edunov, S., Grangier, D., & Auli, M. (2018) · Proceedings of the Third Conference on Machine Translation: Research Papers · doi:10.18653/v1/w18-6301 · Cited by 544

Xception: Deep Learning with Depthwise Separable Convolutions

Chollet, F. (2016) · arXiv · doi:10.48550/arxiv.1610.02357 · Cited by 538

Arithmetic Complexity of Computations

Winograd, S. (1980) · Society for Industrial and Applied Mathematics eBooks · doi:10.1137/1.9781611970364 · Cited by 535

Deep Neuroevolution: Genetic Algorithms Are a Competitive Alternative for Training Deep Neural Networks for Reinforcement Learning

Such, F. P., Madhavan, V., Conti, E., Lehman, J., Stanley, K. O., & Clune, J. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1712.06567 · Cited by 525

Scalable distributed DNN training using commodity GPU cloud computing

Strom, N. (2015) · Interspeech 2015 · doi:10.21437/interspeech.2015-354 · Cited by 521

Efficient Neural Architecture Search via Parameter Sharing

Pham, H., Guan, M. Y., Zoph, B., Le, Q. V., & Dean, J. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1802.03268 · Cited by 517

Deep Learning of Representations: Looking Forward

Bengio, Y. (2013) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-39593-2_1 · Cited by 513

Optimal Distributed Online Prediction using Mini-Batches

Dekel, O., Gilad-Bachrach, R., Shamir, O., & Xiao, L. (2010) · arXiv (Cornell University) · doi:10.48550/arxiv.1012.1367 · Cited by 512

More Effective Distributed ML via a Stale Synchronous Parallel Parameter Server

Ho, Q., Cipar, J., Cui, H., Kim, J. K., Lee, S., Gibbons, P. B., Gibson, G. A., Ganger, G. R., & Xing, E. P. (2013) · PubMed · doi:10.1184/r1/6475898 · Cited by 506

Untitled work

(2006) · Cited by 490

BinaryNet: Training Deep Neural Networks with Weights and Activations Constrained to +1 or -1Preprint

Courbariaux, M., & Bengio, Y. (2016) · arXiv (Cornell University) · Cited by 484

Can FPGAs Beat GPUs in Accelerating Next-Generation Deep Neural Networks?

Nurvitadhi, E., Venkatesh, G., Sim, J., Marr, D., Huang, R., Ong Gee Hock, J., Liew, Y. T., Srivatsan, K., Moss, D., Subhaschandra, S., et al. (2017) · Proceedings of the 2017 ACM/SIGDA International Symposium on Field-Programmable Gate Arrays · doi:10.1145/3020078.3021740 · Cited by 467

Large Batch Training of Convolutional Networks

You, Y., Gitman, I., & Ginsburg, B. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1708.03888 · Cited by 463

A Stochastic Quasi-Newton Method for Large-Scale Optimization

Byrd, R. H., Hansen, S. L., Nocedal, J., & Singer, Y. (2016) · SIAM Journal on Optimization · doi:10.1137/140954362 · Cited by 459

1-bit stochastic gradient descent and its application to data-parallel distributed training of speech DNNs

Seide, F., Fu, H., Droppo, J., Li, G., & Yu, D. (2014) · Interspeech 2014 · doi:10.21437/interspeech.2014-274 · Cited by 443

Neural Architecture Optimization

Luo, R., Tian, F., Qin, T., Chen, E., & Liu, T. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1808.07233 · Cited by 427

Bandwidth optimal all-reduce algorithms for clusters of workstations

Patarasuk, P., & Yuan, X. (2009) · Journal of Parallel and Distributed Computing · doi:10.1016/j.jpdc.2008.09.002 · Cited by 409

Dual Path Networks

Chen, Y., Li, J., Xiao, H., Jin, X., Yan, S., & Feng, J. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1707.01629 · Cited by 397

Can Decentralized Algorithms Outperform Centralized Algorithms? A Case Study for Decentralized Parallel Stochastic Gradient Descent

Lian, X., Zhang, C., Zhang, H., Hsieh, C., Zhang, W., & Liu, J. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1705.09056 · Cited by 395

Petuum: A New Platform for Distributed Machine Learning on Big Data

Xing, E. P., Ho, Q., Dai, W., Kim, J. K., Wei, J., Lee, S., Zheng, X., Xie, P., Kumar, A., & Yu, Y. (2015) · IEEE Transactions on Big Data · doi:10.1109/tbdata.2015.2472014 · Cited by 388

Genetic CNN

Xie, L., & Yuille, A. (2017) · IEEE/CVF International Conference on Computer Vision (ICCV) · Cited by 377

Minimizing Computation in Convolutional Neural Networks

Cong, J., & Xiao, B. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-11179-7_36 · Cited by 372

SMASH: One-Shot Model Architecture Search through HyperNetworks

Brock, A., Lim, T., Ritchie, J. M., & Weston, N. J. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1708.05344 · Cited by 351

Fast Training of Convolutional Networks through FFTs

Mathieu, M., Henaff, M., & LeCun, Y. (2014) · International Conference on Learning Representations · Cited by 347

Adaptive Subgradient Methods for Online Learning and Stochastic Optimization

(2011) · Journal of Machine Learning Research · doi:10.5555/1953048.2021068 · Cited by 346

Regularized Evolution for Image Classifier Architecture Search

Real, E., Aggarwal, A., Huang, Y., & Le, Q. V. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.33014780 · Cited by 328

Asynchronous Parallel Stochastic Gradient for Nonconvex Optimization

Lian, X., Huang, Y., Li, Y., & Liu, J. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1506.08272 · Cited by 324

Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis

(2019) · doi:10.1145/3295500 · Cited by 324

Hybrid deterministic-stochastic methods for data fitting

Friedlander, M. P., & Schmidt, M. (2011) · Cited by 314

Meta Learning Shared HierarchiesPreprint

Frans, K., Ho, J., Chen, X., Abbeel, P., & Schulman, J. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1710.09767 · Cited by 297

Tiled convolutional neural networks

Ngiam, J., Chen, Z., Chia, D., Koh, P. W., Le, Q. V., & Ng, A. Y. (2010) · neural information processing systems · Cited by 294

GeePS: scalable deep learning on distributed GPUs with a GPU-specialized parameter server

Cui, H., Zhang, H., Ganger, G. R., Gibbons, P. B., & Xing, E. P. (2016) · Proceedings of the Eleventh European Conference on Computer Systems · doi:10.1145/2901318.2901323 · Cited by 286

Highly Scalable Deep Learning Training System with Mixed-Precision: Training ImageNet in Four Minutes

Jia, X., Song, S., He, W., Wang, Y., Rong, H., Zhou, F., Xie, L., Guo, Z., Yang, Y., Yu, L.-W., et al. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1807.11205 · Cited by 281

FireCaffe: Near-Linear Acceleration of Deep Neural Network Training on Compute Clusters

Iandola, F. N., Moskewicz, M. W., Ashraf, K., & Keutzer, K. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.284 · Cited by 280

Poseidon: An Efficient Communication Architecture for Distributed Deep Learning on GPU ClustersPreprint

Zhang, H., Zheng, Z., Xu, S., Dai, W., Ho, Q., Liang, X., Hu, Z., Wei, J., Xie, P., & Xing, E. P. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1706.03292 · Cited by 264

Neural Architecture Search with Bayesian Optimisation and Optimal Transport

Kandasamy, K., Neiswanger, W., Schneider, J., Póczos, B., & Xing, E. P. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1802.07191 · Cited by 259

Shallow vs. Deep Sum-Product Networks

Delalleau, O., & Bengio, Y. (2011) · Cited by 254

One Model To Learn Them All

Kaiser, L., Gomez, A. N., Shazeer, N., Vaswani, A., Parmar, N., Jones, L., & Uszkoreit, J. (2017) · arXiv · doi:10.48550/arxiv.1706.05137 · Cited by 253

Efficient computation of the DFT with only a subset of input or output points

Sorensen, H. V., & Burrus, C. S. (1993) · IEEE Transactions on Signal Processing · doi:10.1109/78.205723 · Cited by 245

Tensor Comprehensions: Framework-Agnostic High-Performance Machine Learning Abstractions

Vasilache, N., Zinenko, O., Theodoridis, T., Goyal, P., DeVito, Z., Moses, W. S., Verdoolaege, S., Adams, A., & Cohen, A. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1802.04730 · Cited by 238

Measuring the Effects of Data Parallelism on Neural Network Training

Shallue, C. J., Lee, J., Antognini, J., Sohl-Dickstein, J., Frostig, R., & Dahl, G. E. (2018) · arXiv · doi:10.48550/arxiv.1811.03600 · Cited by 229

Deep Learning on FPGAs: Past, Present, and FuturePreprint

Lacey, G., Taylor, G. W., & Areibi, S. M. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1602.04283 · Cited by 221

Scalable stacking and learning for building deep architectures

Deng, L., Yu, D., & Platt, J. (2012) · 2012 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2012.6288333 · Cited by 210

Heterogeneity-aware Distributed Parameter Servers

Jiang, J., Cui, B., Zhang, C., & Yu, L. (2017) · Proceedings of the 2017 ACM International Conference on Management of Data · doi:10.1145/3035918.3035933 · Cited by 202

Proceedings of the 29th Annual IEEE/ACM International Symposium on Microarchitecture. MICRO 29

(1996) · doi:10.1109/micro.1996 · Cited by 198

DeepArchitect: Automatically Designing and Training Deep ArchitecturesPreprint

Negrinho, R. M. P., & Gordon, G. J. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1704.08792 · Cited by 197

Implementation and performance analysis of non-blocking collective operations for MPI

Hoefler, T., Lumsdaine, A., & Rehm, W. (2007) · Proceedings of the 2007 ACM/IEEE conference on Supercomputing · doi:10.1145/1362622.1362692 · Cited by 193

Proceedings International Conference on Artificial Neural Networks ICANN'95

Bishop, C. (1995) · Cited by 189

A Linearly-Convergent Stochastic L-BFGS AlgorithmPreprint

Moritz, P., Nishihara, R., & Jordan, M. I. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1508.02087 · Cited by 188

Optimization of Collective Reduction Operations

Rabenseifner, R. (2004) · Lecture Notes in Computer Science · doi:10.1007/978-3-540-24685-5_1 · Cited by 185

Why M Heads are Better than One: Training a Diverse Ensemble of Deep Networks

Lee, S., Purushwalkam, S., Cogswell, M., Crandall, D., & Batra, D. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1511.06314 · Cited by 184

Learning Curve Prediction with Bayesian Neural Networks

Klein, A., Falkner, S., Springenberg, J. T., & Hutter, F. (2017) · International Conference on Learning Representations · Cited by 176

Progressive Neural Architecture Search

Liu, C., Zoph, B., Neumann, M., Shlens, J., Hua, W., Li, L.-J., Fei-Fei, L., Yuille, A., Huang, J., & Murphy, K. (2017) · arXiv · doi:10.48550/arxiv.1712.00559 · Cited by 171

Staleness-aware Async-SGD for Distributed Deep Learning

Zhang, W., Gupta, S., Lian, X., & Liu, J. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1511.05950 · Cited by 171

Simple And Efficient Architecture Search for Convolutional Neural Networks

Elsken, T., Metzen, J. H., & Hutter, F. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1711.04528 · Cited by 170

AdaComp : Adaptive Residual Gradient Compression for Data-Parallel Distributed Training

Chen, C.-Y., Choi, J., Brand, D., Agrawal, A., Zhang, W., & Gopalakrishnan, K. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.11728 · Cited by 166

8-Bit Approximations for Parallelism in Deep LearningPreprint

Dettmers, T. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1511.04561 · Cited by 165

Flexpoint: An Adaptive Numerical Format for Efficient Training of Deep Neural Networks

Köster, U., Webb, T. J., Wang, X., Nassar, M., Bansal, A. K., Constable, W., Elibol, O. H., Gray, S., Hall, S., Hornof, L., et al. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1711.02213 · Cited by 164

Parallel Multi Channel convolution using General Matrix Multiplication

Vasudevan, A., Anderson, A., & Gregg, D. (2017) · IEEE International Conference on Application-specific Systems, Architectures and Processors (ASAP) · doi:10.1109/asap.2017.7995254 · Cited by 159

Paleo: A Performance Model for Deep Neural Networks

Qi, H., Sparks, E., & Talwalkar, A. (2017) · International Conference on Learning Representations · Cited by 154

Model Accuracy and Runtime Tradeoff in Distributed Deep Learning: A Systematic Study

Gupta, S., Zhang, W., & Wang, F. (2016) · Proceedings · doi:10.1109/icdm.2016.0028 · Cited by 148

Scalable training of deep learning machines by incremental block training with intra-block parallel optimization and blockwise model-update filtering

Chen, K., & Huo, Q. (2016) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2016.7472805 · Cited by 137

Memory-Efficient Backpropagation Through TimePreprint

Gruslys, A., Munos, R., Danihelka, I., Lanctot, M., & Graves, A. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1606.03401 · Cited by 136

S-Caffe: Co-designing MPI Runtimes and Caffe for Scalable Deep Learning on Modern GPU Clusters

Awan, A. A., Hamidouche, K., Hashmi, J. M., & Panda, D. K. D. (2017) · ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming · doi:10.1145/3018743.3018769 · Cited by 133

SparCML: high-performance sparse communication for machine learning

Renggli, C., Ashkboos, S., Aghagolzadeh, M., Alistarh, D., & Hoefler, T. (2019) · International Conference for High Performance Computing, Networking, Storage and Analysis (SC) · doi:10.1145/3295500.3356222 · Cited by 128

Asynchronous stochastic gradient descent for DNN training

Zhang, S., Zhang, C., You, Z., Zheng, R., & Xu, B. (2013) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2013.6638950 · Cited by 124

TVM: End-to-End Optimization Stack for Deep LearningPreprint

Chen, T., Moreau, T., Jiang, Z., Shen, H., Yan, E., Wang, L., Hu, Y., Ceze, L., Guestrin, C., & Krishnamurthy, A. (2018) · arXiv (Cornell University) · Cited by 123

Image Classification at Supercomputer ScalePreprint

Ying, C., Kumar, S., Chen, D., Wang, T., & Cheng, Y. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1811.06992 · Cited by 123

Efficient Sparse-Winograd Convolutional Neural NetworksPreprint

Liu, X., Pool, J., Han, S., & Dally, W. J. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1802.06367 · Cited by 121

Communication quantization for data-parallel training of deep neural networks

Dryden, N., Jacobs, S. A., Moon, T., & Essen, B. V. (2016) · IEEE International Conference on High Performance Computing, Data, and Analytics · doi:10.5555/3018874.3018875 · Cited by 111

Parallel training of Deep Neural Networks with Natural Gradient and Parameter AveragingPreprint

Povey, D., Zhang, X., & Khudanpur, S. P. (2014) · arXiv (Cornell University) · Cited by 109

The future of digital health with federated learning

Rieke, N., Hancox, J., Li, W., Milletarì, F., Roth, H. R., Albarqouni, S., Bakas, S., Galtier, M. N., Landman, B. A., Maier-Hein, K., et al. (2020) · npj Digital Medicine · doi:10.1038/s41746-020-00323-1 · Cited by 4,015

Parallel convolutional processing using an integrated photonic tensor core

Feldmann, J., Youngblood, N., Karpov, M., Gehring, H., Li, X., Stappers, M., Le Gallo, M., Fu, X., Lukashchuk, A., Raja, A. S., et al. (2021) · Nature · doi:10.1038/s41586-020-03070-1 · Cited by 1,900

Pre-trained models: Past, present and future

Han, X., Zhang, Z., Ding, N., Gu, Y., Liu, X., Huo, Y., Qiu, J., Yao, Y., Zhang, A., Zhang, L., et al. (2021) · AI Open · doi:10.1016/j.aiopen.2021.08.002 · Cited by 994

Federated Learning: A Survey on Enabling Technologies, Protocols, and Applications

Aledhari, M., Razzak, R., Parizi, R. M., & Saeed, F. (2020) · IEEE Access · doi:10.1109/access.2020.3013541 · Cited by 733

Machine Unlearning

Bourtoule, L., Chandrasekaran, V., Choquette-Choo, C. A., Jia, H., Travers, A., Zhang, B., Lie, D., & Papernot, N. (2021) · 2021 IEEE Symposium on Security and Privacy (SP) · doi:10.1109/sp40001.2021.00019 · Cited by 810