A survey of techniques for optimizing deep learning on gpus
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The usual DL model requires large computing power such as a GPU-enabled server.
“In addition, the usual DL model requires large computing power such as GPU enabled sever, although researchers are trying hard to optimizing it [41], [42].”
From A survey on deep transfer learning to edge computing for mitigating the COVID-19 pandemic (2020) · cites this paper as [42]
Cites
Proceedings of IEEE Conference on Computer Vision and Pattern Recognition
(1993) · doi:10.1109/cvpr.1993 · Cited by 17,989 · see citing works
PipeDream: generalized pipeline parallelism for DNN training
Narayanan, D., Harlap, A., Phanishayee, A., Seshadri, V., Devanur, N. R., Ganger, G., Gibbons, P. B., & Zaharia, M. (2019) · Symposium on Operating Systems Principles · doi:10.1145/3341301.3359646 · Cited by 1,362 · see citing works
TVM: an automated end-to-end optimizing compiler for deep learning
(2018) · arXiv (Cornell University) · doi:10.5555/3291168.3291211 · Cited by 1,170 · see citing works
A Survey of Techniques for Approximate Computing
Mittal, S. (2016) · ACM Computing Surveys · doi:10.1145/2893356 · Cited by 1,067 · see citing works
Self-citationDemystifying Parallel and Distributed Deep Learning
Ben‐Nun, T., & Hoefler, T. (2019) · ACM Computing Surveys · doi:10.1145/3320060 · Cited by 747 · see citing works
(2013) · Cited by 651 · see citing works
Applied Machine Learning at Facebook: A Datacenter Infrastructure Perspective
Hazelwood, K., Bird, S., Brooks, D., Chintala, S., Diril, U., Dzhulgakov, D., Fawzy, M., Jia, B., Jia, Y., Kalro, A., et al. (2018) · 2018 IEEE International Symposium on High Performance Computer Architecture (HPCA) · doi:10.1109/hpca.2018.00059 · Cited by 590 · see citing works
A Survey of CPU-GPU Heterogeneous Computing Techniques
Mittal, S., & Vetter, J. S. (2015) · ACM Computing Surveys · doi:10.1145/2788396 · Cited by 543 · see citing works
Self-citationA survey of FPGA-based accelerators for convolutional neural networks
Mittal, S. (2018) · Neural Computing and Applications · doi:10.1007/s00521-018-3761-1 · Cited by 433 · see citing works
Self-citationYou, Y., Zhang, Z., Hsieh, C., Demmel, J. W., & Keutzer, K. (2018) · Proceedings of the International Conference on Parallel Processing · doi:10.1145/3225058.3225069 · Cited by 393 · see citing works
GeePS: scalable deep learning on distributed GPUs with a GPU-specialized parameter server
Cui, H.-G., Zhang, H., Ganger, G., Gibbons, P. B., & Xing, E. (2016) · European Conference on Computer Systems · doi:10.1145/2901318.2901323 · Cited by 339 · see citing works
A Survey on optimized implementation of deep learning models on the NVIDIA Jetson platform
Mittal, S. (2019) · Journal of Systems Architecture · doi:10.1016/j.sysarc.2019.01.011 · Cited by 295 · see citing works
Self-citationJia, X., Song, S., He, W., Wang, Y., Rong, H., Zhou, F., Xie, L., Guo, Z., Yang, Y., Yu, L.-W., et al. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1807.11205 · Cited by 288 · see citing works
FireCaffe: Near-Linear Acceleration of Deep Neural Network Training on Compute Clusters
Iandola, F., Moskewicz, M. W., Ashraf, K. M., & Keutzer, K. (2016) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2016.284 · Cited by 285 · see citing works
Extremely Large Minibatch SGD: Training ResNet-50 on ImageNet in 15 MinutesPreprint
Akiba, T., Suzuki, S., & Fukuda, K. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1711.04325 · Cited by 277 · see citing works
Yu, J., Lukefahr, A., Palframan, D. J., Dasika, G. S., Das, R., & Mahlke, S. A. (2017) · Proceedings - International Symposium on Computer Architecture/Proceedings · doi:10.1145/3079856.3080215 · Cited by 269 · see citing works
Dissecting the NVIDIA Volta GPU Architecture via Microbenchmarking
Jia, Z., Maggioni, M., Staiger, B., & Scarpazza, D. P. (2018) · arXiv · doi:10.48550/arxiv.1804.06826 · Cited by 233 · see citing works
A Survey of Methods for Analyzing and Improving GPU Energy Efficiency
Mittal, S., & Vetter, J. S. (2014) · ACM Computing Surveys · doi:10.1145/2636342 · Cited by 213 · see citing works
Self-citation2010 International Conference on High Performance Computing
(2010) · doi:10.1109/hipc17319.2010 · Cited by 204 · see citing works
Hauswald, J., Kang, Y., Laurenzano, M. A., Chen, Q., Li, C., Mudge, T., Dreslinski, R., Mars, J., & Tang, L. (2015) · Proceedings - International Symposium on Computer Architecture/Proceedings · doi:10.1145/2749469.2749472 · Cited by 161 · see citing works
Performance Analysis of GPU-Based Convolutional Neural Networks
Li, X., Zhang, G., Huang, H., Wang, Z., & Zheng, W. (2016) · Proceedings of the International Conference on Parallel Processing · doi:10.1109/icpp.2016.15 · Cited by 142 · see citing works
Awan, A. A., Hamidouche, K., Hashmi, J. M., & Panda, D. K. D. (2017) · ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming · doi:10.1145/3018743.3018769 · Cited by 128 · see citing works
Performance, Design, and Autotuning of Batched GEMM for GPUs
Abdelfattah, A., Haidar, A., Tomov, S. Z., et al. (2016) · Lecture notes in computer science · doi:10.1007/978-3-319-41321-1_2 · Cited by 121 · see citing works
Optimizing Memory Efficiency for Deep Convolutional Neural Networks on GPUs
Li, C., Yang, Y., Feng, M., Chakradhar, S., & Zhou, H. (2016) · International Conference for High Performance Computing, Networking, Storage and Analysis (SC) · doi:10.1109/sc.2016.53 · Cited by 104 · see citing works
2011 International Conference on Parallel Processing
(2011) · doi:10.1109/icpp18554.2011 · Cited by 102 · see citing works
A coordinated tiling and batching framework for efficient GEMM on GPUs
Li, X., Liang, Y., Yan, S., et al. (2019) · doi:10.1145/3293883.3295734 · Cited by 79 · see citing works
Performance Evaluation of cuDNN Convolution Algorithms on NVIDIA Volta GPUs
Jordà, M., Valero‐Lara, P., & Peña, A. J. (2019) · IEEE Access · doi:10.1109/access.2019.2918851 · Cited by 78 · see citing works
Towards Pervasive and User Satisfactory CNN across GPU Microarchitectures
Song, M., Hu, Y., Chen, H., & Li, T. (2017) · IEEE International Symposium on High-Performance Computer Architecture (HPCA) · doi:10.1109/hpca.2017.52 · Cited by 77 · see citing works
An In-depth Performance Characterization of CPU- and GPU-based DNN Training on Modern Architectures
Awan, A. A., Subramoni, H., & Panda, D. K. D. (2017) · doi:10.1145/3146347.3146356 · Cited by 77 · see citing works
Scaling deep learning on GPU and knights landing clusters
You, Y., Buluç, A., & Demmel, J. W. (2017) · International Conference for High Performance Computing, Networking, Storage and Analysis (SC) · doi:10.1145/3126908.3126912 · Cited by 73 · see citing works
Proceedings of the Fourteenth EuroSys Conference 2019
(2019) · doi:10.1145/3302424 · Cited by 67 · see citing works
Hill, P., Jain, A., Hill, M., Zamirai, B., Hsu, C.-H., Laurenzano, M. A., Mahlke, S. A., Tang, L., & Mars, J. (2017) · IEEE/ACM International Symposium on Microarchitecture (MICRO) · doi:10.1145/3123939.3123970 · Cited by 57 · see citing works
A survey of spintronic architectures for processing-in-memory and neural networks
Umesh, S., & Mittal, S. (2018) · Journal of Systems Architecture · doi:10.1016/j.sysarc.2018.11.005 · Cited by 57 · see citing works
Self-citationDeepCPU: serving RNN-based deep learning models 10x faster
Zhang, M., Rajbhandari, S., Wang, W., & He, Y. (2018) · USENIX Annual Technical Conference · Cited by 53 · see citing works
A Survey Of Techniques for Architecting DRAM Caches
Mittal, S., & Vetter, J. S. (2015) · IEEE Transactions on Parallel and Distributed Systems · doi:10.1109/tpds.2015.2461155 · Cited by 51 · see citing works
Self-citation1988 International Conference on Supercomputing
(1987) · Computer · doi:10.1109/mc.1987.1663431 · Cited by 49 · see citing works
Optimizing convolution operations on GPUs using adaptive tiling
van Werkhoven, B., Maassen, J., Bal, H. E., et al. (2013) · Future Generation Computer Systems · doi:10.1016/j.future.2013.09.003 · Cited by 45 · see citing works
A Survey of Techniques for Architecting and Managing GPU Register File
Mittal, S. (2016) · IEEE Transactions on Parallel and Distributed Systems · doi:10.1109/tpds.2016.2546249 · Cited by 45 · see citing works
Self-citationDryden, N., Maruyama, N., Moon, T., et al. (2018) · doi:10.1109/mlhpc.2018.8638639 · Cited by 45 · see citing works
Holmes, C., Mawhirter, D., He, Y., Yan, F., & Wu, B. (2019) · European Conference on Computer Systems · doi:10.1145/3302424.3303949 · Cited by 45 · see citing works
Lym, S., Lee, D., O’Connor, M., Chatterjee, N., & Erez, M. (2019) · IEEE International Symposium on Performance Analysis of Systems and Software (ISPASS) · doi:10.1109/ispass.2019.00041 · Cited by 44 · see citing works
Zero and data reuse-aware fast convolution for deep neural networks on GPU
Park, H., Kim, D., Ahn, J., & Yoo, S. (2016) · International Conference on Hardware/Software Codesign and System Synthesis · doi:10.1145/2968456.2968476 · Cited by 43 · see citing works
GradientFlow: Optimizing Network Performance for Large-Scale Distributed DNN Training
Sun, P., Wen, Y., Han, R., et al. (2019) · IEEE Transactions on Big Data · doi:10.1109/tbdata.2019.2957478 · Cited by 43 · see citing works
Tian, J. W., & Hong, S. (2019) · International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS) · doi:10.1145/3297858.3304038 · Cited by 41 · see citing works
Performance Evaluation and Optimization of HBM-Enabled GPU for Data-Intensive Applications
Zhu, M., Zhuo, Y., Wang, C., et al. (2018) · IEEE Transactions on Very Large Scale Integration (VLSI) Systems · doi:10.1109/tvlsi.2018.2791442 · Cited by 38 · see citing works
International Conference on Parallel Processing, 2004. ICPP 2004.
(2004) · doi:10.1109/icpp.2004 · Cited by 38 · see citing works
Awan, A. A., Chu, C.-H., Subramoni, H., et al. (2018) · IEEE International Conference on High Performance Computing (HiPC) · doi:10.1109/hipc.2018.00024 · Cited by 37 · see citing works
A survey of encoding techniques for reducing data-movement energy
Mittal, S., & Nag, S. (2018) · Journal of Systems Architecture · doi:10.1016/j.sysarc.2018.11.001 · Cited by 33 · see citing works
Self-citationEvaluating On-Node GPU Interconnects for Deep Learning Workloads
Tallent, N. R., Gawande, N. A., Siegel, C. M., et al. (2017) · Lecture notes in computer science · doi:10.1007/978-3-319-72971-8_1 · Cited by 30 · see citing works
Deep Learning at Scale on NVIDIA V100 Accelerators
Xu, R., Han, F., & Ta, Q. (2018) · doi:10.1109/pmbs.2018.8641600 · Cited by 30 · see citing works
Novel HPC techniques to batch execution of many variable size BLAS computations on GPUs
Abdelfattah, A., Haidar, A., Tomov, S. Z., et al. (2017) · International Conference on Supercomputing · doi:10.1145/3079079.3079103 · Cited by 29 · see citing works
A Survey of Techniques for Improving Security of GPUs
Mittal, S., Abhinaya, S. B., Reddy, M., & Ali, I. (2018) · Journal of Hardware and Systems Security · doi:10.1007/s41635-018-0039-0 · Cited by 29 · see citing works
Self-citationHigh Performance Computing Symposium 2011
Talon, S., Mousseau, N., Peslherbe, G. H., et al. (2012) · Journal of Physics Conference Series · doi:10.1088/1742-6596/341/1/011001 · Cited by 28 · see citing works
SuperNeurons: Dynamic GPU Memory Management for Training Deep Neural NetworksPreprint
Wang, L., Ye, J., Zhao, Y., Wu, W., Li, A., Song, S. L., Xu, Z., & Kraska, T. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1801.04380 · Cited by 28 · see citing works
moDNN: Memory optimal DNN training on GPUs
Chen, X., Chen, D. Z., & Hu, X. S. (2018) · Design, Automation, and Test in Europe · doi:10.23919/date.2018.8341972 · Cited by 28 · see citing works
Dong, S., Gong, X., Sun, Y., et al. (2018) · ACM/SPEC International Conference on Performance Engineering (ICPE) · doi:10.1145/3184407.3184423 · Cited by 27 · see citing works
Accelerating Deep Learning Frameworks with Micro-Batches
Oyama, Y., Ben‐Nun, T., Hoefler, T., & Matsuoka, S. (2018) · IEEE International Conference on Cluster Computing (CLUSTER) · doi:10.1109/cluster.2018.00058 · Cited by 24 · see citing works
Bridging the Semantic Gaps of GPU Acceleration for Scale-out CNN-based Big Data Processing
Song, M., Hu, Y., Xu, Y., et al. (2016) · International Conference on Parallel Architectures and Compilation Techniques · doi:10.1145/2967938.2967944 · Cited by 20 · see citing works
ZNNi: maximizing the inference throughput of 3D convolutional networks on CPUs and GPUs
Zlateski, A., Lee, K., & Seung, H. S. (2016) · IEEE International Conference on High Performance Computing, Data, and Analytics · doi:10.5555/3014904.3015002 · Cited by 20 · see citing works
DLoBD: A Comprehensive Study of Deep Learning over Big Data Stacks on HPC Clusters
Lu, X., Shi, H., Biswas, R., et al. (2018) · IEEE Transactions on Multi-Scale Computing Systems · doi:10.1109/tmscs.2018.2845886 · Cited by 20 · see citing works
A performance analysis framework for exploiting GPU microarchitectural capability
Zhou, K., Tan, G., Zhang, X., Wang, C., & Sun, N. (2017) · International Conference on Supercomputing · doi:10.1145/3079079.3079083 · Cited by 19 · see citing works
Campos, V., Sastre, F., Yagües, M., Torres, J., & Giró-i-Nieto, X. (2017) · IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing (CCGrid) · doi:10.1109/ccgrid.2017.110 · Cited by 16 · see citing works
Speeding up Deep Learning with Transient Servers
Li, S., Walls, R. J., Xu, L., et al. (2019) · IEEE International Conference on Autonomic Computing (ICAC) · doi:10.1109/icac.2019.00024 · Cited by 16 · see citing works
Efficient Communications in Training Large Scale Neural Networks
Zhao, Y., Wang, L., Wu, W., George Bosilca, A., Vuduc, R. W., Ye, J., Tang, W., & Xu, Z. (2017) · Thematic Workshops of ACM Multimedia · doi:10.1145/3126686.3126749 · Cited by 15 · see citing works
Optimizing Memory Efficiency for Convolution Kernels on Kepler GPUs
Chen, X., Chen, J., Chen, D. Z., & Hu, X. S. (2017) · ACM/IEEE Design Automation Conference (DAC) · doi:10.1145/3061639.3062297 · Cited by 14 · see citing works
A survey on applications and architectural-optimizations of Micron’s Automata Processor
Mittal, S. (2019) · Journal of Systems Architecture · doi:10.1016/j.sysarc.2019.07.006 · Cited by 12 · see citing works
Self-citationFast Convolution Operations on Many-Core Architectures
Li, S., Zhang, Y., Xiang, C., & Shi, L. (2015) · IEEE International Conference on High Performance Computing and Communications; IEEE International Conference on Smart City; IEEE International Conference on Data Science and Systems (HPCC/SmartCity/DSS) · doi:10.1109/hpcc-css-icess.2015.94 · Cited by 11 · see citing works
Involving CPUs into Multi-GPU Deep Learning
Le, T., Sekiyama, T., Negishi, Y., et al. (2018) · ACM/SPEC International Conference on Performance Engineering (ICPE) · doi:10.1145/3184407.3184424 · Cited by 10 · see citing works
Fast convolution kernels on pascal GPU with high memory efficiency
Chang, Q., Onishi, M., & Maruyama, T. (2018) · High Performance Computing Symposium · doi:10.5555/3213069.3213072 · Cited by 3 · see citing works
A GPU memory efficient speed-up scheme for training ultra-deep neural networks
Guo, J., Liu, W., Wang, W., Lu, Q., Hu, S., Han, J., & Li, R. (2019) · ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming · doi:10.1145/3293883.3295718 · Cited by 3 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Review of deep learning: concepts, CNN architectures, challenges, applications, future directions
Alzubaidi, L., Zhang, J., Humaidi, A. J., Al-Dujaili, A., Duan, Y., Al-Shamma, O., Santamaría, J., Fadhel, M. A., Al-Amidie, M., & Farhan, L. (2021) · Journal of Big Data · doi:10.1186/s40537-021-00444-8 · Cited by 7,906 · see citing works
A review of green artificial intelligence: Towards a more sustainable future
Bolón-Canedo, V., Morán-Fernández, L., Cancela, B., & Alonso-Betanzos, A. (2024) · Neurocomputing · doi:10.1016/j.neucom.2024.128096 · Cited by 486 · see citing works
A survey on deep transfer learning to edge computing for mitigating the COVID-19 pandemic
Sufian, A., Ghosh, A., Sadiq, A. S., & Smarandache, F. (2020) · Journal of Systems Architecture · doi:10.1016/j.sysarc.2020.101830 · Cited by 156 · see citing works
Benchmark Analysis of YOLO Performance on Edge Intelligence Devices
Feng, H., Mu, G., Zhong, S., Zhang, P., & Yuan, T. (2022) · Cryptography · doi:10.3390/cryptography6020016 · Cited by 100 · see citing works
A survey on hardware security of DNN models and accelerators
Mittal, S., Gupta, H., & Srivastava, S. (2021) · Journal of Systems Architecture · doi:10.1016/j.sysarc.2021.102163 · Cited by 47 · see citing works
Self-citation
- Authors
- Mittal, S., & Vaishay, S.
- Affiliations
- Indian Institute of Technology Hyderabad (IN)
- Corresponding author
- Mittal, Sparsh
- ORCID
- Mittal, Sparsh
- Venue
- Journal of Systems Architecture
- Publisher
- Elsevier BV
- Published
- 2019-10
- Volume
- 99
- Pages
- 101635
- ISSN
- 1383-7621 (ISSN-L)
- OpenAlex
- W2969414999
- Cited by
- 197
- References
- 76
- Citations by year
- 2026: 17 · 2025: 30 · 2024: 35 · 2023: 19 · 2022: 39 · 2021: 33 · 2020: 22 · 2019: 2
- FWCI
- 24.44
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Journal of Systems Architecture (publishedVersion)
- Research Archive of Indian Institute of Technology Hyderabad (Indian Institute of Technology Hyderabad) (submittedVersion)
- Funders
- Illinois State University
- Oak Ridge National Laboratory
- Semiconductor Research Corporation (2018-IR-2852, SRC)
- Topics
- Physical Sciences › Computer Science › Hardware and Architecture › Parallel Computing and Optimization Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Computer Science › Hardware and Architecture › Embedded Systems Design Techniques
- Keywords
- deep learning optimization, GPU acceleration, inference acceleration, training acceleration, multi-GPU, system-level optimization, high-performance computing
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
