Diagonalwise Refactorization: An Efficient Training Method for Depthwise Convolutions
Open accessAbstract
Depthwise convolutions provide significant performance benefits owing to the reduction in both parameters and mult-adds. However, training depthwise convolution layers with GPUs is slow in current deep learning frameworks because their implementations cannot fully utilize the GPU capacity. To address this problem, in this paper we present an efficient method (called diagonalwise refactorization) for accelerating the training of depthwise convolution layers. Our key idea is to rearrange the weight vectors of a depthwise convolution into a large diagonal weight matrix so as to convert the depthwise convolution into one single standard convolution, which is well supported by the cuDNN library that is highly-optimized for GPU computations. We have implemented our training method in five popular deep learning frameworks. Evaluation results show that our proposed method gains 15.4× training speedup on Darknet, 8.4× on Caffe, 5.4× on PyTorch, 3.5× on MXNet, and 1.4× on TensorFlow, compared to their original implementations of depthwise convolutions.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Qin et al. (2018) show that Depthwise 3x3 Convolutions comprised just 3.06% of an example network's Multiply-Add operations, while utilizing 82.86% of the total training time in the FPO-efficient MobileNet architecture.
“Similarly, Qin et al. (2018) show how Depthwise 3x3 Convolutions comprised just 3.06% of an example network's Multiply-Add operations, while utilizing 82.86% of the total training time in the FPO-efficient MobileNet architecture Howard et al. (2017).”
From Towards the Systematic Reporting of the Energy and Carbon Footprints of Machine Learning (2020) · cites this paper as Qin et al. (2018)
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · see citing works
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155 · see citing works
Very Deep Convolutional Networks for Large-Scale Image Recognition
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1409.1556 · Cited by 74,248 · see citing works
Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
Ren, S., He, K., Girshick, R., & Sun, J. (2017) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2577031 · Cited by 56,035 · see citing works
Going deeper with convolutions
Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., & Rabinovich, A. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298594 · Cited by 48,562 · see citing works
Xception: Deep Learning with Depthwise Separable Convolutions
Chollet, F. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.195 · Cited by 20,380 · see citing works
YOLO9000: Better, Faster, Stronger
Redmon, J., & Farhadi, A. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.690 · Cited by 19,892 · see citing works
SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentation
Badrinarayanan, V., Kendall, A., & Cipolla, R. (2017) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2644615 · Cited by 19,631 · see citing works
MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
Howard, A. G., Zhu, M., Chen, B., Kalenichenko, D., Wang, W., Weyand, T., Andreetto, M., & Adam, H. (2017) · arXiv · doi:10.48550/arxiv.1704.04861 · Cited by 19,015 · see citing works
TensorFlow: A system for large-scale machine learningPreprint
Abadi, M., Barham, P., Chen, J., Chen, Z., Davis, A., Dean, J., Devin, M., Ghemawat, S., Irving, G., Isard, M., et al. (2016) · arXiv (Cornell University) · doi:10.5555/3026877.3026899 · Cited by 16,369 · see citing works
Aggregated Residual Transformations for Deep Neural Networks
Xie, S., Girshick, R., Dollar, P., Tu, Z., & He, K. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.634 · Cited by 12,465 · see citing works
Caffe: Convolutional Architecture for Fast Feature Embedding
Jia, Y., Shelhamer, E., Donahue, J., Karayev, S., Long, J., Girshick, R., Guadarrama, S., & Darrell, T. (2014) · Proceedings of the 22nd ACM international conference on Multimedia · doi:10.1145/2647868.2654889 · Cited by 11,077 · see citing works
ShuffleNet: An Extremely Efficient Convolutional Neural Network for Mobile Devices
Zhang, X., Zhou, X., Lin, M., & Sun, J. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00716 · Cited by 9,982 · see citing works
SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <0.5MB model size
Iandola, F. N., Han, S., Moskewicz, M. W., Ashraf, K., Dally, W. J., & Keutzer, K. (2016) · arXiv · doi:10.48550/arxiv.1602.07360 · Cited by 6,201 · see citing works
Han, S., Mao, H., & Dally, W. J. (2015) · arXiv · doi:10.48550/arxiv.1510.00149 · Cited by 4,662 · see citing works
XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networks
Rastegari, M., Ordonez, V., Redmon, J., & Farhadi, A. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46493-0_32 · Cited by 4,408 · see citing works
Learning Efficient Convolutional Networks through Network Slimming
Liu, Z., Li, J., Shen, Z., Huang, G., Yan, S., & Zhang, C. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.298 · Cited by 2,761 · see citing works
Speed/Accuracy Trade-Offs for Modern Convolutional Object Detectors
Huang, J., Rathod, V., Sun, C., Zhu, M., Korattikara, A., Fathi, A., Fischer, I., Wojna, Z., Song, Y., Guadarrama, S., et al. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.351 · Cited by 2,743 · see citing works
Channel Pruning for Accelerating Very Deep Neural Networks
He, Y., Zhang, X., & Sun, J. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.155 · Cited by 2,647 · see citing works
Binarized Neural NetworksPreprint
Hubara, I., Soudry, D., & El-Yaniv, R. (2016) · arXiv.org · doi:10.48550/arxiv.1602.02505 · Cited by 2,241 · see citing works
BinaryConnect: Training Deep Neural Networks with binary weights during propagations
Courbariaux, M., Bengio, Y., & David, J.-P. (2015) · arXiv · doi:10.48550/arxiv.1511.00363 · Cited by 2,180 · see citing works
MXNet: A Flexible and Efficient Machine Learning Library for Heterogeneous Distributed Systems
Chen, T., Li, M., Li, Y., Lin, M., Wang, N., Wang, M., Xiao, T., Xu, B., Zhang, C., & Zhang, Z. (2015) · arXiv · doi:10.48550/arxiv.1512.01274 · Cited by 1,819 · see citing works
Self-citationDoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients
Zhou, S., Wu, Y., Ni, Z., Zhou, X., Wen, H., & Zou, Y. (2016) · arXiv · doi:10.48550/arxiv.1606.06160 · Cited by 1,690 · see citing works
Exploiting Linear Structure Within Convolutional Networks for Efficient Evaluation
Denton, R., Zaremba, W., Bruna, J., LeCun, Y., & Fergus, R. (2014) · arXiv · doi:10.48550/arxiv.1404.0736 · Cited by 1,414 · see citing works
cuDNN: Efficient Primitives for Deep LearningPreprint
Chetlur, S., Woolley, C., Vandermersch, P., Cohen, J., Tran, J. J., Catanzaro, B., & Shelhamer, E. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1410.0759 · Cited by 1,394 · see citing works
Torch7: A Matlab-like Environment for Machine Learning
Collobert, R., Kavukcuoglu, K., & Farabet, C. (2011) · Infoscience (Ecole Polytechnique Fédérale de Lausanne) · Cited by 1,241 · see citing works
Incremental Network Quantization: Towards Lossless CNNs with Low-Precision WeightsPreprint
Zhou, A., Yao, A., Guo, Y., Xu, L., & Chen, Y. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1702.03044 · Cited by 963 · see citing works
Accelerating Very Deep Convolutional Networks for Classification and Detection
Zhang, X., Zou, J., He, K., & Sun, J. Z. (2015) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2015.2502579 · Cited by 847 · see citing works
Speeding-up Convolutional Neural Networks Using Fine-tuned CP-DecompositionPreprint
Lebedev, V., Ganin, Y., Rakhuba, M., Oseledets, I. V., & Lempitsky, V. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1412.6553 · Cited by 511 · see citing works
Shift: A Zero FLOP, Zero Parameter Alternative to Spatial Convolutions
Wu, B., Wan, A., Yue, X., Jin, P., Zhao, S., Golmant, N., Gholaminejad, A., Gonzalez, J. E., & Keutzer, K. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00951 · Cited by 395 · see citing works
FxpNet: Training a deep convolutional neural network in fixed-point representation
Chen, X., Hu, X., Zhou, H., & Xu, N. (2017) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn.2017.7966159 · Cited by 48 · see citing works
Flagged first, then by citations
6 of the 37 references couldn’t be matched to a known paper.
Cited by
Sorted by citations
Towards the Systematic Reporting of the Energy and Carbon Footprints of Machine Learning
Henderson, P., Hu, J., Romoff, J., Brunskill, E., Jurafsky, D., & Pineau, J. (2020) · arXiv · doi:10.48550/arxiv.2002.05651 · Cited by 393 · see citing works
“Similarly, Qin et al. (2018) show how Depthwise 3x3 Convolutions comprised just 3.06% of an example network’s Multiply-Add operations, while utilizing 82.86% of the total training time in the FPO-efficient MobileNet architecture Howard et al. (2017).”
BackgroundMethodInfluentialA Lightweight Spectral-Spatial Convolution Module for Hyperspectral Image Classification
Meng, Z., Jiao, L., Liang, M., & Zhao, F. (2021) · IEEE Geoscience and Remote Sensing Letters · doi:10.1109/lgrs.2021.3069202 · Cited by 60 · see citing works
High Performance Depthwise and Pointwise Convolutions on Mobile Devices
Zhang, P., Lo, E., & Lu, B. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i04.6159 · Cited by 57 · see citing works
Real-Time Classification of Rubber Wood Boards Using an SSR-Based CNN
Liu, S., Jiang, W., Wu, L., Wen, H., Liu, M., & Wang, Y. (2020) · IEEE Transactions on Instrumentation and Measurement · doi:10.1109/tim.2020.3001370 · Cited by 48 · see citing works
Learn to Code Sustainably: An Empirical Study on Green Code Generation
Vartziotis, T., Dellatolas, I., Dasoulas, G., Schmidt, M., Schneider, F., Hoffmann, T., Kotsopoulos, S. D., & Keckeisen, M. (2024) · International Workshop on Large Language Models for Code (LLM4Code) · doi:10.1145/3643795.3648394 · Cited by 23 · see citing works
Mao, Y., He, Z., Ma, Z., Tang, X., & Wang, Z. (2019) · IEEE Access · doi:10.1109/access.2019.2932733 · Cited by 22 · see citing works
Han, M., Wang, L., Xiao, L., Cai, T., Wang, Z., Xu, X., & Zhang, C. (2024) · IEEE Transactions on Computers · doi:10.1109/tc.2024.3398500 · Cited by 16 · see citing works
Trash Detection Algorithm Suitable for Mobile Robots Using Improved YOLO
Harada, R., OYAMA, T., Fujimoto, K., Shimizu, T., Ozawa, M., Samuel, A. J., & Sakai, M. (2023) · Journal of Advanced Computational Intelligence and Intelligent Informatics · doi:10.20965/jaciii.2023.p0622 · Cited by 15 · see citing works
LeanConvNets: Low-Cost Yet Effective Convolutional Neural Networks
Ephrath, J., Eliasof, M., Ruthotto, L., Haber, E., & Treister, E. (2020) · IEEE Journal of Selected Topics in Signal Processing · doi:10.1109/jstsp.2020.2972775 · Cited by 12 · see citing works
He, H., Gao, K., Tan, W., Wang, L., Chen, N., Ma, L., & Li, J. (2022) · International Journal of Applied Earth Observation and Geoinformation · doi:10.1016/j.jag.2022.102826 · Cited by 12 · see citing works
Real-Time Monocular Depth Estimation on Embedded Systems
Feng, C., Zhang, C., Chen, Z., Hu, W., & Ge, L. (2024) · Proceedings - International Conference on Image Processing · doi:10.1109/icip51287.2024.10648152 · Cited by 11 · see citing works
Optimizing depthwise separable convolution on DCU
Liu, Z., Hao, M., Zhang, W., Lu, G., Tian, X., Yang, S., Xie, M., Dai, J., Yuan, C., Wang, D., et al. (2024) · CCF Transactions on High Performance Computing · doi:10.1007/s42514-024-00200-3 · Cited by 11 · see citing works
Kumar, G., Sharma, N., & Paul, A. (2023) · Medical Physics · doi:10.1002/mp.16722 · Cited by 9 · see citing works
Accelerating Compact Convolutional Neural Networks with Multi-threaded Data Streaming
Chen, W., Wang, Z., Li, S., Yu, Z., & Li, H. (2019) · IEEE Computer Society Annual Symposium on VLSI · doi:10.1109/isvlsi.2019.00099 · Cited by 8 · see citing works
Decentralized Federated Learning for Agricultural Plant Diseases Identification on Edge Devices
Thonglek, K., & Rattanatamrong, P. (2023) · International Joint Symposium on Artificial Intelligence and Natural Language Processing (iSAI-NLP) · doi:10.1109/isai-nlp60301.2023.10354694 · Cited by 7 · see citing works
Singh, A. K., & Tao, X. (2020) · IEEE Symposium Series on Computational Intelligence (SSCI) · doi:10.1109/ssci47803.2020.9308292 · Cited by 5 · see citing works
Optimizing 3D Convolution Kernels on Stereo Matching for Resource Efficient Computations
Xiao, J., Ma, D., & Yamane, S. (2021) · Sensors · doi:10.3390/s21206808 · Cited by 4 · see citing works
LeanResNet: A Low-cost Yet Effective Convolutional Residual NetworksPreprint
Ephrath, J., Ruthotto, L., Haber, E., & Treister, E. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1904.06952 · Cited by 3 · see citing works
Accelerating Deep Convolutional Neural Network Inference Based on OpenCL
Wu, Y., Zhu, H., Zhang, L., Hou, B., & Jiao, L. (2022) · IFIP advances in information and communication technology · doi:10.1007/978-3-031-14903-0_11 · Cited by 2 · see citing works
Accelerating the Training of Video Super-resolution Models
Lin, L., Wang, X., Qi, Z., & Shan, Y. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i2.25246 · Cited by 2 · see citing works
Nguyen, D. T., Je, H., Nguyen, T. N., Ryu, S., Lee, K., & Lee, H. (2022) · IEEE Transactions on Circuits and Systems I Regular Papers · doi:10.1109/tcsi.2022.3153288 · Cited by 47 · see citing works
Caterpillar: A Pure-MLP Architecture with Shifted-Pillars-Concatenation
Sun, J., Shi, X., Wang, Z., Xu, K., Shen, H. T., & Zhu, X. (2024) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3664647.3680809 · Cited by 1 · see citing works
Lai, K. W., Yeoh, P. S. Q., Goh, S. L., Hasikin, K. B., & Wu, X. (2022) · International Conference on Healthcare Engineering (ICHE) · doi:10.1109/iche55634.2022.10179865 · Cited by 1 · see citing works
Flamingo: A User-Centric System for Fast and Energy-Efficient DNN Training on Smartphones
Singapuram, S. S. V., Hu, C., Lai, F., Zhang, C., & Chowdhury, M. (2023) · doi:10.1145/3630048.3630183
Leveraging Contextual Hierarchies in Semantic Segmentation
Meyarian, A. (2024) · doi:10.12794/metadc2415954
Flash-DWC: Making Depthwise Convolution Compute-Efficient on GPUs
Zhang, Z., Zhao, Y., Sun, J., Liang, Q., Chen, J., Shi, J., & An, H. (2026) · ACM Transactions on Architecture and Code Optimization · doi:10.1145/3837088
Self-citation
- Authors
- Qin, Z., Zhang, Z., Li, D., Zhang, Y., & Peng, Y.
- Affiliations
- National University of Defense Technology (CN)
- Venue
- 2018 International Joint Conference on Neural Networks (IJCNN)
- Publisher
- IEEE
- Published
- 2018-07
- Pages
- 1-8
- ISSN
- 2161-4407 (ISSN-L)
- OpenAlex
- W2963404959
- Cited by
- 27
- References
- 37
- Citations by year
- 2026: 1 · 2024: 6 · 2023: 5 · 2022: 4 · 2021: 3 · 2020: 5 · 2019: 3
- FWCI
- 0.83
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- International Joint Conference on Neural Networks (IJCNN) (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- Major State Basic Research Development Program of China
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Computer Science › Hardware and Architecture › Parallel Computing and Optimization Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Computer Science and Engineering
- Keywords
- depthwise convolution, training acceleration, GPU acceleration, cuDNN, dark web, Caffe, PyTorch, MXNet, TensorFlow
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
