The DOI matches its Crossref record; 63 of the 63 references it cites were found in a registry or an index.
DepGraph: Towards Any Structural Pruning
Open accessAbstract
Structural pruning enables model acceleration by removing structurally-grouped parameters from neural networks. However, the parameter-grouping patterns vary widely across different models, making architecture-specific pruners, which rely on manually-designed grouping schemes, non-generalizable to new architectures. In this work, we study a highly-challenging yet barely-explored task, any structural pruning, to tackle general structural pruning of arbitrary architecture like CNNs, RNNs, GNNs and Transformers. The most prominent obstacle towards this goal lies in the structural coupling, which not only forces different layers to be pruned simultaneously, but also expects all removed parameters to be consistently unimportant, thereby avoiding structural issues and significant performance degradation after pruning. To address this problem, we propose a general and fully automatic method, Dependency Graph (DepGraph), to explicitly model the dependency between layers and comprehensively group coupled parameters for pruning. In this work, we extensively evaluate our method on several architectures and tasks, including ResNe(X)t, DenseNet, MobileNet and Vision transformer for images, GAT for graph, DGCNN for 3D point cloud, alongside LSTM for language, and demonstrate that, even with a simple norm-based criterion, the proposed method consistently yields gratifying performances.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
Very Deep Convolutional Networks for Large-Scale Image Recognition
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1409.1556 · Cited by 114,148
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333
Densely Connected Convolutional Networks
Huang, G., Liu, Z., Van Der Maaten, L., & Weinberger, K. Q. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.243 · Cited by 48,566
Going deeper with convolutions
Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., & Rabinovich, A. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298594 · Cited by 48,562
MobileNetV2: Inverted Residuals and Linear Bottlenecks
Sandler, M., Howard, A., Zhu, M., Zhmoginov, A., & Chen, L.-C. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00474 · Cited by 28,755
Learning Multiple Layers of Features from Tiny Images
Krizhevsky, A. (2024) · doi:10.57702/zp44cu3g · Cited by 23,395
Distilling the Knowledge in a Neural Network
Hinton, G., Vinyals, O., & Dean, J. (2015) · arXiv · doi:10.48550/arxiv.1503.02531 · Cited by 14,204
Aggregated Residual Transformations for Deep Neural Networks
Xie, S., Girshick, R., Dollar, P., Tu, Z., & He, K. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.634 · Cited by 12,465
Veličković, P., Cucurull, G., Casanova, A., Romero, A., Liò, P., & Bengio, Y. (2017) · arXiv · doi:10.48550/arxiv.1710.10903 · Cited by 12,107
Inductive Representation Learning on Large Graphs
Hamilton, W. L., Ying, R., & Leskovec, J. (2017) · arXiv · doi:10.48550/arxiv.1706.02216 · Cited by 10,639
Veličković, P., Casanova, A., Liò, P., Cucurull, G., Romero, A., & Bengio, Y. (2017) · arXiv (Cornell University) · doi:10.17863/cam.48429 · Cited by 8,657
Dynamic Graph CNN for Learning on Point Clouds
Wang, Y., Sun, Y., Liu, Z., Sarma, S. E., Bronstein, M. M., & Solomon, J. M. (2019) · ACM Transactions on Graphics · doi:10.1145/3326362 · Cited by 7,199
Targeted Branching for the Maximum Independent Set Problem Using Graph Neural Networks
Silva, G., Rodrigues, M., Teixeira, A., & Amorim, M. (2024) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.sea.2024.20 · Cited by 5,183
Han, S., Mao, H., & Dally, W. J. (2015) · arXiv · doi:10.48550/arxiv.1510.00149 · Cited by 4,662
3D ShapeNets: A deep representation for volumetric shapes
Wu, Z., Song, S., Khosla, A., Yu, F., Zhang, L., Tang, X., & Xiao, J. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298801 · Cited by 4,550
Character-level Convolutional Networks for Text Classification
Zhang, X., Zhao, J., & LeCun, Y. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1509.01626 · Cited by 3,205
Learning Efficient Convolutional Networks through Network Slimming
Liu, Z., Li, J., Shen, Z., Huang, G., Yan, S., & Zhang, C. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.298 · Cited by 2,761
Graves, A. (2012) · Studies in Computational Intelligence · doi:10.1007/978-3-642-24797-2_4 · Cited by 2,727
Channel Pruning for Accelerating Very Deep Neural Networks
He, Y., Zhang, X., & Sun, J. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.155 · Cited by 2,647 · cited 2×
“In the literature, researchers handle layer dependencies and enable structural pruning on deep neural networks with manually-designed and model-specific schemes [21, 28].”
BackgroundDesigning Network Design Spaces
Radosavovic, I., Kosaraju, R. P., Girshick, R., He, K., & Dollar, P. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.01044 · Cited by 2,084
ThiNet: A Filter Level Pruning Method for Deep Neural Network Compression
Luo, J.-H., Wu, J., & Lin, W. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.541 · Cited by 1,832
AMC: AutoML for Model Compression and Acceleration on Mobile Devices
He, Y., Lin, J., Liu, Z., Wang, H., Li, L.-J., & Han, S. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01234-2_48 · Cited by 1,400
Filter Pruning via Geometric Median for Deep Convolutional Neural Networks Acceleration
He, Y., Liu, P., Wang, Z., Hu, Z., & Yang, Y. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00447 · Cited by 1,217
Quantized Convolutional Neural Networks for Mobile Devices
Wu, J., Leng, C., Wang, Y., Hu, Q., & Cheng, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.521 · Cited by 1,212
Soft Filter Pruning for Accelerating Deep Convolutional Neural Networks
He, Y., Kang, G., Dong, X., Fu, Y., & Yang, Y. (2018) · Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2018/309 · Cited by 1,020
Importance Estimation for Neural Network Pruning
Molchanov, P., Mallya, A., Tyree, S., Frosio, I., & Kautz, J. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.01152 · Cited by 983 · cited 1×
“It is also worth noting that in structural pruning, grouped layers are pruned simultaneously, which expects the parameters in the removed group to be consistently unimportant, which brings certain difficulties to existing importance criteria designed for a single layer [20,26,28,39].”
BackgroundPruning and quantization for deep neural network acceleration: A survey
Liang, T., Glossner, J., Wang, L., Shi, S., & Zhang, X. (2021) · Neurocomputing · doi:10.1016/j.neucom.2021.07.045 · Cited by 943
HRank: Filter Pruning Using High-Rank Feature Map
Lin, M., Ji, R., Wang, Y., Zhang, Y., Zhang, B., Tian, Y., & Shao, L. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00160 · Cited by 866
NISP: Pruning Networks Using Neuron Importance Score Propagation
Yu, R., Li, A., Chen, C.-F., Lai, J.-H., Morariu, V. I., Han, X., Gao, M., Lin, C.-Y., & Davis, L. S. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00958 · Cited by 794
The Transitive Reduction of a Directed Graph
Aho, A. V., Garey, M. R., & Ullman, J. D. (1972) · SIAM Journal on Computing · doi:10.1137/0201008 · Cited by 718
NetAdapt: Platform-Aware Neural Network Adaptation for Mobile Applications
Yang, T.-J., Howard, A., Chen, B., Zhang, X., Go, A., Sandler, M., Sze, V., & Adam, H. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01249-6_18 · Cited by 572
MetaPruning: Meta Learning for Automatic Neural Network Channel Pruning
Liu, Z., Mu, H., Zhang, X., Guo, Z., Yang, X., Cheng, K.-T., & Sun, J. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00339 · Cited by 563
Data-Driven Sparse Structure Selection for Deep Neural Networks
Huang, Z., & Wang, N. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01270-0_19 · Cited by 549
Torchvision the machine-vision package of torch
Marcel, S., & Rodriguez, Y. (2010) · Proceedings of the 18th ACM international conference on Multimedia · doi:10.1145/1873951.1874254 · Cited by 416
Layer-adaptive sparsity for the Magnitude-based PruningPreprint
Lee, J., Park, S., Mo, S., Ahn, S., & Shin, J. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.07611 · Cited by 398
Dataset Distillation: A Comprehensive Review
Yu, R., Liu, S., & Wang, X. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3323376 · Cited by 302
Self-citationDistilling Knowledge From Graph Convolutional Networks
Yang, Y., Qiu, J., Song, M., Tao, D., & Wang, X. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00710 · Cited by 248
Self-citationMovement pruning: Adaptive sparsity by fine-tuning
Sanh, V., Wolf, T., & Rush, A. M. (2020) · arXiv · doi:10.48550/arxiv.2005.07683 · Cited by 218
Dataset Distillation Via Factorization
Liu, S., Wang, K., Wang, X., Ye, J., & Wang, X. (2022) · neural information processing systems · doi:10.52202/068431-0081 · Cited by 208
Self-citationCentripetal SGD for Pruning Very Deep Convolutional Networks With Complicated Structure
Ding, X., Ding, G., Guo, Y., & Han, J. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00508 · Cited by 193
Collaborative Channel Pruning for Deep Networks
Peng, H., Wu, J., Chen, S., & Huang, J. (2019) · International Conference on Machine Learning · Cited by 191
Comparing Rewinding and Fine-tuning in Neural Network Pruning
Renda, A., Frankle, J., & Carbin, M. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2003.02389 · Cited by 183
ResRep: Lossless CNN Pruning via Decoupling Remembering and Forgetting
Ding, X., Hao, T., Tan, J., Liu, J., Han, J., Guo, Y., & Ding, G. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00447 · Cited by 177
Network Pruning via Performance Maximization
Gao, S., Huang, F., Cai, W., & Huang, H. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.00915 · Cited by 167
Towards Efficient Model Compression via Learned Global Ranking
Chin, T.-W., Ding, R., Zhang, C., & Marculescu, D. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00159 · Cited by 165
Factorizing Knowledge in Neural Networks
Yang, X., Ye, J., & Wang, X. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-19830-4_5 · Cited by 154
Self-citationNeural Network Pruning With Residual-Connections and Limited-Data
Luo, J.-H., & Wu, J. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.00153 · Cited by 147
Auto-Balanced Filter Pruning for Efficient Convolutional Neural Networks
Ding, X., Ding, G., Han, J., & Tang, S. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.12262 · Cited by 117
Approximated Oracle Filter Pruning for Destructive CNN Width Optimization
Ding, X., Ding, G., Guo, Y., Han, J., & Yan, C. (2019) · arXiv · doi:10.48550/arxiv.1905.04748 · Cited by 96
Neuron-level Structured Pruning using Polarization Regularizer
Zhuang, T., Zhang, Z., Huang, Y., Zeng, X., Shuang, K., & Li, X. (2020) · neural information processing systems · Cited by 96
CP-ViT: Cascade Vision Transformer Pruning via Progressive Sparsity PredictionPreprint
Song, Z., Xu, Y., He, Z., Li, J., Jing, N., & Liang, X. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2203.04570 · Cited by 71
Slimmable Dataset Condensation
Liu, S., Ye, J., Yu, R., & Wang, X. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.00366 · Cited by 65
Self-citationNetwork Slimming by Slimmable Networks: Towards One-Shot Architecture Search for Channel NumbersPreprint
Yu, J.-H., & Huang, T. (2019) · arXiv.org · doi:10.48550/arxiv.1903.11728 · Cited by 59
Accelerate CNNs from Three Dimensions: A Comprehensive Pruning FrameworkPreprint
Wang, W., Chen, M., Zhao, S., Chen, L., Hu, J., Liu, H., Cai, D., He, X., & Liu, W. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.04879 · Cited by 55
Joint-DetNAS: Upgrade Your Detector with NAS, Pruning and Dynamic Distillation
Yao, L., Pi, R., Xu, H., Zhang, W., Li, Z., & Zhang, T. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.01004 · Cited by 47
A Gradient Flow Framework For Analyzing Network PruningPreprint
Lubana, E. S., & Dick, R. P. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2009.11839 · Cited by 45
Ma, X., Yuan, G., Lin, S., Li, Z., Sun, H., & Wang, Y. (2019) · IEEE/ACM International Symposium on Nanoscale Architectures (NANOARCH) · doi:10.1109/nanoarch47378.2019.181304 · Cited by 44
Self-citationMeta-Aggregator: Learning to Aggregate for 1-bit Graph Neural Networks
Jing, Y., Yang, Y., Wang, X., Song, M., & Tao, D. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00525 · Cited by 38
Self-citationWang, H., Qin, C., Bai, Y., & Fu, Y. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2301.05219 · Cited by 35
Ye, J., Liu, S., & Wang, X. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01928 · Cited by 22
Self-citationParameterized Structured Pruning for Deep Neural Networks
Schindler, G., Roth, W., Pernkopf, F., & Fröning, H. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-64580-9_3 · Cited by 10
Slimmable neural networks for edge devices
Yu, J. (2019) · Illinois Digital Environment for Access to Learning and Scholarship (University of Illinois at Urbana-Champaign) · Cited by 3
Cited by
A Survey on Deep Neural Network Pruning: Taxonomy, Comparison, Analysis, and Recommendations
Cheng, H., Zhang, M., & Shi, J. Q. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3447085 · Cited by 497
NTIRE 2023 Challenge on Efficient Super-Resolution: Methods and Results
Li, Y., Zhang, Y., Timofte, R., Van Gool, L., Yu, L., Li, Y., Li, X., Jiang, T., Wu, Q., Han, M., et al. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw59228.2023.00189 · Cited by 72
“A big problem about collecting more data and creating a new larger dataset for training a bigger and more powerful model is that we always need to update our model to adapt to the new dataset for learning more knowledge than before.”
BackgroundSelf-citationDiffusion Model as Representation Learner
Yang, X., & Wang, X. (2023) · 2023 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01736 · Cited by 58
Self-citation
Metadata
- Authors
- Fang, G., Ma, X., Song, M., Bi Mi, M., & Wang, X.
- Venue
- 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
- Publisher
- IEEE
- Published
- 2023-06
- Pages
- 16091-16101
- ISSN
- 1063-6919 (ISSN-L)
- OpenAlex
- W4386083031
- Cited by
- 501
- References
- 63
- FWCI
- 56.97
- Type
- conference paper
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.