Pruning from Scratch
Open accessAbstract
Network pruning is an important research field aiming at reducing computational costs of neural networks. Conventional approaches follow a fixed paradigm which first trains a large and redundant network, and then determines which units (e.g., channels) are less important and thus can be removed. In this work, we find that pre-training an over-parameterized model is not necessary for obtaining the target pruned structure. In fact, a fully-trained over-parameterized model will reduce the search space for the pruned structure. We empirically show that more diverse pruned structures can be directly pruned from randomly initialized weights, including potential models with better performance. Therefore, we propose a novel network pruning pipeline which allows pruning from scratch with little training overhead. In the experiments for compressing classification models on CIFAR10 and ImageNet datasets, our approach not only greatly reduces the pre-training burden of traditional pruning methods, but also achieves similar or even higher accuracy under the same computation budgets. Our results facilitate the community to rethink the effectiveness of existing techniques used for network pruning.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Deployment of pre-trained DL models into tiny edge devices is now possible after performing techniques to compress DL models and optimize inference.
“Furthermore, deployment of pre-trained DL models into tiny edge devices is now possible, after performing some techniques to compress DL models and optimize the inference.”
From TinyML: Enabling of Inference Deep Learning Models on Ultra-Low-Power IoT Edge Devices for AI Applications (2022) · cites this paper as [15]
The Introduction (p. 1) says deep neural networks are deployed on mobile devices and that pruning compresses models and accelerates inference. It does not establish that these techniques make deployment of pre-trained models on tiny edge devices possible; the paper instead emphasizes pruning without pre-training. (LLM, conf=0.87)
Experimental — accuracy not yet measuredPruning techniques allow removal of redundant structures of network and parameters.
“Pruning techniques allow removal of redundant structures of network and parameters [15,16].”
From TinyML: Enabling of Inference Deep Learning Models on Ultra-Low-Power IoT Edge Devices for AI Applications (2022) · cites this paper as [15]
Quantization techniques convert float-point numbers to minimize precision numbers, intending to shrink the size of the DL model with minimal degradation of accuracy.
“For instance, using quantization techniques, which are conversion techniques that convert float-point numbers to minimize precision numbers, intending to shrink the size of the DL model with minimal degradation of accuracy.”
From TinyML: Enabling of Inference Deep Learning Models on Ultra-Low-Power IoT Edge Devices for AI Applications (2022) · cites this paper as [15]
The Related Work section (p. 2) describes pruning as removing parameters or channels, while the Introduction (p. 2) discusses pruning with comparable accuracy. A search for quantization and lower-precision conversion found no passage explaining those techniques or their effect on model size and accuracy; quantization appears only in a reference title (p. 9). (LLM, conf=0.76)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · see citing works
Self-citationVery Deep Convolutional Networks for Large-Scale Image Recognition
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1409.1556 · Cited by 74,248 · see citing works
ImageNet Large Scale Visual Recognition Challenge
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al. (2015) · International Journal of Computer Vision · doi:10.1007/s11263-015-0816-y · Cited by 39,380 · see citing works
Self-citationRethinking the Inception Architecture for Computer Vision
Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., & Wojna, Z. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.308 · Cited by 32,138 · see citing works
MobileNetV2: Inverted Residuals and Linear Bottlenecks
Sandler, M., Howard, A., Zhu, M., Zhmoginov, A., & Chen, L.-C. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00474 · Cited by 28,755 · see citing works
Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
Ioffe, S., & Szegedy, C. (2015) · arXiv · doi:10.48550/arxiv.1502.03167 · Cited by 27,818 · see citing works
Learning Multiple Layers of Features from Tiny Images
Krizhevsky, A. (2024) · doi:10.57702/zp44cu3g · Cited by 23,890 · see citing works
MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
Howard, A. G., Zhu, M., Chen, B., Kalenichenko, D., Wang, W., Weyand, T., Andreetto, M., & Adam, H. (2017) · arXiv · doi:10.48550/arxiv.1704.04861 · Cited by 19,015 · see citing works
SGDR: Stochastic Gradient Descent with Warm Restarts
Loshchilov, I., & Hutter, F. (2016) · arXiv · doi:10.48550/arxiv.1608.03983 · Cited by 5,309 · see citing works
Han, S., Mao, H., & Dally, W. J. (2015) · arXiv · doi:10.48550/arxiv.1510.00149 · Cited by 4,662 · see citing works
DARTS: Differentiable architecture search
Liu, H., Simonyan, K., & Yang, Y. (2018) · arXiv · doi:10.48550/arxiv.1806.09055 · Cited by 3,248 · see citing works
(1989) · Cited by 3,214 · see citing works
Learning Efficient Convolutional Networks through Network Slimming
Liu, Z., Li, J., Shen, Z., Huang, G., Yan, S., & Zhang, C. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.298 · Cited by 2,761 · see citing works
Channel Pruning for Accelerating Very Deep Neural Networks
He, Y., Zhang, X., & Sun, J. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.155 · Cited by 2,647 · see citing works
Self-citationMobileNetV2: Inverted Residuals and Linear BottlenecksPreprint
Sandler, M., Howard, A., Zhu, M., et al. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1801.04381 · Cited by 2,255 · see citing works
EIE: efficient inference engine on compressed deep neural network
Han, S., Liu, X., Mao, H., Pu, J., Pedram, A., Horowitz, M. A., & Dally, W. J. (2016) · ACM SIGARCH Computer Architecture News · doi:10.1145/3007787.3001163 · Cited by 1,937 · see citing works
ThiNet: A Filter Level Pruning Method for Deep Neural Network Compression
Luo, J.-H., Wu, J., & Lin, W. (2017) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.541 · Cited by 1,889 · see citing works
ProxylessNAS: Direct Neural Architecture Search on Target Task and HardwarePreprint
Cai, H., Zhu, L., & Han, S. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1812.00332 · Cited by 1,652 · see citing works
The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks
Frankle, J., & Carbin, M. (2018) · arXiv · doi:10.48550/arxiv.1803.03635 · Cited by 1,529 · see citing works
AMC: AutoML for Model Compression and Acceleration on Mobile Devices
He, Y., Lin, J., Liu, Z., Wang, H., Li, L.-J., & Han, S. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01234-2_48 · Cited by 1,400 · see citing works
Rethinking the Value of Network PruningPreprint
Liu, Z., Sun, M., Zhou, T., Huang, G. M., & Darrell, T. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1810.05270 · Cited by 1,225 · see citing works
Soft Filter Pruning for Accelerating Deep Convolutional Neural Networks
He, Y., Kang, G., Dong, X., Fu, Y., & Yang, Y. (2018) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2018/309 · Cited by 1,086 · see citing works
EIE: Efficient Inference Engine on Compressed Deep Neural Network
Han, S., Liu, X., Mao, H., Pu, J., Pedram, A., Horowitz, M. A., & Dally, W. J. (2016) · 2016 ACM/IEEE 43rd Annual International Symposium on Computer Architecture (ISCA) · doi:10.1109/isca.2016.30 · Cited by 1,077 · see citing works
The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks.Preprint
Frankle, J., & Carbin, M. (2019) · DSpace@MIT (Massachusetts Institute of Technology) · Cited by 701 · see citing works
NetAdapt: Platform-Aware Neural Network Adaptation for Mobile Applications
Yang, T.-J., Howard, A., Chen, B., Zhang, X., Go, A., Sandler, M., Sze, V., & Adam, H. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01249-6_18 · Cited by 572 · see citing works
Self-citationUnderstanding and Simplifying One-Shot Architecture Search
Bender, G., Kindermans, P.-J., Zoph, B., Vasudevan, V., & Le, Q. V. (2018) · International Conference on Machine Learning · Cited by 558 · see citing works
SMASH: One-Shot Model Architecture Search through HyperNetworksPreprint
Brock, A., Lim, T., Ritchie, J. M., & Weston, N. J. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1708.05344 · Cited by 500 · see citing works
Discrimination-aware channel pruning for deep neural networks
Zhuang, Z., Tan, M., Zhuang, B., Liu, J., Guo, Y. F., Wu, Q., Huang, J., & Zhu, J. (2018) · neural information processing systems · Cited by 432 · see citing works
Channel Pruning for Accelerating Very Deep Neural NetworksPreprint
He, Y., Zhang, X., & Sun, J. Z. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1707.06168 · Cited by 346 · see citing works
Self-citationMorphNet: Fast & Simple Resource-Constrained Structure Learning of Deep Networks
Gordon, A., Eban, E., Nachum, O., Chen, B., Wu, H., Yang, T.-J., & Choi, E. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00171 · Cited by 345 · see citing works
Learning Efficient Convolutional Networks through Network SlimmingPreprint
Liu, Z., Li, J., Shen, Z., et al. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1708.06519 · Cited by 266 · see citing works
AutoPruner: An end-to-end trainable filter pruning method for efficient deep model inference
Luo, J.-H., & Wu, J. (2020) · Pattern Recognition · doi:10.1016/j.patcog.2020.107461 · Cited by 221 · see citing works
Discrimination-aware Channel Pruning for Deep Neural NetworksPreprint
Zhuang, Z., Tan, M., Zhuang, B., Liu, J., Guo, Y. F., Wu, Q., Huang, J., & Zhu, J. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1810.11809 · Cited by 156 · see citing works
ThiNet: A Filter Level Pruning Method for Deep Neural Network CompressionPreprint
Luo, J.-H., Wu, J., & Lin, W. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1707.06342 · Cited by 105 · see citing works
NetAdapt: Platform-Aware Neural Network Adaptation for Mobile ApplicationsPreprint
Yang, T.-J., Howard, A., Chen, B., et al. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1804.03230 · Cited by 70 · see citing works
MorphNet: Fast & Simple Resource-Constrained Structure Learning of Deep NetworksPreprint
Gordon, A., Eban, E., Nachum, O., et al. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1711.06798 · Cited by 48 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Alajlan, N. N., & Ibrahim, D. M. (2022) · Micromachines · doi:10.3390/mi13060851 · Cited by 224 · see citing works
“Pruning techniques allow removal of redundant structures of network and parameters [15,16].”
MethodBringing ai to edge: from deep learning’s perspective
Liu, D., Kong, H., Luo, X., Liu, W., & Subramaniam, R. (2022) · Neurocomputing · doi:10.1016/j.neucom.2021.04.141 · Cited by 171 · see citing works
“Some recent research [104] [105] [106] [107] strives to empirically or theoretically find an answer for this question.”
BackgroundXu, W., Fang, W., Ding, Y., Zou, M., & Xiong, N. (2021) · IEEE Access · doi:10.1109/access.2021.3063291 · Cited by 112 · see citing works
CondenseNet V2: Sparse Feature Reactivation for Deep Networks
Yang, L., Jiang, H., Cai, R., Wang, Y., Song, S., Huang, G., & Tian, Q. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.00357 · Cited by 71 · see citing works
Self-citationA survey of model compression strategies for object detection
Lyu, Z., Yu, T., Pan, F., Zhang, Y., Luo, J., Zhang, D., Chen, Y., Zhang, B., & Li, G. (2023) · Multimedia Tools and Applications · doi:10.1007/s11042-023-17192-x · Cited by 65 · see citing works
Self-citationFrom Algorithm to Hardware: A Survey on Efficient and Safe Deployment of Deep Neural Networks
Geng, X., Wang, Z., Chen, C., Xu, Q., Xu, K., Jin, C., Gupta, M., Yang, X., Chen, Z., Sabry Aly, M. M., et al. (2025) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2024.3394494 · Cited by 26 · see citing works
- Authors
- Wang, Y., Zhang, X., Xie, L., Zhou, J., Su, H., Zhang, B., & Hu, X.
- Affiliations
- Tsinghua University (CN)
- Ant Group (China) (CN)
- Huawei Technologies (China) (CN)
- Venue
- Proceedings of the AAAI Conference on Artificial Intelligence
- Published
- 2020-04-03
- Volume
- 34
- Issue
- 07
- Pages
- 12273-12280
- ISSN
- 2159-5399 (ISSN-L)
- 2374-3468
- OpenAlex
- W2975721122
- Cited by
- 8
- References
- 36
- Citations by year
- 2024: 2 · 2021: 3 · 2020: 3
- FWCI
- 0.46
- Type
- conference paper
- Language
- en
- Open access
- diamond
- doi.org
- Licence
- www.aaai.org/ (unspecified)
- Locations
- Proceedings of the AAAI Conference on Artificial Intelligence (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- National Natural Science Foundation of China (61621136008, 61571261, 61620106010, 61836014)
- National Key Research and Development Program of China (2017YFA0700904)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Data Classification
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Keywords
- network pruning, overparameterized models, random initialization, model compression, computational complexity reduction, channel pruning, classification models
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
