Learning Sparse Neural Networks through $L_0$ Regularization
Open accessAbstract
We propose a practical method for $L_0$ norm regularization for neural networks: pruning the network during training by encouraging weights to become exactly zero. Such regularization is interesting since (1) it can greatly speed up training and inference, and (2) it can improve generalization. AIC and BIC, well-known model selection criteria, are special cases of $L_0$ regularization. However, since the $L_0$ norm of weights is non-differentiable, we cannot incorporate it directly as a regularization term in the objective function. We propose a solution through the inclusion of a collection of non-negative stochastic gates, which collectively determine which weights to set to zero. We show that, somewhat surprisingly, for certain distributions over the gates, the expected $L_0$ norm of the resulting gated weights is differentiable with respect to the distribution parameters. We further propose the \emph{hard concrete} distribution for the gates, which is obtained by "stretching" a binary concrete distribution and then transforming its samples with a hard-sigmoid. The parameters of the distribution over the gates can then be jointly optimized with the original network parameters. As a result our method allows for straightforward and efficient learning of model structures with stochastic gradient descent and allows for conditional computation in a principled way. We perform various experiments to demonstrate the effectiveness of the resulting approach and regularizer.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Many methods based on the sparse learning (SL) framework have been proven to be effective.
“Many methods based on the sparse learning (SL) framework have also been proven to be effective [43-47]”
From A survey of model compression strategies for object detection (2023) · cites this paper as [44]
Cites
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946 · see citing works
Self-citationGradient-based learning applied to document recognition
Lecun, Y., Bottou, L., Bengio, Y., & Haffner, P. (1998) · Proceedings of the IEEE · doi:10.1109/5.726791 · Cited by 58,805 · see citing works
Regression shrinkage and selection via the lasso
Tibshirani, R. (1996) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.2517-6161.1996.tb02080.x · Cited by 51,482 · see citing works
Estimating the Dimension of a Model
Schwarz, G. (1978) · The Annals of Statistics · doi:10.1214/aos/1176344136 · Cited by 47,033 · see citing works
Auto-Encoding Variational Bayes
Kingma, D. P., & Welling, M. (2013) · arXiv · doi:10.48550/arxiv.1312.6114 · Cited by 20,067 · see citing works
Self-citationInformation Theory and an Extension of the Maximum Likelihood Principle
Akaike, H. (1998) · Springer series in statistics · doi:10.1007/978-1-4612-1694-0_15 · Cited by 18,384 · see citing works
Simple Statistical Gradient-Following Algorithms for Connectionist Reinforcement Learning
Williams, R. J. (1992) · Machine Learning · doi:10.1007/bf00992696 · Cited by 7,936 · see citing works
Zagoruyko, S., & Komodakis, N. (2016) · Procedings of the British Machine Vision Conference 2016 · doi:10.5244/c.30.87 · Cited by 7,244 · see citing works
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2015) · arXiv · doi:10.48550/arxiv.1512.03385 · Cited by 4,196 · see citing works
Stochastic Backpropagation and Approximate Inference in Deep Generative ModelsPreprint
Rezende, D. J., Mohamed, S., & Wierstra, D. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1401.4082 · Cited by 3,679 · see citing works
Categorical Reparameterization with Gumbel-Softmax
Jang, E., Gu, S., & Poole, B. (2016) · arXiv · doi:10.48550/arxiv.1611.01144 · Cited by 3,616 · see citing works
(1989) · Cited by 3,214 · see citing works
Understanding deep learning requires rethinking generalization
Zhang, C., Bengio, S., Hardt, M., Recht, B., & Vinyals, O. (2016) · arXiv · doi:10.48550/arxiv.1611.03530 · Cited by 2,838 · see citing works
Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation
Bengio, Y., Léonard, N., & Courville, A. (2013) · arXiv · doi:10.48550/arxiv.1308.3432 · Cited by 1,910 · see citing works
Learning Structured Sparsity in Deep Neural NetworksPreprint
Wen, W. X., Wu, C., Wang, Y., Chen, Y., & Li, H. H. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1608.03665 · Cited by 1,861 · see citing works
The Concrete Distribution: A Continuous Relaxation of Discrete Random VariablesPreprint
Maddison, C. J., Mnih, A., & Teh, Y. W. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1611.00712 · Cited by 1,771 · see citing works
(2003) · Cited by 1,680 · see citing works
Bayesian Variable Selection in Linear Regression
Mitchell, T. J., & Beauchamp, J. J. (1988) · Journal of the American Statistical Association · doi:10.1080/01621459.1988.10478694 · Cited by 1,427 · see citing works
Approximating the Kullback Leibler Divergence Between Gaussian Mixture Models
Hershey, J. R., & Olsen, P. A. (2007) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2007.366913 · Cited by 1,219 · see citing works
Variational Dropout Sparsifies Deep Neural NetworksPreprint
Molchanov, D., Ashukha, A., & Vetrov, D. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1701.05369 · Cited by 612 · see citing works
Bayesian Compression for Deep LearningPreprint
Louizos, C., Ullrich, K., & Welling, M. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1705.08665 · Cited by 363 · see citing works
Self-citationDiscrete Variational AutoencodersPreprint
Rolfe, J. (2016) · International Conference on Learning Representations · doi:10.48550/arxiv.1609.02200 · Cited by 291 · see citing works
Generative models for discovering sparse distributed representations
Hinton, G. E., & Ghahramani, Z. (1997) · Philosophical Transactions of the Royal Society B Biological Sciences · doi:10.1098/rstb.1997.0101 · Cited by 225 · see citing works
Conditional Computation in Neural Networks for faster modelsPreprint
Bengio, E., Bacon, P., Pineau, J., & Precup, D. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1511.06297 · Cited by 189 · see citing works
A Structured Variational Auto-encoder for Learning Deep Hierarchies of Sparse FeaturesPreprint
Salimans, T. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1602.08734 · Cited by 7 · see citing works
Flagged first, then by citations
4 of the 29 references couldn’t be matched to a known paper.
Cited by
Sorted by citations
The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks
Frankle, J., & Carbin, M. (2018) · arXiv · doi:10.48550/arxiv.1803.03635 · Cited by 1,529 · see citing works
Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Pruned
Voita, E., Talbot, D., Moiseev, F., Sennrich, R., & Titov, I. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1580 · Cited by 1,146 · see citing works
Importance Estimation for Neural Network Pruning
Molchanov, P., Mallya, A., Tyree, S., Frosio, I., & Kautz, J. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.01152 · Cited by 983 · see citing works
SNAS: Stochastic Neural Architecture Search
Xie, S., Zheng, H., Liu, C., & Lin, L. (2018) · arXiv · doi:10.48550/arxiv.1812.09926 · Cited by 731 · see citing works
Explainability in Graph Neural Networks: A Taxonomic Survey
Yuan, H., Yu, H., Gui, S., & Ji, S. (2022) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2022.3204236 · Cited by 640 · see citing works
A systematic review on overfitting control in shallow and deep neural networks
Bejani, M. M., & Ghatee, M. (2021) · Artificial Intelligence Review · doi:10.1007/s10462-021-09975-1 · Cited by 598 · see citing works
Embracing Change: Continual Learning in Deep Neural Networks
Hadsell, R., Rao, D., Rusu, A. A., & Pascanu, R. (2020) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2020.09.004 · Cited by 467 · see citing works
The State of Sparsity in Deep Neural Networks
Gale, T., Elsen, E., & Hooker, S. (2019) · arXiv · doi:10.48550/arxiv.1902.09574 · Cited by 440 · see citing works
Hoefler, T., Alistarh, D., Ben-Nun, T., Dryden, N., & Peste, A. (2021) · ETH Zurich · doi:10.3929/ethz-b-000519202 · Cited by 423 · see citing works
fastmachinelearning/hls4ml: coris
Vloncar, Summers, S., Duarte, J., Tran, N., Kreis, B., Jngadiub, Ghielmetti, N., Hoang, D., Kreinar, E., Lin, K., et al. (2021) · Zenodo (CERN European Organization for Nuclear Research) · doi:10.5281/zenodo.5680908 · Cited by 363 · see citing works
Compressing BERT: Studying the Effects of Weight Pruning on Transfer Learning
Gordon, M., Duh, K., & Andrews, N. (2020) · Proceedings of the 5th Workshop on Representation Learning for NLP · doi:10.18653/v1/2020.repl4nlp-1.18 · Cited by 303 · see citing works
Parameter-Efficient Transfer Learning with Diff Pruning
Guo, D., Rush, A., & Kim, Y. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.378 · Cited by 220 · see citing works
Movement pruning: Adaptive sparsity by fine-tuning
Sanh, V., Wolf, T., & Rush, A. M. (2020) · arXiv · doi:10.48550/arxiv.2005.07683 · Cited by 218 · see citing works
DocBERT: BERT for Document ClassificationPreprint
Adhikari, A., Ram, A., Tang, R., & Lin, J. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1904.08398 · Cited by 286 · see citing works
Integration of Neural Network-Based Symbolic Regression in Deep Learning for Scientific Discovery
Kim, S., Lu, P. Y., Mukherjee, S., Gilbert, M., Jing, L., Ceperic, V., & Soljacic, M. (2021) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2020.3017010 · Cited by 215 · see citing works
Rigging the Lottery: Making All Tickets WinnersPreprint
Evci, U., Gale, T., Menick, J., Castro, P. S., & Elsen, E. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1911.11134 · Cited by 290 · see citing works
Adaptively Sparse Transformers
Correia, G. M., Niculae, V., & Martins, A. F. T. (2019) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d19-1223 · Cited by 211 · see citing works
Ablation Studies in Artificial Neural NetworksPreprint
Meyes, R., Lu, M., de Puiseau, C. W., & Meisen, T. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1901.08644 · Cited by 358 · see citing works
Adaptive Extreme Edge Computing for Wearable Devices
Covi, E., Donati, E., Liang, X., Kappel, D., Heidari, H., Payvand, M., & Wang, W. (2020) · Frontiers in Neuroscience · doi:10.3389/fnins.2021.611300 · Cited by 182 · see citing works
Towards Efficient Model Compression via Learned Global Ranking
Chin, T.-W., Ding, R., Zhang, C., & Marculescu, D. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.00159 · Cited by 180 · see citing works
Sparse GPU Kernels for Deep Learning
Gale, T., Zaharia, M., Young, C., & Elsen, E. (2020) · International Conference for High Performance Computing, Networking, Storage and Analysis (SC) · doi:10.1109/sc41405.2020.00021 · Cited by 183 · see citing works
Network Pruning via Transformable Architecture SearchPreprint
Dong, X., & Yang, Y. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1905.09717 · Cited by 177 · see citing works
Lightweight Deep Learning: An Overview
Wang, C.-H., Huang, K.-Y., Yao, Y., Chen, J.-C., Shuai, H.-H., & Cheng, W.-H. (2022) · IEEE Consumer Electronics Magazine · doi:10.1109/mce.2022.3181759 · Cited by 172 · see citing works
Robust flow control and optimal sensor placement using deep reinforcement learning
Paris, R., Beneddine, S., & Dandois, J. (2021) · Journal of Fluid Mechanics · doi:10.1017/jfm.2020.1170 · Cited by 164 · see citing works
Learning Sparse Sharing Architectures for Multiple Tasks
Sun, T., Shao, Y., Li, X., Liu, P., Yan, H. R., Qiu, X., & Huang, X. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i05.6424 · Cited by 148 · see citing works
Sparse low rank factorization for deep neural network compression
Swaminathan, S., Garg, D., Kannan, R., & Andrès, F. (2020) · Neurocomputing · doi:10.1016/j.neucom.2020.02.035 · Cited by 148 · see citing works
Training with Quantization Noise for Extreme Model Compression
Fan, A., Stock, P., Graham, B., Grave, E., Gribonval, R., Jegou, H., & Joulin, A. (2020) · arXiv · doi:10.48550/arxiv.2004.07320 · Cited by 145 · see citing works
Survey of Low-Resource Machine Translation
Haddow, B., Bawden, R., Barone, A. V. M., Helcl, J., & Birch, A. (2022) · Computational Linguistics · doi:10.1162/coli_a_00446 · Cited by 185 · see citing works
Layer-adaptive sparsity for the Magnitude-based PruningPreprint
Lee, J., Park, S., Mo, S., Ahn, S., & Shin, J. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.07611 · Cited by 201 · see citing works
Stabilizing the Lottery Ticket HypothesisPreprint
Frankle, J., Dziugaite, G. K., Roy, D. M., & Carbin, M. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1903.01611 · Cited by 137 · see citing works
Structured Pruning Learns Compact and Accurate Models
Xia, M., Zhong, Z., & Chen, D. (2022) · Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2022.acl-long.107 · Cited by 136 · see citing works
Data-Free Network Quantization With Adversarial Knowledge Distillation
Choi, Y., Choi, J., El-Khamy, M., & Lee, J. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw50498.2020.00363 · Cited by 135 · see citing works
Compressing Neural Networks using the Variational Information BottleneckPreprint
Dai, B., Zhu, C. G., & Wipf, D. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1802.10399 · Cited by 132 · see citing works
Multi-Component Graph Convolutional Collaborative Filtering
Wang, X., Wang, R., Shi, C., Song, G., & Li, Q. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i04.6094 · Cited by 128 · see citing works
Block Pruning For Faster Transformers
Lagunas, F., Charlaix, E., Sanh, V., & Rush, A. M. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.829 · Cited by 144 · see citing works
Soft Threshold Weight Reparameterization for Learnable SparsityPreprint
Kusupati, A., Ramanujan, V., Somani, R., Wortsman, M., Jain, P., Kakade, S. M., & Farhadi, A. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2002.03231 · Cited by 127 · see citing works
The Stem Cell Hypothesis: Dilemma behind Multi-Task Learning with Transformer Encoders
He, H. W., & Choi, J. D. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.451 · Cited by 130 · see citing works
Dynamic Model Pruning with Feedback
Lin, T., Stich, S. U., Barba, L., Dmitriev, D., & Jaggi, M. (2020) · arXiv · doi:10.48550/arxiv.2006.07253 · Cited by 126 · see citing works
SNR: Sub-Network Routing for Flexible Parameter Sharing in Multi-Task Learning
Ma, J., Zhao, Z., Chen, J., Li, A., Hong, L., & Ed, H. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.3301216 · Cited by 124 · see citing works
DeepCABAC: A Universal Compression Algorithm for Deep Neural Networks
Wiedemann, S., Kirchhoffer, H., Matlage, S., Haase, P., Marban, A., Marinc, T., Neumann, D., Nguyen, T., Schwarz, H., Wiegand, T., et al. (2020) · IEEE Journal of Selected Topics in Signal Processing · doi:10.1109/jstsp.2020.2969554 · Cited by 113 · see citing works
Maximizing information from chemical engineering data sets: Applications to machine learning
Thebelt, A., Wiebe, J., Kronqvist, J., Tsay, C., & Misener, R. (2022) · Chemical Engineering Science · doi:10.1016/j.ces.2022.117469 · Cited by 113 · see citing works
Elsen, E., Dukhan, M., Gale, T., & Simonyan, K. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.01464 · Cited by 111 · see citing works
Interpreting Graph Neural Networks for NLP With Differentiable Edge MaskingPreprint
Schlichtkrull, M., Cao, N. D., & Titov, I. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.00577 · Cited by 158 · see citing works
Ghaoui, L. E., Gu, F., Travacca, B., Askari, A., & Tsai, A. Y. (2021) · SIAM Journal on Mathematics of Data Science · doi:10.1137/20m1358517 · Cited by 112 · see citing works
(2019) · neural information processing systems · Cited by 98 · see citing works
Fast convolutional neural networks on FPGAs with hls4ml
Aarrestad, T. K., Lončar, V., Ghielmetti, N., Maurizio Pierini, B., Summers, S., Ngadiuba, J., Petersson, C., Linander, H., Iiyama, Y., Guglielmo, G. D., et al. (2021) · DSpace@MIT (Massachusetts Institute of Technology) · Cited by 93 · see citing works
What Do Compressed Deep Neural Networks Forget?Preprint
Hooker, S., Courville, A. C., Clark, G., Dauphin, Y., & Frome, A. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1911.05248 · Cited by 125 · see citing works
Layer-refined Graph Convolutional Networks for Recommendation
Zhou, X., Lin, D., Liu, Y., & Miao, C. (2023) · IEEE International Conference on Data Engineering (ICDE) · doi:10.1109/icde55515.2023.00100 · Cited by 91 · see citing works
Structured Pruning of Neural Networks With Budget-Aware Regularization
Lemaire, C., Achkar, A., & Jodoin, P.-M. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00932 · Cited by 90 · see citing works
Neural Pruning via Growing RegularizationPreprint
Wang, H., Qin, C., Zhang, Y., & Fu, Y. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2012.09243 · Cited by 108 · see citing works
Discrete Model Compression With Resource Constraint for Deep Neural Networks
Gao, S., Huang, F., Pei, J., & Huang, H. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.00197 · Cited by 84 · see citing works
A Review of the Gumbel-max Trick and its Extensions for Discrete Stochasticity in Machine Learning
Huijben, I. A. M., Kool, W., Paulus, M. B., & van Sloun, R. J. G. (2022) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2022.3157042 · Cited by 85 · see citing works
Progressive learning: A deep learning framework for continual learning
Fayek, H. M., Cavedon, L., & Wu, H. R. (2020) · Neural Networks · doi:10.1016/j.neunet.2020.05.011 · Cited by 81 · see citing works
On Negative Interference in Multilingual Models: Findings and A Meta-Learning Treatment
Wang, Z., Lipton, Z. C., & Tsvetkov, Y. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.359 · Cited by 84 · see citing works
Understanding Event-Generation Networks via Uncertainties
Bellagente, M., Haußmann, M., Luchmann, M., & Plehn, T. (2022) · SciPost Physics · doi:10.21468/scipostphys.13.1.003 · Cited by 77 · see citing works
Learning Sparse Networks Using Targeted DropoutPreprint
Gomez, A. N., Zhang, I., Kamalakara, S. R., Madaan, D., Swersky, K., Gal, Y., & Hinton, G. E. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1905.13678 · Cited by 85 · see citing works
Detecting Beneficial Feature Interactions for Recommender Systems
Su, Y., Zhang, R., Erfani, S., & Xu, Z. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i5.16561 · Cited by 76 · see citing works
Bayesian Bits: Unifying Quantization and PruningPreprint
van Baalen, M., Louizos, C., Nagel, M., Amjad, R. A., Wang, Y., Blankevoort, T., & Welling, M. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2005.07093 · Cited by 76 · see citing works
Self-citationA Low Effort Approach to Structured CNN Design Using PCA
Garg, I., Panda, P., & Roy, K. (2019) · IEEE Access · doi:10.1109/access.2019.2961960 · Cited by 75 · see citing works
Training Neural Networks with Fixed Sparse MasksPreprint
Sung, Y.-L., Nair, V., & Raffel, C. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2111.09839 · Cited by 74 · see citing works
End-to-end learnable EEG channel selection for deep neural networks with Gumbel-softmax
Strypsteen, T., & Bertrand, A. (2021) · Journal of Neural Engineering · doi:10.1088/1741-2552/ac115d · Cited by 70 · see citing works
Le, V., Quinn, T. P., Tran, T., & Venkatesh, S. (2020) · BMC Genomics · doi:10.1186/s12864-020-6652-7 · Cited by 71 · see citing works
Discovering Neural WiringsPreprint
Wortsman, M., Farhadi, A., & Rastegari, M. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1906.00586 · Cited by 75 · see citing works
Channel Selection Using Gumbel Softmax
Herrmann, C., Bowen, R. S., & Ramin Zabih, A. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58583-9_15 · Cited by 68 · see citing works
Structured Pruning of a BERT-based Question Answering ModelPreprint
McCarley, J. S., Chakravarti, R., & Sil, A. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1910.06360 · Cited by 107 · see citing works
HFCM-LSTM: A novel hybrid framework for state-of-health estimation of lithium-ion battery
Gao, M., Bao, Z., Zhu, C., Jiang, J., He, Z., Dong, Z., & Song, Y. (2023) · Energy Reports · doi:10.1016/j.egyr.2023.01.109 · Cited by 67 · see citing works
A survey of model compression strategies for object detection
Lyu, Z., Yu, T., Pan, F., Zhang, Y., Luo, J., Zhang, D., Chen, Y., Zhang, B., & Li, G. (2023) · Multimedia Tools and Applications · doi:10.1007/s11042-023-17192-x · Cited by 65 · see citing works
Relaxed Quantization for Discretized Neural Networks
Louizos, C., Reisser, M., Blankevoort, T., Gavves, E., & Welling, M. (2018) · UvA-DARE (University of Amsterdam) · doi:10.48550/arxiv.1810.01875 · Cited by 64 · see citing works
Self-citationAccelerate CNN via Recursive Bayesian Pruning
Zhou, Y., Zhang, Y., Wang, Y., & Tian, Q. C. (2019) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00340 · Cited by 64 · see citing works
Surpassing early stopping: A novel correlation-based stopping criterion for neural networks
Miseta, T., Fodor, A., & Vathy-Fogarassy, Á. (2023) · Neurocomputing · doi:10.1016/j.neucom.2023.127028 · Cited by 64 · see citing works
A Survey on Model Compression and Acceleration for Pretrained Language Models
Xu, C., & McAuley, J. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i9.26255 · Cited by 70 · see citing works
Structured Graph Convolutional Networks with Stochastic Masks for Recommender Systems
Chen, H., Wang, L., Lin, Y., Yeh, C. M., Wang, F., & Yang, H. (2021) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3404835.3462868 · Cited by 61 · see citing works
SparseRT: Accelerating Unstructured Sparsity on GPUs for Deep Learning InferencePreprint
Wang, Z. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2008.11849 · Cited by 61 · see citing works
DeepHoyer: Learning Sparser Neural Network with Differentiable Scale-Invariant Sparsity MeasuresPreprint
Yang, H., Wen, W., & Li, H. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1908.09979 · Cited by 59 · see citing works
Accelerating Convolutional Neural Networks via Activation Map Compression
Georgiadis, G. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.00725 · Cited by 58 · see citing works
Batch-Shaping for Learning Conditional Channel Gated NetworksPreprint
Bejnordi, B. E., Blankevoort, T., & Welling, M. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1907.06627 · Cited by 58 · see citing works
Self-citationLookahead: A Far-Sighted Alternative of Magnitude-based Pruning
Park, S., Lee, J., Mo, S., & Shin, J. (2020) · arXiv · doi:10.48550/arxiv.2002.04809 · Cited by 58 · see citing works
Li, Z., Wallace, E., Shen, S., Lin, K., Keutzer, K., Klein, D., & Gonzalez, J. E. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2002.11794 · Cited by 58 · see citing works
A novel graph attention model for predicting frequencies of drug–side effects from multi-view data
Zhao, H., Zheng, K., Li, Y., & Wang, J. (2021) · Briefings in Bioinformatics · doi:10.1093/bib/bbab239 · Cited by 57 · see citing works
Winning the Lottery with Continuous SparsificationPreprint
Savarese, P., Silva, H., & Maire, M. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1912.04427 · Cited by 57 · see citing works
How do Decisions Emerge across Layers in Neural Models? Interpretation with Differentiable Masking
Cao, N. D., Schlichtkrull, M., Aziz, W. F., & Titov, I. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.262 · Cited by 56 · see citing works
Learning Sparse Neural Networks via Sensitivity-Driven RegularizationPreprint
Tartaglione, E., Lepsøy, S., Fiandrotti, A., & Francini, G. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1810.11764 · Cited by 54 · see citing works
The Lottery Ticket Hypothesis for Object Recognition
Girish, S., Maiya, S. R., Gupta, K., Chen, H., Davis, L. S., & Shrivastava, A. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.00082 · Cited by 53 · see citing works
FedSDG-FS: Efficient and Secure Feature Selection for Vertical Federated Learning
Li, A., Peng, H., Zhang, L., Huang, J., Guo, Q., Yu, H., & Liu, Y. (2023) · IEEE Conference on Computer Communications (INFOCOM) · doi:10.1109/infocom53939.2023.10228895 · Cited by 53 · see citing works
Fuhg, J. N., Jones, R. E., & Bouklas, N. (2024) · Computer Methods in Applied Mechanics and Engineering · doi:10.1016/j.cma.2024.116973 · Cited by 53 · see citing works
Skip-Convolutions for Efficient Video Processing
Habibian, A., Abati, D., Cohen, T., & Bejnordi, B. E. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.00272 · Cited by 52 · see citing works
Resource-Efficient Neural Networks for Embedded SystemsPreprint
Roth, W., Schindler, G., Klein, B., Peharz, R., Tschiatschek, S., Fröning, H., Pernkopf, F., & Ghahramani, Z. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2001.03048 · Cited by 57 · see citing works
Sparse evolutionary deep learning with over one million artificial neurons on commodity hardware
Liu, S., Mocanu, D. C., Matavalam, A. R. R., Pei, Y., & Pechenizkiy, M. (2020) · Neural Computing and Applications · doi:10.1007/s00521-020-05136-7 · Cited by 51 · see citing works
Kalina, K. A., Brummund, J., Sun, W., & Kästner, M. (2025) · Computer Methods in Applied Mechanics and Engineering · doi:10.1016/j.cma.2024.117725 · Cited by 51 · see citing works
WoodFisher: Efficient Second-Order Approximation for Neural Network CompressionPreprint
Singh, S. P., & Alistarh, D. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2004.14340 · Cited by 51 · see citing works
Do We Actually Need Dense Over-Parameterization? In-Time Over-Parameterization in Sparse TrainingPreprint
Liu, S., Yin, L., Mocanu, D. C., & Pechenizkiy, M. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2102.02887 · Cited by 49 · see citing works
Fire Together Wire Together: A Dynamic Pruning Approach with Self-Supervised Mask Prediction
Elkerdawy, S., Elhoushi, M., Zhang, H., & Ray, N. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01213 · Cited by 50 · see citing works
Deep neural networks for genomic prediction do not estimate marker effects
Ubbens, J. R., Parkin, I. A. P., Eynck, C., Stavness, I., & Sharpe, A. (2021) · The Plant Genome · doi:10.1002/tpg2.20147 · Cited by 46 · see citing works
The Difficulty of Training Sparse Neural NetworksPreprint
Evci, U., Pedregosa, F., Gomez, A. N., & Elsen, E. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1906.10732 · Cited by 47 · see citing works
Sparse Training via Boosting Pruning Plasticity with NeuroregenerationPreprint
Liu, S., Chen, T., Chen, X., Atashgahi, Z., Yin, L., Kou, H., Shen, L., Pechenizkiy, M., Wang, Z., & Mocanu, D. C. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.10404 · Cited by 47 · see citing works
Cui, B., Li, Y., & Zhang, Z. M. (2021) · Neurocomputing · doi:10.1016/j.neucom.2021.05.084 · Cited by 46 · see citing works
SSN: Learning Sparse Switchable Normalization via SparsestMax
Shao, W., Meng, T., Li, J., Zhang, R., Li, Y., Wang, X., & Luo, P. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.00053 · Cited by 45 · see citing works
Structured pruning of recurrent neural networks through neuron selection
Wen, L., Zhang, X., Bai, H., & Xu, Z. (2019) · Neural Networks · doi:10.1016/j.neunet.2019.11.018 · Cited by 45 · see citing works
TT-Rec: Tensor Train Compression for Deep Learning Recommendation ModelsPreprint
Yin, C., Acun, B., Liu, X., & Wu, C.-J. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2101.11714 · Cited by 45 · see citing works
- Authors
- Louizos, C., Welling, M., & Kingma, D. P.
- ORCID
- Welling, Max
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2017
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 1712.01312
- OpenAlex
- W2771655537
- Cited by
- 628
- References
- 29
- Citations by year
- 2026: 3 · 2025: 16 · 2024: 31 · 2023: 83 · 2022: 54 · 2021: 189 · 2020: 143 · 2019: 82 · 2018: 27
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- TNO Repository (submittedVersion)
- Repository hosted by TU Delft Library (TU Delft) (submittedVersion)
- Data Archiving and Networked Services (DANS) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Neural Networks and Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Data Classification
- Keywords
- sparse neural networks, L0 regularization, weight pruning, concrete distribution, model selection, AIC and BIC, conditional computation, generalization
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
