The DOI matches its DataCite record; 25 of the 29 references it cites were found in a registry or an index.
Learning Sparse Neural Networks through $L_0$ Regularization
Open accessAbstract
We propose a practical method for $L_0$ norm regularization for neural networks: pruning the network during training by encouraging weights to become exactly zero. Such regularization is interesting since (1) it can greatly speed up training and inference, and (2) it can improve generalization. AIC and BIC, well-known model selection criteria, are special cases of $L_0$ regularization. However, since the $L_0$ norm of weights is non-differentiable, we cannot incorporate it directly as a regularization term in the objective function. We propose a solution through the inclusion of a collection of non-negative stochastic gates, which collectively determine which weights to set to zero. We show that, somewhat surprisingly, for certain distributions over the gates, the expected $L_0$ norm of the resulting gated weights is differentiable with respect to the distribution parameters. We further propose the \emph{hard concrete} distribution for the gates, which is obtained by "stretching" a binary concrete distribution and then transforming its samples with a hard-sigmoid. The parameters of the distribution over the gates can then be jointly optimized with the original network parameters. As a result our method allows for straightforward and efficient learning of model structures with stochastic gradient descent and allows for conditional computation in a principled way. We perform various experiments to demonstrate the effectiveness of the resulting approach and regularizer.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Many methods based on the sparse learning (SL) framework have also been proven to be effective [43-47]”
From A survey of model compression strategies for object detection (2023) · cites this paper as [44]
Citations
Cites
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
Self-citationGradient-based learning applied to document recognition
Lecun, Y., Bottou, L., Bengio, Y., & Haffner, P. (1998) · Proceedings of the IEEE · doi:10.1109/5.726791 · Cited by 58,805
Regression shrinkage and selection via the lasso
Tibshirani, R. (1996) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.2517-6161.1996.tb02080.x · Cited by 51,482
Estimating the Dimension of a Model
Schwarz, G. (1978) · The Annals of Statistics · doi:10.1214/aos/1176344136 · Cited by 47,033
Auto-Encoding Variational Bayes
Kingma, D. P., & Welling, M. (2013) · arXiv · doi:10.48550/arxiv.1312.6114 · Cited by 20,067
Self-citationInformation Theory and an Extension of the Maximum Likelihood Principle
Akaike, H. (1998) · Springer series in statistics · doi:10.1007/978-1-4612-1694-0_15 · Cited by 18,636
Simple Statistical Gradient-Following Algorithms for Connectionist Reinforcement Learning
Williams, R. J. (1992) · Machine Learning · doi:10.1007/bf00992696 · Cited by 8,608
Zagoruyko, S., & Komodakis, N. (2016) · Procedings of the British Machine Vision Conference 2016 · doi:10.5244/c.30.87 · Cited by 7,244
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2015) · arXiv · doi:10.48550/arxiv.1512.03385 · Cited by 4,196
Stochastic Backpropagation and Approximate Inference in Deep Generative ModelsPreprint
Rezende, D. J., Mohamed, S., & Wierstra, D. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1401.4082 · Cited by 3,921
Cited by
Sorted by citations
Items 601–628 of 628
(2021) · SciPost Physics · doi:10.21468/scipost.report.3020
On the Compression of Neural Networks Using $\ell_0$-Norm Regularization and Weight Pruning
Oliveira, F. D. D. R., Batista, E. L. O., & Seara, R. (2021) · PubMed · doi:10.48550/arxiv.2109.05075 · Cited by 2
The Evolution of Image Denoising From Model-Driven to Machine Learning
Aetesam, H., Maji, S. K., & Boulanger, J. (2022) · Advances in computational intelligence and robotics book series · doi:10.4018/978-1-7998-8892-5.ch011
Bayesian Deep Structured Semantic Model for Sub-Linear-Time Information Retrieval
Mo, Y., & Scott, S. (2022) · Proceedings · doi:10.1109/icdm54844.2022.00142
Abdullah, Faye, I., & Islam, M. R. (2023) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2023.106122
RetractedInduced Feature Selection by Structured PruningPreprint
Hubens, N., Delvigne, V., Mancaş, M., Gosselin, B. H., Preda, M., & Zaharia, T. B. (2023) · HAL · doi:10.48550/arxiv.2303.10999
Pruning Pre-trained Language Models with Principled Importance and Self-regularization
Ren, S., & Zhu, K. Q. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.573
Hu, X., Liu, X., & Xiao, N. (2023) · Journal of Industrial and Management Optimization · doi:10.3934/jimo.2023149
Adapter Pruning using Tropical Characterization
Bhardwaj, R., Vaidya, T. S., & Poria, S. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.116 · Cited by 2
Simpler neural networks prefer subregular languages
Torres, C., & Futrell, R. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.112
A Closer Look at Transformer Attention for Multilingual Translation
Zhang, J., de Melo, G., Xu, H., & Chen, K. (2023) · Conference on Machine Translation (WMT) · doi:10.18653/v1/2023.wmt-1.45
Information Fusion Channel Pruning for Accelerating Neural Networks
Zhu, Z., Zhai, Y., & Zhang, Y. (2023) · International Conference on Big Data Computing and Communications (BigCom) · doi:10.1109/bigcom61073.2023.00040
Block-wise Pruning for Convolutional Neural Networks
Luo, H., Rao, J., & Zhang, J. (2023) · International Conference on Image Processing, Computer Vision and Machine Learning (ICICML) · doi:10.1109/icicml60161.2023.10424908
Wang, X., She, B., Wu, S.-Q., Yin, D.-L., & Yan, X. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.4844840
Bouayad, A., Akallouch, M., Mahdaouy, A. E., Alami, H., & Berrada, I. (2024) · Journal of Information Security and Applications · doi:10.1016/j.jisa.2024.103891 · Cited by 1
LoRA Meets Foundation Models: Unlocking Efficient Specialization for Scalable AIPreprint
Shun, J. (2025) · doi:10.31224/4513
Super-Fast Rates of Convergence for Neural Networks Classifiers Under the Hard Margin ConditionPreprint
Tepakbong, N., Zhou, D., & Zhou, X. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5240228
Understanding beyond outputs: A novel knowledge distillation method using Schur decomposition
Pan, C., Chen, S.-B., Jiang, B., & Luo, B. (2025) · Neurocomputing · doi:10.1016/j.neucom.2025.130841
Lightweight Vehicle Detection Model Based on Feature Map Similarity Measurement
Yang, P., Yang, X., Lei, J., & Liu, N. (2025) · International Conference on Computer and Communication Systems (ICCCS) · doi:10.1109/icccs65393.2025.11069548
Generative Models Reimagined: Review into Resource-Efficient Training
Teke, A., & Pise, N. N. (2025) · International Conference on Control, Communication and Computing (ICCC) · doi:10.1109/iccc64910.2025.11077170
Farisi, Z. (2025) · International Journal of Advanced Computer Science and Applications · doi:10.14569/ijacsa.2025.0160715
From Networks to Circuits: A Structured Approach to AI Interpretability
Kowalska, B., & Kwaśnicka, H. (2025) · Communications in computer and information science · doi:10.1007/978-3-032-10209-6_26
IoU-based overlapping metric and aggregated loss function for morphological-linear neural networks
Hernández, G., Rodríguez-Molina, A., Gómez, E. Z., Rojas-López, A. G., & Villarreal-Cervantes, M. G. (2026) · PeerJ Computer Science · doi:10.7717/peerj-cs.3797
Zhu, H., Song, J., Zhang, W., Zhu, Q., & Li, K. (2026) · Journal of Physics Conference Series · doi:10.1088/1742-6596/3249/1/012024
Latent Sparsification for Gaussian Splatting Compression in Free-Viewpoint Video
Kim, T., Park, J., & Sim, D. (2026) · Journal of Broadcast Engineering · doi:10.5909/jbe.2026.31.5.1052
SNIP: Single-shot Network Pruning based on Connection Sensitivity
Lee, N., Ajanthan, T., & Torr, P. H. S. (2018) · arXiv · doi:10.48550/arxiv.1810.02340 · Cited by 685
Up or Down? Adaptive Rounding for Post-Training Quantization
Nagel, M., Amjad, R. A., van Baalen, M., Louizos, C., & Blankevoort, T. (2020) · arXiv · doi:10.48550/arxiv.2004.10568 · Cited by 252
Self-citation
Metadata
- Authors
- Louizos, C., Welling, M., & Kingma, D. P.
- ORCID
- Welling, Max
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2017
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 1712.01312
- OpenAlex
- W2771655537
- Cited by
- 628
- References
- 29
- Citations by year
- 2026: 3 · 2025: 16 · 2024: 31 · 2023: 83 · 2022: 54 · 2021: 189 · 2020: 143 · 2019: 82 · 2018: 27
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- TNO Repository (submittedVersion)
- Repository hosted by TU Delft Library (TU Delft) (submittedVersion)
- Data Archiving and Networked Services (DANS) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Neural Networks and Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Data Classification
- Keywords
- sparse neural networks, L0 regularization, weight pruning, concrete distribution, model selection, AIC and BIC, conditional computation, generalization
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
