PreprintCited by 628Advanced Neural Network ApplicationsNeural Networks and ApplicationsMachine Learning and Data Classification

The DOI matches its DataCite record; 25 of the 29 references it cites were found in a registry or an index.

Learning Sparse Neural Networks through $L_0$ Regularization

Open access

Louizos, Christos, Welling, Max, Kingma, Diederik P.·2017·arXiv

DOI 10.48550/arxiv.1712.01312arXiv 1712.01312

Open this paper in CoChat — free account Open PDF

Abstract

We propose a practical method for $L_0$ norm regularization for neural networks: pruning the network during training by encouraging weights to become exactly zero. Such regularization is interesting since (1) it can greatly speed up training and inference, and (2) it can improve generalization. AIC and BIC, well-known model selection criteria, are special cases of $L_0$ regularization. However, since the $L_0$ norm of weights is non-differentiable, we cannot incorporate it directly as a regularization term in the objective function. We propose a solution through the inclusion of a collection of non-negative stochastic gates, which collectively determine which weights to set to zero. We show that, somewhat surprisingly, for certain distributions over the gates, the expected $L_0$ norm of the resulting gated weights is differentiable with respect to the distribution parameters. We further propose the \emph{hard concrete} distribution for the gates, which is obtained by "stretching" a binary concrete distribution and then transforming its samples with a hard-sigmoid. The parameters of the distribution over the gates can then be jointly optimized with the original network parameters. As a result our method allows for straightforward and efficient learning of model structures with stochastic gradient descent and allows for conditional computation in a principled way. We perform various experiments to demonstrate the effectiveness of the resulting approach and regularizer.

Integrity signals

Registry

DataCite · 10.48550/arxiv.1712.01312

Cites

25 of 25 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Claims the paper makes

We couldn’t retrieve this paper’s full text, so its claims and self-check weren’t run. The registry and citation checks still apply.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
Self-citation
Gradient-based learning applied to document recognition
Lecun, Y., Bottou, L., Bengio, Y., & Haffner, P. (1998) · Proceedings of the IEEE · doi:10.1109/5.726791 · Cited by 58,805
Regression shrinkage and selection via the lasso
Tibshirani, R. (1996) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.2517-6161.1996.tb02080.x · Cited by 51,482
Estimating the Dimension of a Model
Schwarz, G. (1978) · The Annals of Statistics · doi:10.1214/aos/1176344136 · Cited by 47,033
Auto-Encoding Variational Bayes
Kingma, D. P., & Welling, M. (2013) · arXiv · doi:10.48550/arxiv.1312.6114 · Cited by 20,067
Self-citation
Information Theory and an Extension of the Maximum Likelihood Principle
Akaike, H. (1998) · Springer series in statistics · doi:10.1007/978-1-4612-1694-0_15 · Cited by 18,636
Simple Statistical Gradient-Following Algorithms for Connectionist Reinforcement Learning
Williams, R. J. (1992) · Machine Learning · doi:10.1007/bf00992696 · Cited by 8,608
Wide Residual Networks
Zagoruyko, S., & Komodakis, N. (2016) · Procedings of the British Machine Vision Conference 2016 · doi:10.5244/c.30.87 · Cited by 7,244
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2015) · arXiv · doi:10.48550/arxiv.1512.03385 · Cited by 4,196
Stochastic Backpropagation and Approximate Inference in Deep Generative ModelsPreprint
Rezende, D. J., Mohamed, S., & Wierstra, D. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1401.4082 · Cited by 3,921

Show all 29 references

Sorted by citations

Items 301–400 of 628

Differentiable Feature Selection, A Reparameterization Approach
Donà, J., & Gallinari, P. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-86523-8_25 · Cited by 9
ESPN: Extremely Sparse Pruned Networks
Cho, M., Joshi, A., & Hegde, C. (2021) · IEEE Data Science Workshop (DSW) · doi:10.1109/dslw51110.2021.9523404 · Cited by 9
Learning to Win Lottery Tickets in BERT Transfer via Task-agnostic Mask Training
Liu, Y., Meng, F., Lin, Z., Fu, P., Cao, Y., Wang, W., & Zhou, J. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.428 · Cited by 9
The emergence of division of labour through decentralized social sanctioning
Yaman, A., Leibo, J., Iacca, G., & Lee, S. W. (2022) · Proceedings of the Royal Society B · doi:10.1098/rspb.2023.1716 · Cited by 11
SPT: Learning to Selectively Insert Prompts for Better Prompt Tuning
Zhu, W., & Tan, M. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.727 · Cited by 9
Sparsification via Compressed Sensing for Automatic Speech Recognition
Zhen, K., Nguyen, H. D., Chang, F.-J., Mouchtaris, A., & Rastrow, A. (2021) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp39728.2021.9413844 · Cited by 8
Batch-Shaped Channel Gated Networks.Preprint
Bejnordi, B. E., Blankevoort, T., & Welling, M. (2019) · arXiv (Cornell University) · Cited by 8
Self-citation
Campfire: Compressible, Regularization-Free, Structured Sparse Training for Hardware AcceleratorsPreprint
Gamboa, N., Kudrolli, K., Dhoot, A., & Pedram, A. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2001.03253 · Cited by 16
A Bregman Learning Framework for Sparse Neural NetworksPreprint
Bungert, L., Roith, T., Tenbrinck, D., & Burger, M. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2105.04319 · Cited by 16
Deciphering the Enigma: A Deep Dive into Understanding and Interpreting LLM OutputsPreprint
Wang, Y. (2023) · doi:10.36227/techrxiv.24085833.v1 · Cited by 9
Equivariant graph convolutional neural networks for the representation of homogenized anisotropic microstructural mechanical response
Patel, R. G., Safta, C., & Jones, R. E. (2024) · Computer Methods in Applied Mechanics and Engineering · doi:10.1016/j.cma.2024.117334 · Cited by 8
Adversary Distillation for One-Shot Attacks on 3D Target Tracking
Wang, Z., Wang, X., Sohel, F. A., Bennamoun, M., Liao, Y., & Yu, J. (2022) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp43922.2022.9747080 · Cited by 7
Learning Adversarially Robust Sparse Networks via Weight Reparameterization
Li, C., Qiu, Q., Zhang, Z., Guo, J., & Cheng, X. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i7.26027 · Cited by 7
Learning Sparse Neural Networks Through Mixture-Distributed Regularization
Huang, C.-T., Chen, J.-C., & Wu, J. (2020) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw50498.2020.00355 · Cited by 7
DL-Reg: A deep learning regularization technique using linear regression
Dialameh, M., Hamzeh, A., Rahmani, H., Dialameh, S., & Kwon, H. (2024) · Expert Systems with Applications · doi:10.1016/j.eswa.2024.123182 · Cited by 12
Energy-Efficient Massive MIMO With Decentralized Precoder Design
Zhang, S., Yin, B., Cheng, Y., Cai, L. X., Zhou, S., Niu, Z., & Shan, H. (2020) · IEEE Transactions on Vehicular Technology · doi:10.1109/tvt.2020.3040619 · Cited by 7
TIMBERT: Toponym Identifier For The Medical Domain Based on BERT
Davari, M., Kosseim, L., & Bui, T. D. (2020) · International Conference on Computational Linguistics · doi:10.18653/v1/2020.coling-main.58 · Cited by 8
SD-Conv: Towards the Parameter-Efficiency of Dynamic Convolution
He, S., Jiang, C., Dong, D., & Liang, D. (2023) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv56688.2023.00639 · Cited by 7
EnCoDe: Enhancing Compressed Deep Learning Models Through Feature - - - Distillation and Informative Sample Selection
Gaire, R. R., Tabrizchi, S., & Roohi, A. (2023) · International Conference on Machine Learning and Applications · doi:10.1109/icmla58977.2023.00093 · Cited by 7
Pruning of Network Filters for Small Dataset
Li, Z., Xu, L., & Zhu, S. (2019) · IEEE Access · doi:10.1109/access.2019.2963080 · Cited by 7
Magnitude and Uncertainty Pruning Criterion for Neural Networks
Ko, V., Oehmcke, S., & Gieseke, F. C. (2019) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata47090.2019.9005692 · Cited by 7
GASL: Guided Attention for Sparsity Learning in Deep Neural NetworksPreprint
Torfi, A., Shirvani, R. A., Soleymani, S., & Nasrabadi, N. M. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1901.01939 · Cited by 9
SSN: Learning Sparse Switchable Normalization via SparsestMaxPreprint
Shao, W., Meng, T., Li, J., Zhang, R., Li, Y., Wang, X., & Luo, P. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1903.03793 · Cited by 7
On Sparsifying Encoder Outputs in Sequence-to-Sequence Models
Zhang, B., Titov, I., & Sennrich, R. (2020) · Zurich Open Repository and Archive (University of Zurich) · doi:10.48550/arxiv.2004.11854 · Cited by 7
Shapley Value as Principled Metric for Structured Network PruningPreprint
Ancona, M., Öztireli, C., & Groß, M. H. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.01795 · Cited by 18
Revisiting Loss Modelling for Unstructured PruningPreprint
Laurent, C., Ballas, C., George, T., Ballas, N., & Vincent, P. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.12279 · Cited by 12
A General Family of Stochastic Proximal Gradient Methods for Deep LearningPreprint
Yun, J., Lozano, A. C., & Yang, E. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2007.07484 · Cited by 7
PareCO: Pareto-aware Channel Optimization for Slimmable Neural NetworksPreprint
Chin, R., Morcos, A. S., & Marculescu, D. (2021) · arXiv (Cornell University) · Cited by 7
Restructuring, Pruning, and Adjustment of Deep Models for Parallel Distributed InferencePreprint
Abdi, A., Rashidi, S., Fekri, F., & Krishna, T. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2008.08289 · Cited by 8
ES-ENAS: Combining Evolution Strategies with Neural Architecture Search at No Extra Cost for Reinforcement Learning.Preprint
Song, X., Choromański, K., Parker-Holder, J., Tang, Y., Peng, D., Jain, D., Gao, W., Pacchiano, A., Sarlós, T., & Yang, Y. (2021) · arXiv (Cornell University) · Cited by 7
Truly Sparse Neural Networks at ScalePreprint
Curci, S., Mocanu, D. C., & Pechenizkiy, M. (2021) · Research Square · doi:10.21203/rs.3.rs-133395/v1 · Cited by 22
Informed Equation LearningPreprint
Werner, M. K., Junginger, A., Hennig, P., & Martius, G. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2105.06331 · Cited by 7
Pruning Randomly Initialized Neural Networks with Iterative RandomizationPreprint
Chijiwa, D., Yamaguchi, S., Ida, Y., Umakoshi, K., & Inoue, T. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.09269 · Cited by 10
Rate-Distortion Theoretic Model Compression: Successive Refinement for PruningPreprint
Isik, B., No, A., & Weissman, T. (2021) · arXiv (Cornell University) · Cited by 7
Physics augmented machine learning discovery of composition-dependent constitutive laws for 3D printed digital materials
Yang, S. J., Levin, M., Padmanabha, G. A., Borshevsky, M., Cohen, O., Seidl, D. T., Jones, R. E., Bouklas, N., & Cohen, N. (2025) · International Journal of Engineering Science · doi:10.1016/j.ijengsci.2025.104381 · Cited by 6
DAC: Data-Free Automatic Acceleration of Convolutional Networks
Li, X., Zhang, S., Jiang, B., Qi, Y., Chuah, M. C., & Bi, N. (2019) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv.2019.00175 · Cited by 6
YOLOv3-VD
Sang, D. V., & Hung, D. V. (2019) · International Symposium on Information and Communication Technology · doi:10.1145/3368926.3369691 · Cited by 6
HFP: Hardware-Aware Filter Pruning for Deep Convolutional Neural Networks Acceleration
Yu, F., Han, C., Wang, P., Huang, R., Huang, X., & Cui, L. (2021) · Proceedings - International Conference on Pattern Recognition/Proceedings/International Conference on Pattern Recognition · doi:10.1109/icpr48806.2021.9412294 · Cited by 6
ACSL: Adaptive correlation-driven sparsity learning for deep neural network compression
He, W. P., Wu, M., & Lam, S.-K. (2021) · Neural Networks · doi:10.1016/j.neunet.2021.09.012 · Cited by 7
Intelligent Edge-Embedded Technologies for Digitising Industry
Vermesan, O., & Nava, M. D. (2022) · doi:10.13052/rp-9788770226103 · Cited by 7
Structure Level Pruning of Efficient Convolutional Neural Networks with Sparse Group LASSO
(2022) · International Journal of Machine Learning and Computing · doi:10.18178/ijmlc.2022.12.5.1111 · Cited by 6
Pruning Filter via Gaussian Distribution Feature for Deep Neural Networks Acceleration
Xu, J., Diao, B., Cui, B., Yang, K., Li, C., & Hong, H. (2022) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn55064.2022.9892389 · Cited by 6
Language Anisotropic Cross-Lingual Model Editing
Yang, X., Hou, Y., Che, W., & Zhang, M. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.343 · Cited by 8
A scalable framework for multi-objective PDE-constrained design of building insulation under uncertainty
Tan, J., & Faghihi, D. (2023) · Computer Methods in Applied Mechanics and Engineering · doi:10.1016/j.cma.2023.116628 · Cited by 6
Efficient fine-texture image retrieval using deep multi-view hashing
Xiang, J., Zhang, N., Pan, R., & Gao, W. (2021) · Computers & Graphics · doi:10.1016/j.cag.2021.10.004 · Cited by 6
Answering POI-recommendation Questions using Tourism Reviews
Contractor, D., Shah, K., Partap, A., Singla, P., & Mausam, M. (2021) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3459637.3482320 · Cited by 7
L0 Regularization based Fine-grained Neural Network Pruning Method
Xie, Q., Li, C., Diao, B., An, Z., & Xu, Y. (2019) · International Conference on Electronics, Computers and Artificial Intelligence (ECAI) · doi:10.1109/ecai46879.2019.9041962 · Cited by 5
Sparse evolutionary Deep Learning with over one million artificial neurons on commodity hardwarePreprint
Liu, S., Mocanu, D. C., Matavalam, A. R. R., Pei, Y., & Pechenizkiy, M. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1901.09181 · Cited by 6
Evolving and understanding sparse deep neural networks using cosine similarityPreprint
Pieterse, J., & Mocanu, C. (2019) · TU/e Research Portal · doi:10.48550/arxiv.1903.07138 · Cited by 6
Speaker-Oriented Latent Structures for Dialogue-Based Relation ExtractionPreprint
Nan, G., Luo, G., Leng, S., Xiao, Y., & Lu, W. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2109.05182 · Cited by 7
Powerpropagation: A sparsity inducing weight reparameterisationPreprint
Schwarz, J., Jayakumar, S. M., Pascanu, R., Latham, P. E., & Teh, Y. W. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.00296 · Cited by 11
Personalized Lightweight Text-to-Speech: Voice Cloning with Adaptive Structured Pruning
Huang, S.-F., Chen, C.-P., Chen, Z., Tsai, Y.-P., & Lee, H.-Y. (2023) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49357.2023.10097178 · Cited by 5
Modeling microbiome-trait associations with taxonomy-adaptive neural networks
Jiang, Y., Aton, M., Zhu, Q., & Lu, Y. Y. (2025) · Microbiome · doi:10.1186/s40168-025-02080-3 · Cited by 9
Fast CNN Pruning via Redundancy-Aware Training
Dong, X., Liu, L., Li, G., Zhao, P., & Feng, X. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-01418-6_1 · Cited by 5
Stochastic weight pruning and the role of regularization in shaping network structure
Ziv, Y., Goldberger, J., & Raviv, T. R. (2021) · Neurocomputing · doi:10.1016/j.neucom.2021.08.007 · Cited by 6
Novel Pruning of Dendritic Neuron Models for Improved System Implementation and Performance
Wen, X., Zhou, M., Luo, X., Huang, L., & Wang, Z. (2021) · Conference proceedings/Conference proceedings - IEEE International Conference on Systems, Man, and Cybernetics · doi:10.1109/smc52423.2021.9659103 · Cited by 5
Pruning during training by network efficacy modeling
Rajpal, M., Zhang, Y., & Low, B. K. H. (2023) · Machine Learning · doi:10.1007/s10994-023-06304-1 · Cited by 6
Sparse Multimodal Vision Transformer for Weakly Supervised Semantic Segmentation
Hanna, J., Mommert, M., & Borth, D. (2023) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw59228.2023.00208 · Cited by 5
Reweighted Alternating Direction Method of Multipliers for DNN weight pruning
Yuan, M., Du, L., Jiang, F., Bai, J., & Chen, G. (2024) · Neural Networks · doi:10.1016/j.neunet.2024.106534 · Cited by 5
Methodical Design and Trimming of Deep Learning Networks: Enhancing External BP Learning with Internal Omnipresent-supervision Training Paradigm
Kung, S., Hou, Z., & Liu, Y. (2019) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2019.8682208 · Cited by 5
Ensembled sparse‐input hierarchical networks for high‐dimensional datasets
Feng, J., & Simon, N. (2022) · Statistical Analysis and Data Mining The ASA Data Science Journal · doi:10.1002/sam.11579 · Cited by 5
Bayesian Nested Neural Networks for Uncertainty Calibration and Adaptive Compression
Cui, Y., Liu, Z., Li, Q., Chan, A. B., & Xue, C. J. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.00242 · Cited by 5
Doping: A technique for Extreme Compression of LSTM Models using Sparse Structured Additive Matrices
Thakker, U., Whatmough, P. N., Liu, Z. G., Mattina, M., & Beu, J. G. (2021) · Proceedings of Machine Learning and Systems · Cited by 5
Dynamic Narrowing of VAE Bottlenecks Using GECO and L0 Regularization
Boom, C. D., Wauthier, S. T., Verbelen, T., & Dhoedt, B. (2021) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn52387.2021.9533671 · Cited by 5
Input-specific Attention Subnetworks for Adversarial Detection
Biju, E., Sriram, A., Kumar, P., & Khapra, M. M. (2022) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2022.findings-acl.4 · Cited by 6
A revisit to MacKay algorithm and its application to deep network compression
Li, C., Mao, Y., Zhang, R., & Huai, J. (2020) · Frontiers of Computer Science · doi:10.1007/s11704-019-8390-z · Cited by 5
Sparsifying Networks via Subdifferential Inclusion
Verma, S., & Pesquet, J.-C. (2021) · HAL · Cited by 5
SSGD: Sparsity-Promoting Stochastic Gradient Descent Algorithm for Unbiased Dnn Pruning
Lee, C.-H., Fedorov, I., Rao, B. D., & Garudadri, H. (2020) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp40776.2020.9054436 · Cited by 5
Adaptive Network Sparsification with Dependent Variational Beta-Bernoulli DropoutPreprint
Lee, J., Kim, S., Yoon, J., Lee, H., Yang, E., & Hwang, S. J. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1805.10896 · Cited by 13
Large Scale Question Answering using Tourism DataPreprint
Contractor, D., Shah, K., Partap, A., Mausam, & Singla, P. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1909.03527 · Cited by 9
Position-based Scaled Gradient for Model Quantization and Sparse Training.Preprint
Kim, J., Yoo, K., & Kwak, N. (2020) · arXiv (Cornell University) · Cited by 5
Fine-Grained Stochastic Architecture SearchPreprint
Chaudhuri, S. R., Eban, E., Li, H., Moroz, M., & Movshovitz-Attias, Y. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.09581 · Cited by 5
Deep Neural Network Training with Frank-WolfePreprint
Pokutta, S., Spiegel, C., & Zimmer, M. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.07243 · Cited by 26
Differentiable Architecture Search Without Training Nor Labels: A Pruning Perspective.Preprint
Zhang, M., Su, S. W., Pan, S., Chang, X., Huang, W., & Haffari, G. (2021) · arXiv (Cornell University) · Cited by 5
An Expectation-Maximization Perspective on Federated LearningPreprint
Louizos, C., Reisser, M., Soriaga, J. B., & Welling, M. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2111.10192 · Cited by 17
Self-citation
Regularized nonlinear regression for simultaneously selecting and estimating key model parameters: Application to head-neck position tracking
Yoon, K., You, H., Wu, W.-Y., Lim, C. Y., Choi, J., Boss, C. J., Ramadan, A. A., Popovich, J. M., Cholewicki, J., Reeves, N. P., et al. (2022) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2022.104974 · Cited by 4
Pruning Pre-trained Language Models Without Fine-Tuning
Jiang, T., Wang, D., Zhuang, F., Xie, R., & Xia, F. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.35 · Cited by 7
Phase-field Models for Lightweight Graph Convolutional Networks
Sahbi, H. (2023) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw59228.2023.00489 · Cited by 4
Variational Degeneration to Structural Refinement: A Unified Framework for Superimposed Image Decomposition
Li, W., Xu, Y., Yang, Y., Ji, H., & Lang, Y. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01121 · Cited by 4
Overcoming sparse datasets with multi-task learning as applied to high entropy alloys
Debnath, A., & Reinhart, W. F. (2025) · Machine Learning Science and Technology · doi:10.1088/2632-2153/adb53c · Cited by 4
Accessing dynamic functional connectivity using l0 -regularized sparse-smooth inverse covariance estimation from fMRI
Zhang, L., Fu, Z., Zhang, W., Huang, G., Liang, Z., Li, L., Biswal, B. B., Calhoun, V. D., & Zhang, Z. (2021) · Neurocomputing · doi:10.1016/j.neucom.2021.02.081 · Cited by 4
Learnable Heterogeneous Convolution: Learning both topology and strength
Zhao, R., Wu, Z., & Zhang, Q. (2021) · Neural Networks · doi:10.1016/j.neunet.2021.03.038 · Cited by 4
Exploration of block-wise dynamic sparseness
Hadifar, A., Deleu, J., Develder, C., & Demeester, T. (2021) · Pattern Recognition Letters · doi:10.1016/j.patrec.2021.08.013 · Cited by 4
PD-Net: Point Dropping Network for Flexible Adversarial Example Generation with $L_{0}$ Regularization
Wang, Z., Wang, X., Sohel, F. A., & Liao, Y. (2021) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn52387.2021.9533701 · Cited by 4
Fast Conditional Network Compression Using Bayesian HyperNetworks
Nguyen, P., Tran, T., Le, K., Gupta, S., Rana, S., Nguyen, D., Nguyen, T. T., Ryan, S., & Venkatesh, S. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-86523-8_20 · Cited by 4
MobilePrune: Neural Network Compression via ℓ0 Sparse Group Lasso on the Mobile System
Shao, Y., Zhao, K., Cao, Z., Peng, Z., Peng, X., Li, P., Wang, Y., & Ma, J. (2022) · Sensors · doi:10.3390/s22114081 · Cited by 6
Pruning deep neural networks generates a sparse, bio-inspired nonlinear controller for insect flight
Zahn, O., Bustamante, J. L., Switzer, C. M., Daniel, T. L., & Kutz, J. N. (2022) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1010512 · Cited by 5
Learning sparse reparameterization with layer-wise continuous sparsification
Wang, X., Huang, Y., Zeng, X., Guo, J., & Chen, Y. (2023) · Knowledge-Based Systems · doi:10.1016/j.knosys.2023.110778 · Cited by 4
Gradient-based Intra-attention Pruning on Pre-trained Language Models
Yang, Z., Cui, Y., Yao, X., & Wang, S. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.156 · Cited by 7
Deep Stochastic Logic Gate Networks
Kim, Y. (2023) · IEEE Access · doi:10.1109/access.2023.3328622 · Cited by 9
Structured Bayesian Federated Learning for Green AI: A Decentralized Model Compression Using Turbo-VBI-Based Approach
Xia, C., Tsang, D. H. K., & Lau, V. K. N. (2023) · IEEE Internet of Things Journal · doi:10.1109/jiot.2023.3335971 · Cited by 4
CRESPR: Modular Sparsification of DNNs to Improve Pruning Performance and Model Interpretability
Kang, T., Ding, W., & Chen, P. (2023) · Neural Networks · doi:10.1016/j.neunet.2023.12.021 · Cited by 4
Group Pruning Using a Bounded- $$\ell _p$$ ℓ p Norm for Group Gating and Regularization
Mummadi, C. K., Genewein, T., Zhang, D., Brox, T., & Fischer, V. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-33676-9_10 · Cited by 4
A Latent Morphology Model for Open-Vocabulary Neural Machine Translation
Ataman, D., Aziz, W. F., & Birch, A. (2020) · Edinburgh Research Explorer (University of Edinburgh) · doi:10.5281/zenodo.3923498 · Cited by 4
Relaxed Multivariate Bernoulli Distribution and Its Applications to Deep Generative Models
Wang, X., & Yin, J. (2020) · Uncertainty in Artificial Intelligence · Cited by 5
Stochastic Model Pruning via Weight Dropping Away and Back
Jia, H., Xiang, X., Fan, D., Huang, M., Sun, C., & He, Y. (2020) · International Conference on Image Processing and Machine Vision (IPMV) · doi:10.1145/3421558.3421559 · Cited by 6
Implementing Convolutional Neural Networks Using Hartley Stochastic Computing With Adaptive Rate Feature Map Compression
Mozafari, S. H., Clark, J. J., Gross, W. J., & Meyer, B. H. (2021) · IEEE Open Journal of Circuits and Systems · doi:10.1109/ojcas.2021.3123899 · Cited by 4
Aligned Weight Regularizers for Pruning Pretrained Neural Networks
Neill, J. O., Dutta, S., & Assem, H. (2022) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2022.findings-acl.267 · Cited by 4
Unleashing the Potential of Sparse DNNs Through Synergistic Hardware-Sparsity Co-Design
Ozen, E., & Orailoğlu, A. (2022) · IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems · doi:10.1109/tcad.2022.3191561 · Cited by 4
Cross Domain Model Compression by Structurally Weight Sharing
Gao, S., Deng, C. D., & Huang, H. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.00918 · Cited by 4

Authors: Louizos, C., Welling, M., & Kingma, D. P.
ORCID: Welling, Max
Venue: arXiv
Publisher: arXiv
Published: 2017
ISSN: 2331-8422 (ISSN-L)
DOI: 10.48550/arxiv.1712.01312
arXiv: 1712.01312
OpenAlex: W2771655537
Cited by: 628
References: 29
Citations by year: 2026: 3 · 2025: 16 · 2024: 31 · 2023: 83 · 2022: 54 · 2021: 189 · 2020: 143 · 2019: 82 · 2018: 27
Type: posted content
Language: en
Open access: green
arxiv.org
Licence: arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations: arXiv (Cornell University) (submittedVersion)
TNO Repository (submittedVersion)
Repository hosted by TU Delft Library (TU Delft) (submittedVersion)
Data Archiving and Networked Services (DANS) (submittedVersion)
arXiv (Cornell University)
Topics: Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
Physical Sciences › Computer Science › Artificial Intelligence › Neural Networks and Applications
Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Data Classification
Keywords: sparse neural networks, L0 regularization, weight pruning, concrete distribution, model selection, AIC and BIC, conditional computation, generalization

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Learning Sparse Neural Networks through $L_0$ Regularization

Abstract

Overview

Integrity signals

Verification

Claims the paper makes

Claims other papers make about this one

Citations

Cites

Cited by

Related

Metadata

Record

About these checks