The DOI matches its DataCite record; 25 of the 29 references it cites were found in a registry or an index.
Learning Sparse Neural Networks through $L_0$ Regularization
Open accessAbstract
We propose a practical method for $L_0$ norm regularization for neural networks: pruning the network during training by encouraging weights to become exactly zero. Such regularization is interesting since (1) it can greatly speed up training and inference, and (2) it can improve generalization. AIC and BIC, well-known model selection criteria, are special cases of $L_0$ regularization. However, since the $L_0$ norm of weights is non-differentiable, we cannot incorporate it directly as a regularization term in the objective function. We propose a solution through the inclusion of a collection of non-negative stochastic gates, which collectively determine which weights to set to zero. We show that, somewhat surprisingly, for certain distributions over the gates, the expected $L_0$ norm of the resulting gated weights is differentiable with respect to the distribution parameters. We further propose the \emph{hard concrete} distribution for the gates, which is obtained by "stretching" a binary concrete distribution and then transforming its samples with a hard-sigmoid. The parameters of the distribution over the gates can then be jointly optimized with the original network parameters. As a result our method allows for straightforward and efficient learning of model structures with stochastic gradient descent and allows for conditional computation in a principled way. We perform various experiments to demonstrate the effectiveness of the resulting approach and regularizer.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Many methods based on the sparse learning (SL) framework have also been proven to be effective [43-47]”
From A survey of model compression strategies for object detection (2023) · cites this paper as [44]
Citations
Cites
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
Self-citationGradient-based learning applied to document recognition
Lecun, Y., Bottou, L., Bengio, Y., & Haffner, P. (1998) · Proceedings of the IEEE · doi:10.1109/5.726791 · Cited by 58,805
Regression shrinkage and selection via the lasso
Tibshirani, R. (1996) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.2517-6161.1996.tb02080.x · Cited by 51,482
Estimating the Dimension of a Model
Schwarz, G. (1978) · The Annals of Statistics · doi:10.1214/aos/1176344136 · Cited by 47,033
Auto-Encoding Variational Bayes
Kingma, D. P., & Welling, M. (2013) · arXiv · doi:10.48550/arxiv.1312.6114 · Cited by 20,067
Self-citationInformation Theory and an Extension of the Maximum Likelihood Principle
Akaike, H. (1998) · Springer series in statistics · doi:10.1007/978-1-4612-1694-0_15 · Cited by 18,636
Simple Statistical Gradient-Following Algorithms for Connectionist Reinforcement Learning
Williams, R. J. (1992) · Machine Learning · doi:10.1007/bf00992696 · Cited by 8,608
Zagoruyko, S., & Komodakis, N. (2016) · Procedings of the British Machine Vision Conference 2016 · doi:10.5244/c.30.87 · Cited by 7,244
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2015) · arXiv · doi:10.48550/arxiv.1512.03385 · Cited by 4,196
Stochastic Backpropagation and Approximate Inference in Deep Generative ModelsPreprint
Rezende, D. J., Mohamed, S., & Wierstra, D. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1401.4082 · Cited by 3,921
Cited by
Sorted by citations
The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks
Frankle, J., & Carbin, M. (2018) · arXiv · doi:10.48550/arxiv.1803.03635 · Cited by 1,529
Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Pruned
Voita, E., Talbot, D., Moiseev, F., Sennrich, R., & Titov, I. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1580 · Cited by 1,146
Importance Estimation for Neural Network Pruning
Molchanov, P., Mallya, A., Tyree, S., Frosio, I., & Kautz, J. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.01152 · Cited by 983
SNAS: Stochastic Neural Architecture Search
Xie, S., Zheng, H., Liu, C., & Lin, L. (2018) · arXiv · doi:10.48550/arxiv.1812.09926 · Cited by 731
Explainability in Graph Neural Networks: A Taxonomic Survey
Yuan, H., Yu, H., Gui, S., & Ji, S. (2022) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2022.3204236 · Cited by 730
A systematic review on overfitting control in shallow and deep neural networks
Bejani, M. M., & Ghatee, M. (2021) · Artificial Intelligence Review · doi:10.1007/s10462-021-09975-1 · Cited by 624
Embracing Change: Continual Learning in Deep Neural Networks
Hadsell, R., Rao, D., Rusu, A. A., & Pascanu, R. (2020) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2020.09.004 · Cited by 467
The State of Sparsity in Deep Neural Networks
Gale, T., Elsen, E., & Hooker, S. (2019) · arXiv · doi:10.48550/arxiv.1902.09574 · Cited by 440
Hoefler, T., Alistarh, D., Ben-Nun, T., Dryden, N., & Peste, A. (2021) · ETH Zurich · doi:10.3929/ethz-b-000519202 · Cited by 423
fastmachinelearning/hls4ml: coris
Vloncar, Summers, S., Duarte, J., Tran, N., Kreis, B., Jngadiub, Ghielmetti, N., Hoang, D., Kreinar, E., Lin, K., et al. (2021) · Zenodo (CERN European Organization for Nuclear Research) · doi:10.5281/zenodo.5680908 · Cited by 375
Compressing BERT: Studying the Effects of Weight Pruning on Transfer Learning
Gordon, M., Duh, K., & Andrews, N. (2020) · Proceedings of the 5th Workshop on Representation Learning for NLP · doi:10.18653/v1/2020.repl4nlp-1.18 · Cited by 303
Parameter-Efficient Transfer Learning with Diff Pruning
Guo, D., Rush, A., & Kim, Y. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.378 · Cited by 220
Movement pruning: Adaptive sparsity by fine-tuning
Sanh, V., Wolf, T., & Rush, A. M. (2020) · arXiv · doi:10.48550/arxiv.2005.07683 · Cited by 218
DocBERT: BERT for Document ClassificationPreprint
Adhikari, A., Ram, A., Tang, R., & Lin, J. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1904.08398 · Cited by 286
Integration of Neural Network-Based Symbolic Regression in Deep Learning for Scientific Discovery
Kim, S., Lu, P. Y., Mukherjee, S., Gilbert, M., Jing, L., Ceperic, V., & Soljacic, M. (2021) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2020.3017010 · Cited by 215
Rigging the Lottery: Making All Tickets WinnersPreprint
Evci, U., Gale, T., Menick, J., Castro, P. S., & Elsen, E. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1911.11134 · Cited by 290
Adaptively Sparse Transformers
Correia, G. M., Niculae, V., & Martins, A. F. T. (2019) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d19-1223 · Cited by 233
Ablation Studies in Artificial Neural NetworksPreprint
Meyes, R., Lu, M., de Puiseau, C. W., & Meisen, T. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1901.08644 · Cited by 430
Adaptive Extreme Edge Computing for Wearable Devices
Covi, E., Donati, E., Liang, X., Kappel, D., Heidari, H., Payvand, M., & Wang, W. J. (2021) · Frontiers in Neuroscience · doi:10.3389/fnins.2021.611300 · Cited by 192
Towards Efficient Model Compression via Learned Global Ranking
Chin, T.-W., Ding, R., Zhang, C., & Marculescu, D. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.00159 · Cited by 182
Sparse GPU Kernels for Deep Learning
Gale, T., Zaharia, M., Young, C., & Elsen, E. (2020) · International Conference for High Performance Computing, Networking, Storage and Analysis (SC) · doi:10.1109/sc41405.2020.00021 · Cited by 213
Network Pruning via Transformable Architecture SearchPreprint
Dong, X., & Yang, Y. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1905.09717 · Cited by 213
Lightweight Deep Learning: An Overview
Wang, C.-H., Huang, K.-Y., Yao, Y., Chen, J.-C., Shuai, H.-H., & Cheng, W.-H. (2022) · IEEE Consumer Electronics Magazine · doi:10.1109/mce.2022.3181759 · Cited by 184
Robust flow control and optimal sensor placement using deep reinforcement learning
Paris, R., Beneddine, S., & Dandois, J. (2021) · Journal of Fluid Mechanics · doi:10.1017/jfm.2020.1170 · Cited by 164
Learning Sparse Sharing Architectures for Multiple Tasks
Sun, T., Shao, Y., Li, X., Liu, P., Yan, H. R., Qiu, X., & Huang, X. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i05.6424 · Cited by 150
Sparse low rank factorization for deep neural network compression
Swaminathan, S., Garg, D., Kannan, R., & Andrès, F. (2020) · Neurocomputing · doi:10.1016/j.neucom.2020.02.035 · Cited by 159
Training with Quantization Noise for Extreme Model Compression
Fan, A., Stock, P., Graham, B., Grave, E., Gribonval, R., Jegou, H., & Joulin, A. (2020) · arXiv · doi:10.48550/arxiv.2004.07320 · Cited by 145
Survey of Low-Resource Machine Translation
Haddow, B., Bawden, R., Barone, A. V. M., Helcl, J., & Birch, A. (2022) · Computational Linguistics · doi:10.1162/coli_a_00446 · Cited by 219
Layer-adaptive sparsity for the Magnitude-based PruningPreprint
Lee, J., Park, S., Mo, S., Ahn, S., & Shin, J. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.07611 · Cited by 201
Stabilizing the Lottery Ticket HypothesisPreprint
Frankle, J., Dziugaite, G. K., Roy, D. M., & Carbin, M. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1903.01611 · Cited by 296
Structured Pruning Learns Compact and Accurate Models
Xia, M., Zhong, Z., & Chen, D. (2022) · Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2022.acl-long.107 · Cited by 136
Data-Free Network Quantization With Adversarial Knowledge Distillation
Choi, Y., Choi, J., El-Khamy, M., & Lee, J. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw50498.2020.00363 · Cited by 135
Compressing Neural Networks using the Variational Information BottleneckPreprint
Dai, B., Zhu, C. G., & Wipf, D. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1802.10399 · Cited by 142
Multi-Component Graph Convolutional Collaborative Filtering
Wang, X., Wang, R., Shi, C., Song, G., & Li, Q. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i04.6094 · Cited by 128
Block Pruning For Faster Transformers
Lagunas, F., Charlaix, E., Sanh, V., & Rush, A. M. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.829 · Cited by 204
Soft Threshold Weight Reparameterization for Learnable SparsityPreprint
Kusupati, A., Ramanujan, V., Somani, R., Wortsman, M., Jain, P., Kakade, S. M., & Farhadi, A. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2002.03231 · Cited by 141
The Stem Cell Hypothesis: Dilemma behind Multi-Task Learning with Transformer Encoders
He, H. W., & Choi, J. D. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.451 · Cited by 133
Dynamic Model Pruning with Feedback
Lin, T., Stich, S. U., Barba, L., Dmitriev, D., & Jaggi, M. (2020) · arXiv · doi:10.48550/arxiv.2006.07253 · Cited by 126
SNR: Sub-Network Routing for Flexible Parameter Sharing in Multi-Task Learning
Ma, J., Zhao, Z., Chen, J., Li, A., Hong, L., & Ed, H. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.3301216 · Cited by 124
DeepCABAC: A Universal Compression Algorithm for Deep Neural Networks
Wiedemann, S., Kirchhoffer, H., Matlage, S., Haase, P., Marban, A., Marinc, T., Neumann, D., Nguyen, T., Schwarz, H., Wiegand, T., et al. (2020) · IEEE Journal of Selected Topics in Signal Processing · doi:10.1109/jstsp.2020.2969554 · Cited by 113
Maximizing information from chemical engineering data sets: Applications to machine learning
Thebelt, A., Wiebe, J., Kronqvist, J., Tsay, C., & Misener, R. (2022) · Chemical Engineering Science · doi:10.1016/j.ces.2022.117469 · Cited by 118
Elsen, E., Dukhan, M., Gale, T., & Simonyan, K. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.01464 · Cited by 121
Interpreting Graph Neural Networks for NLP With Differentiable Edge MaskingPreprint
Schlichtkrull, M., Cao, N. D., & Titov, I. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.00577 · Cited by 194
Ghaoui, L. E., Gu, F., Travacca, B., Askari, A., & Tsai, A. Y. (2021) · SIAM Journal on Mathematics of Data Science · doi:10.1137/20m1358517 · Cited by 149
(2019) · neural information processing systems · Cited by 98
Fast convolutional neural networks on FPGAs with hls4ml
Aarrestad, T. K., Lončar, V., Ghielmetti, N., Maurizio Pierini, B., Summers, S., Ngadiuba, J., Petersson, C., Linander, H., Iiyama, Y., Guglielmo, G. D., et al. (2021) · DSpace@MIT (Massachusetts Institute of Technology) · Cited by 93
What Do Compressed Deep Neural Networks Forget?Preprint
Hooker, S., Courville, A. C., Clark, G., Dauphin, Y., & Frome, A. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1911.05248 · Cited by 233
Layer-refined Graph Convolutional Networks for Recommendation
Zhou, X., Lin, D., Liu, Y., & Miao, C. (2023) · IEEE International Conference on Data Engineering (ICDE) · doi:10.1109/icde55515.2023.00100 · Cited by 100
Structured Pruning of Neural Networks With Budget-Aware Regularization
Lemaire, C., Achkar, A., & Jodoin, P.-M. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00932 · Cited by 90
Neural Pruning via Growing RegularizationPreprint
Wang, H., Qin, C., Zhang, Y., & Fu, Y. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2012.09243 · Cited by 132
Discrete Model Compression With Resource Constraint for Deep Neural Networks
Gao, S., Huang, F., Pei, J., & Huang, H. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.00197 · Cited by 87
A Review of the Gumbel-max Trick and its Extensions for Discrete Stochasticity in Machine Learning
Huijben, I. A. M., Kool, W., Paulus, M. B., & van Sloun, R. J. G. (2022) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2022.3157042 · Cited by 106
Progressive learning: A deep learning framework for continual learning
Fayek, H. M., Cavedon, L., & Wu, H. R. (2020) · Neural Networks · doi:10.1016/j.neunet.2020.05.011 · Cited by 89
On Negative Interference in Multilingual Models: Findings and A Meta-Learning Treatment
Wang, Z., Lipton, Z. C., & Tsvetkov, Y. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.359 · Cited by 116
Understanding Event-Generation Networks via Uncertainties
Bellagente, M., Haußmann, M., Luchmann, M., & Plehn, T. (2022) · SciPost Physics · doi:10.21468/scipostphys.13.1.003 · Cited by 77
Learning Sparse Networks Using Targeted DropoutPreprint
Gomez, A. N., Zhang, I., Kamalakara, S. R., Madaan, D., Swersky, K., Gal, Y., & Hinton, G. E. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1905.13678 · Cited by 112
Detecting Beneficial Feature Interactions for Recommender Systems
Su, Y., Zhang, R., Erfani, S., & Xu, Z. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i5.16561 · Cited by 76
Bayesian Bits: Unifying Quantization and PruningPreprint
van Baalen, M., Louizos, C., Nagel, M., Amjad, R. A., Wang, Y., Blankevoort, T., & Welling, M. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2005.07093 · Cited by 95
Self-citationA Low Effort Approach to Structured CNN Design Using PCA
Garg, I., Panda, P., & Roy, K. (2019) · IEEE Access · doi:10.1109/access.2019.2961960 · Cited by 75
Training Neural Networks with Fixed Sparse MasksPreprint
Sung, Y.-L., Nair, V., & Raffel, C. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2111.09839 · Cited by 84
End-to-end learnable EEG channel selection for deep neural networks with Gumbel-softmax
Strypsteen, T., & Bertrand, A. (2021) · Journal of Neural Engineering · doi:10.1088/1741-2552/ac115d · Cited by 70
Le, V., Quinn, T. P., Tran, T., & Venkatesh, S. (2020) · BMC Genomics · doi:10.1186/s12864-020-6652-7 · Cited by 72
Discovering Neural WiringsPreprint
Wortsman, M., Farhadi, A., & Rastegari, M. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1906.00586 · Cited by 79
Channel Selection Using Gumbel Softmax
Herrmann, C., Bowen, R. S., & Zabih, R. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58583-9_15 · Cited by 69
Structured Pruning of a BERT-based Question Answering ModelPreprint
McCarley, J. S., Chakravarti, R., & Sil, A. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1910.06360 · Cited by 139
HFCM-LSTM: A novel hybrid framework for state-of-health estimation of lithium-ion battery
Gao, M., Bao, Z., Zhu, C., Jiang, J., He, Z., Dong, Z., & Song, Y. (2023) · Energy Reports · doi:10.1016/j.egyr.2023.01.109 · Cited by 67
A survey of model compression strategies for object detection
Lyu, Z., Yu, T., Pan, F., Zhang, Y., Luo, J., Zhang, D., Chen, Y., Zhang, B., & Li, G. (2023) · Multimedia Tools and Applications · doi:10.1007/s11042-023-17192-x · Cited by 65
Relaxed Quantization for Discretized Neural Networks
Louizos, C., Reisser, M., Blankevoort, T., Gavves, E., & Welling, M. (2018) · UvA-DARE (University of Amsterdam) · doi:10.48550/arxiv.1810.01875 · Cited by 102
Self-citationAccelerate CNN via Recursive Bayesian Pruning
Zhou, Y., Zhang, Y., Wang, Y., & Tian, Q. C. (2019) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00340 · Cited by 63
Surpassing early stopping: A novel correlation-based stopping criterion for neural networks
Miseta, T., Fodor, A., & Vathy-Fogarassy, Á. (2023) · Neurocomputing · doi:10.1016/j.neucom.2023.127028 · Cited by 64
A Survey on Model Compression and Acceleration for Pretrained Language Models
Xu, C., & McAuley, J. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i9.26255 · Cited by 85
Structured Graph Convolutional Networks with Stochastic Masks for Recommender Systems
Chen, H., Wang, L., Lin, Y., Yeh, C. M., Wang, F., & Yang, H. (2021) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3404835.3462868 · Cited by 62
SparseRT: Accelerating Unstructured Sparsity on GPUs for Deep Learning InferencePreprint
Wang, Z. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2008.11849 · Cited by 61
DeepHoyer: Learning Sparser Neural Network with Differentiable Scale-Invariant Sparsity MeasuresPreprint
Yang, H., Wen, W., & Li, H. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1908.09979 · Cited by 83
Accelerating Convolutional Neural Networks via Activation Map Compression
Georgiadis, G. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.00725 · Cited by 61
Batch-Shaping for Learning Conditional Channel Gated NetworksPreprint
Bejnordi, B. E., Blankevoort, T., & Welling, M. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1907.06627 · Cited by 58
Self-citationLookahead: A Far-Sighted Alternative of Magnitude-based Pruning
Park, S., Lee, J., Mo, S., & Shin, J. (2020) · arXiv · doi:10.48550/arxiv.2002.04809 · Cited by 58
Li, Z., Wallace, E., Shen, S., Lin, K., Keutzer, K., Klein, D., & Gonzalez, J. E. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2002.11794 · Cited by 107
A novel graph attention model for predicting frequencies of drug–side effects from multi-view data
Zhao, H., Zheng, K., Li, Y., & Wang, J. (2021) · Briefings in Bioinformatics · doi:10.1093/bib/bbab239 · Cited by 58
Winning the Lottery with Continuous SparsificationPreprint
Savarese, P., Silva, H., & Maire, M. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1912.04427 · Cited by 57
How do Decisions Emerge across Layers in Neural Models? Interpretation with Differentiable Masking
De Cao, N., Schlichtkrull, M. S., Aziz, W., & Titov, I. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.262 · Cited by 75
Learning Sparse Neural Networks via Sensitivity-Driven RegularizationPreprint
Tartaglione, E., Lepsøy, S., Fiandrotti, A., & Francini, G. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1810.11764 · Cited by 61
The Lottery Ticket Hypothesis for Object Recognition
Girish, S., Maiya, S. R., Gupta, K., Chen, H., Davis, L. S., & Shrivastava, A. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.00082 · Cited by 57
FedSDG-FS: Efficient and Secure Feature Selection for Vertical Federated Learning
Li, A., Peng, H., Zhang, L., Huang, J., Guo, Q., Yu, H., & Liu, Y. (2023) · IEEE Conference on Computer Communications (INFOCOM) · doi:10.1109/infocom53939.2023.10228895 · Cited by 53
Fuhg, J. N., Jones, R. E., & Bouklas, N. (2024) · Computer Methods in Applied Mechanics and Engineering · doi:10.1016/j.cma.2024.116973 · Cited by 53
Skip-Convolutions for Efficient Video Processing
Habibian, A., Abati, D., Cohen, T., & Bejnordi, B. E. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.00272 · Cited by 52
Resource-Efficient Neural Networks for Embedded SystemsPreprint
Roth, W., Schindler, G., Klein, B., Peharz, R., Tschiatschek, S., Fröning, H., Pernkopf, F., & Ghahramani, Z. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2001.03048 · Cited by 67
Sparse evolutionary deep learning with over one million artificial neurons on commodity hardware
Liu, S., Mocanu, D. C., Matavalam, A. R. R., Pei, Y., & Pechenizkiy, M. (2020) · Neural Computing and Applications · doi:10.1007/s00521-020-05136-7 · Cited by 67
Kalina, K. A., Brummund, J., Sun, W., & Kästner, M. (2025) · Computer Methods in Applied Mechanics and Engineering · doi:10.1016/j.cma.2024.117725 · Cited by 51
WoodFisher: Efficient Second-Order Approximation for Neural Network CompressionPreprint
Singh, S. P., & Alistarh, D. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2004.14340 · Cited by 79
Do We Actually Need Dense Over-Parameterization? In-Time Over-Parameterization in Sparse TrainingPreprint
Liu, S., Yin, L., Mocanu, D. C., & Pechenizkiy, M. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2102.02887 · Cited by 59
Fire Together Wire Together: A Dynamic Pruning Approach with Self-Supervised Mask Prediction
Elkerdawy, S., Elhoushi, M., Zhang, H., & Ray, N. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01213 · Cited by 51
Deep neural networks for genomic prediction do not estimate marker effects
Ubbens, J. R., Parkin, I. A. P., Eynck, C., Stavness, I., & Sharpe, A. (2021) · The Plant Genome · doi:10.1002/tpg2.20147 · Cited by 46
The Difficulty of Training Sparse Neural NetworksPreprint
Evci, U., Pedregosa, F., Gomez, A. N., & Elsen, E. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1906.10732 · Cited by 89
Sparse Training via Boosting Pruning Plasticity with NeuroregenerationPreprint
Liu, S., Chen, T., Chen, X., Atashgahi, Z., Yin, L., Kou, H., Shen, L., Pechenizkiy, M., Wang, Z., & Mocanu, D. C. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.10404 · Cited by 52
Cui, B., Li, Y., & Zhang, Z. M. (2021) · Neurocomputing · doi:10.1016/j.neucom.2021.05.084 · Cited by 47
SSN: Learning Sparse Switchable Normalization via SparsestMax
Shao, W., Meng, T., Li, J., Zhang, R., Li, Y., Wang, X., & Luo, P. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.00053 · Cited by 45
Structured pruning of recurrent neural networks through neuron selection
Wen, L., Zhang, X., Bai, H., & Xu, Z. (2019) · Neural Networks · doi:10.1016/j.neunet.2019.11.018 · Cited by 49
TT-Rec: Tensor Train Compression for Deep Learning Recommendation ModelsPreprint
Yin, C., Acun, B., Liu, X., & Wu, C.-J. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2101.11714 · Cited by 52
Metadata
- Authors
- Louizos, C., Welling, M., & Kingma, D. P.
- ORCID
- Welling, Max
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2017
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 1712.01312
- OpenAlex
- W2771655537
- Cited by
- 628
- References
- 29
- Citations by year
- 2026: 3 · 2025: 16 · 2024: 31 · 2023: 83 · 2022: 54 · 2021: 189 · 2020: 143 · 2019: 82 · 2018: 27
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- TNO Repository (submittedVersion)
- Repository hosted by TU Delft Library (TU Delft) (submittedVersion)
- Data Archiving and Networked Services (DANS) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Neural Networks and Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Data Classification
- Keywords
- sparse neural networks, L0 regularization, weight pruning, concrete distribution, model selection, AIC and BIC, conditional computation, generalization
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
