The DOI matches its DataCite record; 25 of the 29 references it cites were found in a registry or an index.
Learning Sparse Neural Networks through $L_0$ Regularization
Open accessAbstract
We propose a practical method for $L_0$ norm regularization for neural networks: pruning the network during training by encouraging weights to become exactly zero. Such regularization is interesting since (1) it can greatly speed up training and inference, and (2) it can improve generalization. AIC and BIC, well-known model selection criteria, are special cases of $L_0$ regularization. However, since the $L_0$ norm of weights is non-differentiable, we cannot incorporate it directly as a regularization term in the objective function. We propose a solution through the inclusion of a collection of non-negative stochastic gates, which collectively determine which weights to set to zero. We show that, somewhat surprisingly, for certain distributions over the gates, the expected $L_0$ norm of the resulting gated weights is differentiable with respect to the distribution parameters. We further propose the \emph{hard concrete} distribution for the gates, which is obtained by "stretching" a binary concrete distribution and then transforming its samples with a hard-sigmoid. The parameters of the distribution over the gates can then be jointly optimized with the original network parameters. As a result our method allows for straightforward and efficient learning of model structures with stochastic gradient descent and allows for conditional computation in a principled way. We perform various experiments to demonstrate the effectiveness of the resulting approach and regularizer.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Many methods based on the sparse learning (SL) framework have also been proven to be effective [43-47]”
From A survey of model compression strategies for object detection (2023) · cites this paper as [44]
Citations
Cites
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
Self-citationGradient-based learning applied to document recognition
Lecun, Y., Bottou, L., Bengio, Y., & Haffner, P. (1998) · Proceedings of the IEEE · doi:10.1109/5.726791 · Cited by 58,805
Regression shrinkage and selection via the lasso
Tibshirani, R. (1996) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.2517-6161.1996.tb02080.x · Cited by 51,482
Estimating the Dimension of a Model
Schwarz, G. (1978) · The Annals of Statistics · doi:10.1214/aos/1176344136 · Cited by 47,033
Auto-Encoding Variational Bayes
Kingma, D. P., & Welling, M. (2013) · arXiv · doi:10.48550/arxiv.1312.6114 · Cited by 20,067
Self-citationInformation Theory and an Extension of the Maximum Likelihood Principle
Akaike, H. (1998) · Springer series in statistics · doi:10.1007/978-1-4612-1694-0_15 · Cited by 18,636
Simple Statistical Gradient-Following Algorithms for Connectionist Reinforcement Learning
Williams, R. J. (1992) · Machine Learning · doi:10.1007/bf00992696 · Cited by 8,608
Zagoruyko, S., & Komodakis, N. (2016) · Procedings of the British Machine Vision Conference 2016 · doi:10.5244/c.30.87 · Cited by 7,244
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2015) · arXiv · doi:10.48550/arxiv.1512.03385 · Cited by 4,196
Stochastic Backpropagation and Approximate Inference in Deep Generative ModelsPreprint
Rezende, D. J., Mohamed, S., & Wierstra, D. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1401.4082 · Cited by 3,921
Cited by
Sorted by citations
Items 101–200 of 628
Neuromorphic deep spiking neural networks for seizure detection
Yang, Y., Eshraghian, J. K., Truong, N. D., Nikpour, A., & Kavehei, O. (2023) · Neuromorphic Computing and Engineering · doi:10.1088/2634-4386/acbab8 · Cited by 43
Progressive Skeletonization: Trimming more fat from a network at initializationPreprint
de Jorge, P., Sanyal, A., Behl, H. S., Torr, P. H. S., Rogez, G., & Dokania, P. K. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.09081 · Cited by 42
Enhancing Monotonic Multihead Attention for Streaming ASR
Inaguma, H., Mimura, M., & Kawahara, T. (2020) · Interspeech (USB) · doi:10.21437/interspeech.2020-1780 · Cited by 40
Learning Accurate and Interpretable Decision Rule Sets from Neural Networks
Qiao, L., Wang, W., & Lin, B. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i5.16555 · Cited by 41
Effective Sparsification of Neural Networks with Global Sparsity Constraint
Zhou, X. Y., Zhang, W., Xu, H., & Zhang, T. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.00360 · Cited by 45
Pruning Algorithms to Accelerate Convolutional Neural Networks for Edge Applications: A SurveyPreprint
Liu, J., Tripathi, S., Kurup, U., & Shah, M. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2005.04275 · Cited by 41
Voghoei, S., Tonekaboni, N. H., Wallace, J. G., & Arabnia, H. R. (2018) · International Conference on Computational Science and Computational Intelligence (CSCI) · doi:10.1109/csci46756.2018.00177 · Cited by 45
SAVE: Sparsity-Aware Vector Engine for Accelerating DNN Training and Inference on CPUs
Gong, Z., Ji, H., Fletcher, C. W., Hughes, C. J., Baghsorkhi, S. S., & Torrellas, J. (2020) · IEEE/ACM International Symposium on Microarchitecture (MICRO) · doi:10.1109/micro50266.2020.00070 · Cited by 40
Deep Learning and Symbolic Regression for Discovering Parametric Equations
Zhang, M., Kim, S., Lu, P. Y., & Marin Soljacic, A. (2023) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2023.3297978 · Cited by 48
ChipNet: Budget-Aware Pruning with Heaviside Continuous ApproximationsPreprint
Tiwari, R., Bamba, U., Chavan, A., & Gupta, D. K. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2102.07156 · Cited by 45
Structured Pruning of Self-Supervised Pre-Trained Models for Speech Recognition and Understanding
Peng, Y., Kim, K., Wu, F., Sridhar, P., & Watanabe, S. (2023) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49357.2023.10095780 · Cited by 38
GDP: Stabilized Neural Network Pruning via Gates with Differentiable Polarization
Guo, Y., Yuan, H., Tan, J., Wang, Z., Yang, S., & Liu, J. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00519 · Cited by 42
AdapterBias: Parameter-efficient Token-dependent Representation Shift for Adapters in NLP Tasks
Fu, C.-L., Chen, Z.-C., Lee, Y.-R., & Lee, H.-Y. (2022) · Findings of the Association for Computational Linguistics: NAACL · doi:10.18653/v1/2022.findings-naacl.199 · Cited by 51
Neural Network Compression Framework for Fast Model Inference
Kozlov, A., Lazarevich, I. A., Shamporov, V. A., Lyalyushkin, N., & Gorbachev, Y. (2021) · Lecture notes in networks and systems · doi:10.1007/978-3-030-80129-8_17 · Cited by 44
Efficient and effective training of sparse recurrent neural networks
Liu, S., Ni’mah, I., Menkovski, V., Mocanu, D. C., & Pechenizkiy, M. (2021) · Neural Computing and Applications · doi:10.1007/s00521-021-05727-y · Cited by 40
Delving in the loss landscape to embed robust watermarks into neural networks
Tartaglione, E., Grangetto, M., Cavagnino, D., & Botta, M. (2021) · Proceedings - International Conference on Pattern Recognition/Proceedings/International Conference on Pattern Recognition · doi:10.1109/icpr48806.2021.9413062 · Cited by 38
Extracting the Auditory Attention in a Dual-Speaker Scenario From EEG Using a Joint CNN-LSTM Model
Kuruvila, I., Muncke, J., Fischer, E., & Hoppe, U. (2021) · Frontiers in Physiology · doi:10.3389/fphys.2021.700655 · Cited by 45
Tsang, M., Liu, H., Purushotham, S., Murali, P., & Liu, Y. (2018) · neural information processing systems · Cited by 36
Multi-objective Pruning for CNNs Using Genetic Algorithm
Yang, C., An, Z., Li, C., Diao, B., & Xu, Y. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-30484-3_25 · Cited by 39
Searching to Sparsify Tensor Decomposition for N-ary Relational Data
Di, S., Yao, Q., & Chen, L. (2021) · International World Wide Web Conference (WWW) · doi:10.1145/3442381.3449853 · Cited by 37
Accelerated Sparse Neural Training: A Provable and Efficient Method to Find N:M Transposable MasksPreprint
Hubara, I., Chmiel, B., Island, M., Banner, R., Naor, J. S., & Soudry, D. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2102.08124 · Cited by 46
Interpretable Neural Predictions with Differentiable Binary Variables
Bastings, J., Aziz, W. F., & Titov, I. (2019) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p19-1284 · Cited by 151
Orders Are Unwanted: Dynamic Deep Graph Convolutional Network for Personality Detection
Yang, T., Deng, J., Quan, X., & Wang, Q. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i11.26627 · Cited by 39
Adaptive Feature Selection for End-to-End Speech Translation
Zhang, B., Titov, I., Haddow, B., & Sennrich, R. (2020) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2020.findings-emnlp.230 · Cited by 33
Evaluating Lottery Tickets Under Distributional Shifts
Desai, S., Zhan, H., & Aly, A. (2019) · Workshop on Deep Learning Approaches for Low-Resource NLP (DeepLo) · doi:10.18653/v1/d19-6117 · Cited by 39
Distributed Pruning Towards Tiny Neural Networks in Federated Learning
Huang, H., Zhang, L., Sun, C., Fang, R., Yuan, X., & Wu, D. O. (2023) · IEEE International Conference on Distributed Computing Systems (ICDCS) · doi:10.1109/icdcs57875.2023.00036 · Cited by 32
Aketi, S. A., Roy, S., Raghunathan, A., & Roy, K. (2020) · IEEE Access · doi:10.1109/access.2020.3024992 · Cited by 33
How fine can fine-tuning be? Learning efficient language modelsPreprint
Radiya-Dixit, E., & Wang, X. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2004.14129 · Cited by 36
Growing Efficient Deep Networks by Structured Continuous SparsificationPreprint
Yuan, X., Savarese, P., & Maire, M. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2007.15353 · Cited by 36
Sparse Weight Activation TrainingPreprint
Raihan, A., & Aamodt, T. M. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2001.01969 · Cited by 40
Polyconvex neural network models of thermoelasticity
Fuhg, J. N., Jadoon, A. A., Weeger, O., Seidl, D. T., & Jones, R. E. (2024) · Journal of the Mechanics and Physics of Solids · doi:10.1016/j.jmps.2024.105837 · Cited by 30
To Filter Prune, or to Layer Prune, That Is the Question
Elkerdawy, S., Elhoushi, M., Singh, A. K., Zhang, H., & Ray, N. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-69535-4_45 · Cited by 31
Rethinking Weight Decay for Efficient Neural Network Pruning
Tessier, H., Gripon, V., Léonardon, M., Arzel, M., Hannagan, T., & Bertrand, D. (2022) · Journal of Imaging · doi:10.3390/jimaging8030064 · Cited by 36
Compressing RNNs for IoT devices by 15-38x using Kronecker Products
Thakker, U., Beu, J., Gope, D., Zhou, C., Fedorov, I., Dasika, G., & Mattina, M. (2019) · arXiv · doi:10.48550/arxiv.1906.02876 · Cited by 30
Pruning Convolutional Neural Networks with Self-SupervisionPreprint
Caron, M., Morcos, A. S., Bojanowski, P., Mairal, J., & Joulin, A. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2001.03554 · Cited by 32
Sanity-Checking Pruning Methods: Random Tickets can Win the JackpotPreprint
Su, J., Chen, Y., Cai, T., Wu, T., Gao, R., Wang, L., & Lee, J. D. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2009.11094 · Cited by 30
Quantisation and Pruning for Neural Network Compression and Regularisation
Paupamah, K., James, S. D., & Klein, R. (2020) · 2020 International SAUPEC/RobMech/PRASA Conference · doi:10.1109/saupec/robmech/prasa48453.2020.9041096 · Cited by 31
LOss-Based SensiTivity rEgulaRization: Towards deep sparse neural networks
Tartaglione, E., Bragagnolo, A., Fiandrotti, A., & Grangetto, M. (2021) · Neural Networks · doi:10.1016/j.neunet.2021.11.029 · Cited by 33
Interpreting Neural Networks as Quantitative Argumentation Frameworks
Potyka, N. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i7.16801 · Cited by 34
CUP: Cluster Pruning for Compressing Deep Neural Networks
Duggal, R., Xiao, C., Vuduc, R. W., Chau, D. H., & Sun, J. (2021) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata52589.2021.9671980 · Cited by 33
Wang, T., Zhou, W., Zeng, Y., & Zhang, X. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.873 · Cited by 67
SEALion: a Framework for Neural Network Inference on Encrypted DataPreprint
van Elsloo, T., Patrini, G., & Ivey-Law, H. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1904.12840 · Cited by 53
NViT: Vision Transformer Compression and Parameter Redistribution.Preprint
Yang, H., Yin, H., Molchanov, P. A., Li, H., & Kautz, J. (2021) · arXiv (Cornell University) · Cited by 28
When deep learning meets watermarking: A survey of application, attacks and defenses
Chen, H., Liu, C., Zhu, T., & Zhou, W. (2024) · Computer Standards & Interfaces · doi:10.1016/j.csi.2023.103830 · Cited by 28
Contrastive learning of graphs under label noise
Li, X., Li, Q., Li, D., Qian, H., & Wang, J. (2024) · Neural Networks · doi:10.1016/j.neunet.2024.106113 · Cited by 27
Compressing Deep Neural Networks With Sparse Matrix Factorization
Wu, K., Guo, Y., & Zhang, C. (2019) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2019.2946636 · Cited by 29
AMR Parsing with Latent Structural Information
Zhou, Q., Zhang, Y., Ji, D., & Tang, H. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.397 · Cited by 25
SASL: Saliency-Adaptive Sparsity Learning for Neural Network Acceleration
Shi, J., Xu, J., Tasaka, K., & Chen, Z. (2020) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2020.3013170 · Cited by 29
Zeng, Y., Mai, S., & Hu, H. (2021) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2021.findings-emnlp.109 · Cited by 31
Detecting Arbitrary Order Beneficial Feature Interactions for Recommender Systems
Su, Y., Zhao, Y., Erfani, S., Gan, J., & Zhang, R. (2022) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3534678.3539238 · Cited by 29
STRAIT: Self-Test and Self-Recovery for AI Accelerator
Lee, H., Kim, J., Park, J., & Kang, S. (2023) · IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems · doi:10.1109/tcad.2023.3236875 · Cited by 27
Entropy-Constrained Training of Deep Neural Networks
Wiedemann, S. M., Marbán, A., Müller, K., & Samek, W. (2019) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn.2019.8852119 · Cited by 27
Exploring Structural Sparsity in CNN via Selective Penalty
Zhao, M., Peng, J., Yu, S., Liu, L., & Wu, N. (2021) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2021.3071532 · Cited by 28
PoGO-Net: Pose Graph Optimization with Graph Neural Networks
Li, X., & Ling, H. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00584 · Cited by 26
Harmonious Coexistence of Structured Weight Pruning and Ternarization for Deep Neural Networks
Yang, L., He, Z., & Fan, D. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i04.6138 · Cited by 26
ShrinkTeaNet: Million-scale Lightweight Face Recognition via Shrinking Teacher-Student NetworksPreprint
Duong, C. N., Luu, K., Quach, K. G., & Le, N. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1905.10620 · Cited by 25
Theory-Inspired Path-Regularized Differential Network Architecture SearchPreprint
Zhou, P., Xiong, C., Socher, R., & Hoi, S. C. H. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.16537 · Cited by 26
Feature Map Transform Coding for Energy-Efficient CNN Inference
Chmiel, B., Baskin, C., Zheltonozhskii, E., Banner, R., Yermolin, Y., Karbachevsky, A., Bronstein, A. M., & Mendelson, A. (2020) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn48605.2020.9206968 · Cited by 26
The Low-Resource Double Bind: An Empirical Study of Pruning for Low-Resource Machine Translation
Ahia, O., Kreutzer, J., & Hooker, S. (2021) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2021.findings-emnlp.282 · Cited by 41
Can Subnetwork Structure be the Key to Out-of-Distribution Generalization?Preprint
Zhang, D., Ahuja, K., Xu, Y., Wang, Y., & Courville, A. C. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.02890 · Cited by 34
Yun, S., & Wong, A. M. (2021) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw53098.2021.00277 · Cited by 26
End-to-End Supermask Pruning: Learning to Prune Image Captioning Models
Tan, J. H., Chan, C. S., & Chuah, J. H. (2021) · Pattern Recognition · doi:10.1016/j.patcog.2021.108366 · Cited by 22
Illuminati: Towards Explaining Graph Neural Networks for Cybersecurity Analysis
He, H., Ji, Y., & Huang, H. H. (2022) · IEEE European Symposium on Security and Privacy · doi:10.1109/eurosp53844.2022.00013 · Cited by 23
Know what you don't need: Single-Shot Meta-Pruning for attention heads
Zhang, Z., Qi, F., Liu, Z., Liu, Q., & Sun, M. (2021) · AI Open · doi:10.1016/j.aiopen.2021.05.003 · Cited by 23
Simple and Effective Stochastic Neural Networks
Yu, T., Yang, Y., Li, D., Hospedales, T. M., & Xiang, T. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i4.16436 · Cited by 24
Automatic Pruning for Quantized Neural Networks
Guerra, L., & Drummond, T. W. (2021) · International Conference on Digital Image Computing: Techniques and Applications (DICTA) · doi:10.1109/dicta52665.2021.9647074 · Cited by 22
A brain-inspired algorithm for training highly sparse neural networks
Atashgahi, Z., Pieterse, J., Liu, S., Mocanu, D. C., Veldhuis, R. N. J., & Pechenizkiy, M. (2022) · Machine Learning · doi:10.1007/s10994-022-06266-w · Cited by 33
From Dense to Sparse: Contrastive Pruning for Better Pre-trained Language Model Compression
Xu, R., Luo, F., Wang, C., Chang, B., Huang, J. G., Huang, S., & Huang, F. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i10.21408 · Cited by 29
FLOPs as a Direct Optimization Objective for Learning Sparse Neural NetworksPreprint
Tang, R., Adhikari, A., & Lin, J. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1811.03060 · Cited by 26
Learned Threshold PruningPreprint
Azarian, K., Bhalgat, Y. S., Lee, J., & Blankevoort, T. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2003.00075 · Cited by 22
Holistic Filter Pruning for Efficient Deep Neural Networks
Enderich, L., Timm, F., & Burgard, W. (2021) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv48630.2021.00264 · Cited by 23
Liu, S., Mocanu, D. C., Pei, Y., & Pechenizkiy, M. (2021) · TU/e Research Portal · doi:10.48550/arxiv.2101.09048 · Cited by 21
ZeroBN: Learning Compact Neural Networks For Latency-Critical Edge Systems
Huai, S., Zhang, L., Liu, D., Liu, W., & Subramaniam, R. (2021) · ACM/IEEE Design Automation Conference (DAC) · doi:10.1109/dac18074.2021.9586309 · Cited by 21
Reusing Deep Neural Network Models through Model Re-engineering
Qi, B., Sun, H., Gao, X. Y., Zhang, H., Li, Z., & Liu, X. (2023) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse48619.2023.00090 · Cited by 24
Sparse Transfer Learning via Winning Lottery TicketsPreprint
Mehta, R. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1905.07785 · Cited by 21
Cyclical Pruning for Sparse Neural Networks
Srinivas, S., Kuzmin, A., Nagel, M., van Baalen, M., Skliar, A., & Blankevoort, T. (2022) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw56347.2022.00312 · Cited by 21
Sparse deep neural networks on imaging genetics for schizophrenia case–control classification
Chen, J., Li, X., Calhoun, V. D., Turner, J. A., van Erp, T. G. M., Wang, L., Andreassen, O. A., Agartz, I., Westlye, L. T., Jönsson, E. G., et al. (2021) · Human Brain Mapping · doi:10.1002/hbm.25387 · Cited by 19
A Semismooth Newton Algorithm for High-Dimensional Nonconvex Sparse Learning
Shi, Y., Huang, J., Jiao, Y., & Yang, Q. (2019) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2019.2935001 · Cited by 20
Pruning via Iterative Ranking of Sensitivity StatisticsPreprint
Verdenius, S., Stol, M., & Forré, P. D. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.00896 · Cited by 38
Parameter-Efficient Transfer Learning with Diff Pruning
Guo, D., Rush, A. M., & Kim, Y. (2020) · arXiv · doi:10.48550/arxiv.2012.07463 · Cited by 20
Adaptative Inference Cost With Convolutional Neural Mixture Models
Ruiz, A., & Verbeek, J. J. (2019) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00196 · Cited by 18
Graph Neural Networks for Particle Tracking and Reconstruction
Duarte, J. M., & Vlimant, J.-R. (2022) · WORLD SCIENTIFIC eBooks · doi:10.1142/9789811234026_0012 · Cited by 80
SeReNe: Sensitivity-Based Regularization of Neurons for Structured Sparsity in Neural Networks
Tartaglione, E., Bragagnolo, A., Odierna, F., Fiandrotti, A., & Grangetto, M. (2021) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2021.3084527 · Cited by 22
Low-Complexity Probing via Finding Subnetworks
Cao, S., Sanh, V., & Rush, A. M. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.74 · Cited by 51
Nonconvex Sparse Regularization for Deep Neural Networks and Its Optimality
Ohn, I., & Kim, Y. (2021) · Neural Computation · doi:10.1162/neco_a_01457 · Cited by 22
Robust Lottery Tickets for Pre-trained Language Models
Zheng, R., Rong, B., Zhou, Y., Liang, D., Wang, S., Wu, W., Gui, T., Zhang, Q., & Huang, X. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.157 · Cited by 40
FLightNNs: Lightweight Quantized Deep Neural Networks for Fast and Accurate Inference
Ding, R., Liu, Z., Chin, T.-W., Marculescu, D., & Blanton, R. D. (. (2019) · Proceedings of the 56th Annual Design Automation Conference 2019 · doi:10.1145/3316781.3317828 · Cited by 19
CURATING: A multi-objective based pruning technique for CNNs
Pattanayak, S., Nag, S., & Mittal, S. (2021) · Journal of Systems Architecture · doi:10.1016/j.sysarc.2021.102031 · Cited by 19
Dendritic normalisation improves learning in sparsely connected artificial neural networks
Bird, A. D., Jedlička, P., & Cuntz, H. (2021) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1009202 · Cited by 21
Enabling Retrain-free Deep Neural Network Pruning Using Surrogate Lagrangian Relaxation
Gurevin, D., Bragin, M. A., Ding, C., Zhou, S., Pepin, L., Li, B., & Miao, F. (2021) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2021/344 · Cited by 20
Generalising to German Plural Noun Classes, from the Perspective of a Recurrent Neural Network
Dankers, V., Langedijk, A., McCurdy, K., Williams, A., & Hupkes, D. (2021) · Conference on Computational Natural Language Learning · doi:10.18653/v1/2021.conll-1.8 · Cited by 18
f-Divergence Minimization for Sequence-Level Knowledge Distillation
Wen, Y., Li, Z., Du, W., & Mou, L. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.605 · Cited by 90
SPADE: Sparse Pillar-based 3D Object Detection Accelerator for Autonomous Driving
Lee, M., Park, S., Kim, H., Yoon, M., Lee, J., Choi, J. W., Kim, N. S., Kang, M., & Choi, J. (2024) · IEEE International Symposium on High-Performance Computer Architecture (HPCA) · doi:10.1109/hpca57654.2024.00041 · Cited by 20
Automating Deep Neural Network Model Selection for Edge Inference
Lu, B., Yang, J., Chen, L. Y., & Ren, S. (2019) · IEEE International Conference on Cognitive Machine Intelligence (CogMI) · doi:10.1109/cogmi48466.2019.00035 · Cited by 18
Progressive kernel pruning with saliency mapping of input-output channels
Zhu, J., & Pei, J. (2021) · Neurocomputing · doi:10.1016/j.neucom.2021.10.009 · Cited by 18
PENNI: Pruned Kernel Sharing for Efficient CNN InferencePreprint
Li, S., Hanson, E., Li, H., & Chen, Y. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2005.07133 · Cited by 18
Robust Pruning at InitializationPreprint
Hayou, S., Ton, J.-F., Arnaud Doucet, B., & Teh, Y. W. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2002.08797 · Cited by 18
Efficient Neural Network Training via Forward and Backward Propagation SparsificationPreprint
Zhou, X., Zhang, W., Chen, Z., Diao, S., & Zhang, T. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2111.05685 · Cited by 20
Probabilistic Selective Encryption of Convolutional Neural Networks for Hierarchical Services
Tian, J., Zhou, J., & Duan, J. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.00224 · Cited by 17
Enabling and Leveraging AI in the Intelligent Edge: A Review of Current Trends and Future Directions
Goethals, T., Volckaert, B., & Turck, F. D. (2021) · IEEE Open Journal of the Communications Society · doi:10.1109/ojcoms.2021.3116437 · Cited by 17
Metadata
- Authors
- Louizos, C., Welling, M., & Kingma, D. P.
- ORCID
- Welling, Max
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2017
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 1712.01312
- OpenAlex
- W2771655537
- Cited by
- 628
- References
- 29
- Citations by year
- 2026: 3 · 2025: 16 · 2024: 31 · 2023: 83 · 2022: 54 · 2021: 189 · 2020: 143 · 2019: 82 · 2018: 27
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- TNO Repository (submittedVersion)
- Repository hosted by TU Delft Library (TU Delft) (submittedVersion)
- Data Archiving and Networked Services (DANS) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Neural Networks and Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Data Classification
- Keywords
- sparse neural networks, L0 regularization, weight pruning, concrete distribution, model selection, AIC and BIC, conditional computation, generalization
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
