The DOI matches its Crossref record; 61 of the 61 references it cites were found in a registry or an index.
Group sparse regularization for deep neural networks
Open accessAbstract
In this paper, we address the challenging task of simultaneously optimizing (i) the weights of a neural network, (ii) the number of neurons for each hidden layer, and (iii) the subset of active input features (i.e., feature selection). While these problems are traditionally dealt with separately, we propose an efficient regularized formulation enabling their simultaneous parallel execution, using standard optimization routines. Specifically, we extend the group Lasso penalty, originally proposed in the linear regression literature, to impose group-level sparsity on the networks connections, where each group is defined as the set of outgoing weights from a unit. Depending on the specific case, the weights can be related to an input variable, to a hidden neuron, or to a bias unit, thus performing simultaneously all the aforementioned tasks in order to obtain a compact network. We carry out an extensive experimental evaluation, in comparison with classical weight decay and Lasso penalties, both on a toy dataset for handwritten digit recognition, and multiple realistic mid-scale classification benchmarks. Comparative results demonstrate the potential of our proposed sparse group Lasso penalty in producing extremely compact networks, with a significantly lower number of input features, with a classification accuracy which is equal or only slightly inferior to standard regularization terms.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
LeCun, Y., Bengio, Y., & Hinton, G. (2015) · Nature · doi:10.1038/nature14539 · Cited by 84,126
Scikit-learn: Machine Learning in Python
Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Müller, A., Nothman, J., Louppe, G., et al. (2012) · arXiv · doi:10.48550/arxiv.1201.0490 · Cited by 71,016
Regression shrinkage and selection via the lasso
Tibshirani, R. (1996) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.2517-6161.1996.tb02080.x · Cited by 51,482
(2014) · Cited by 33,419
Regularization and Variable Selection Via the Elastic Net
Zou, H., & Hastie, T. (2005) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.1467-9868.2005.00503.x · Cited by 20,957
Deep learning in neural networks: an overview
Schmidhuber, J. (2015) · Neural Networks · doi:10.1016/j.neunet.2014.09.003 · Cited by 18,570
Proceedings of IEEE Conference on Computer Vision and Pattern Recognition
(1993) · Proceedings of IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.1993.341010 · Cited by 14,265
Distilling the Knowledge in a Neural Network
Hinton, G., Vinyals, O., & Dean, J. (2015) · arXiv · doi:10.48550/arxiv.1503.02531 · Cited by 14,204
(2010) · Cited by 14,134
An Introduction To Compressive Sampling
Candes, E. J., & Wakin, M. B. (2008) · IEEE Signal Processing Magazine · doi:10.1109/msp.2007.914731 · Cited by 10,017
An introduction to variable and feature selection
GuyonIsabelle, & ElisseeffAndré (2003) · Journal of Machine Learning Research · doi:10.5555/944919.944968 · Cited by 7,715
Neural Networks And Learning Machines
Haykin, S. (2010) · Cited by 7,535
Model Selection and Estimation in Regression with Grouped Variables
Yuan, M., & Lin, Y. (2006) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.1467-9868.2005.00532.x · Cited by 7,261
(2011) · Cited by 6,993
An Introduction to Variable and Feature Selection.
Guyon, I., & Elisseeff, A. (2003) · Cited by 6,576
Proceedings European Conference on Computer Vision 2002
Heyden, A., Sparr, G., Nielsen, M., & Johansen, P. (2002) · Lund University Publications (Lund University) · Cited by 5,153
The MNIST Database of Handwritten Digit Images for Machine Learning Research [Best of the Web]
Deng, L. (2012) · IEEE Signal Processing Magazine · doi:10.1109/msp.2012.2211477 · Cited by 4,836
Regression Shrinkage and Selection via The Lasso: A Retrospective
Tibshirani, R. (2011) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.1467-9868.2011.00771.x · Cited by 3,589
A few useful things to know about machine learning
Domingos, P. (2012) · Communications of the ACM · doi:10.1145/2347736.2347755 · Cited by 3,326 · cited 1×
“Arguments that favor simple models instead of complex models for describing a phenomenon are quite known in the machine learning literature [5].”
BackgroundA survey of deep neural network architectures and their applications
Liu, W., Wang, Z., Liu, X., Zeng, N., Liu, Y., & Alsaadi, F. E. (2017) · Neurocomputing · doi:10.1016/j.neucom.2016.12.038 · Cited by 3,239
(1989) · Cited by 3,214
Deep learning for visual understanding: A review
Guo, Y., Liu, Y., Oerlemans, A., Lao, S., Wu, S., & Lew, M. S. (2016) · Neurocomputing · doi:10.1016/j.neucom.2015.09.116 · Cited by 2,504
Practical Recommendations for Gradient-Based Training of Deep Architectures
Bengio, Y. (2012) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-35289-8_26 · Cited by 2,259
BinaryConnect: Training Deep Neural Networks with binary weights during propagations
Courbariaux, M., Bengio, Y., & David, J.-P. (2015) · arXiv · doi:10.48550/arxiv.1511.00363 · Cited by 2,180
Livescu, K., Bezman, A., Borges, N., Yung, L.-Y., Çetin, Ö., Frankel, J., King, S., Magimai.-Doss, M., Chi, X., & Lavoie, L. (2007) · Edinburgh Research Explorer (University of Edinburgh) · Cited by 2,145
Neural Networks: Tricks of the Trade: Second Edition
(2012) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-35289-8 · Cited by 2,116
Deep Networks with Stochastic Depth
Huang, G., Sun, Y., Liu, Z., Sedra, D., & Weinberger, K. Q. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46493-0_39 · Cited by 2,104
Proceedings of The 32nd International Conference on Machine Learning
Germain, M., Gregor, K., Murray, I., & Larochelle, H. (2015) · Cited by 2,020
(1992) · Cited by 1,405
Simon, N., Friedman, J., Hastie, T., & Tibshirani, R. (2013) · Journal of Computational and Graphical Statistics · doi:10.1080/10618600.2012.681250 · Cited by 1,380
Deep Learning with Limited Numerical Precision
Gupta, S., Agrawal, A., Gopalakrishnan, K., & Narayanan, P. (2015) · arXiv · doi:10.48550/arxiv.1502.02551 · Cited by 1,364 · cited 1×
“…on the gradient of the cost function to remove ‘non salient’ connections after training [14]; (iii) we can work with limited numerical precision to reduce storage [15] (up to the extreme of a single bit per weight [8]); or we can use hash functions to force weight sharing [16]; and so on.”
BackgroundHogwild: A Lock-Free Approach to Parallelizing Stochastic Gradient Descent
Niu, F., Recht, B., Re, C., & Wright, S. J. (2011) · arXiv · doi:10.48550/arxiv.1106.5730 · Cited by 1,211
Theano: A CPU and GPU Math Compiler in Python
Bergstra, J., Breuleux, O., Bastien, F., Lamblin, P., Pascanu, R., Desjardins, G., Turian, J., Warde-Farley, D., & Bengio, Y. (2010) · Proceedings of the Python in Science Conference · doi:10.25080/majora-92bf1922-003 · Cited by 1,052
Compressing Neural Networks with the Hashing Trick
Chen, W., Wilson, J. T., Tyree, S., Weinberger, K. Q., & Chen, Y. (2015) · arXiv · doi:10.48550/arxiv.1504.04788 · Cited by 974
Input feature selection for classification problems
Kwak, N., & Choi, C.-H. (2002) · IEEE Transactions on Neural Networks · doi:10.1109/72.977291 · Cited by 938
Compressing Deep Convolutional Networks using Vector Quantization
Gong, Y., Liu, L., Yang, M., & Bourdev, L. D. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1412.6115 · Cited by 925
Zhu, W., Lan, C., Xing, J., Zeng, W., Li, Y., Shen, L., & Xie, X. (2016) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v30i1.10451 · Cited by 915
Predicting Parameters in Deep Learning
Denil, M., Shakibi, B., Dinh, L., Ranzato, M., & de Freitas, N. (2013) · arXiv · doi:10.48550/arxiv.1306.0543 · Cited by 853
Deep Learning with Limited Numerical Precision
Gupta, S., Agrawal, A., Gopalakrishnan, K., & Narayanan, P. (2015) · Cited by 757
Optimization with Sparsity-Inducing Penalties
Bach, F., Jenatton, R., Mairal, J., & Obozinski, G. (2025) · Foundations and Trends® in Machine Learning · doi:10.1561/2200000015 · Cited by 730
Sparse Convolutional Neural Networks
Liu, B., Wang, M., Foroosh, H., Tappen, M., & Penksy, M. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298681 · Cited by 680
Consistency of the group Lasso and multiple kernel learning
Bach, F. (2007) · arXiv (Cornell University) · doi:10.48550/arxiv.0707.3390 · Cited by 669
A note on the group lasso and a sparse group lasso
Friedman, J. H., Hastie, T. J., & Tibshirani, R. J. (2010) · arXiv (Cornell University) · doi:10.48550/arxiv.1001.0736 · Cited by 619
Optimization with Sparsity-Inducing Penalties
Bach, F. (2011) · now publishers, Inc. eBooks · doi:10.1561/9781601985118 · Cited by 608
Low-rank matrix factorization for Deep Neural Network training with high-dimensional output targets
Sainath, T. N., Kingsbury, B., Sindhwani, V., Arisoy, E., & Ramabhadran, B. (2013) · 2013 IEEE International Conference on Acoustics, Speech and Signal Processing · doi:10.1109/icassp.2013.6638949 · Cited by 593
Multi-Task Feature Learning Via Efficient l2,1-Norm Minimization
Liu, J., Ji, S., & Ye, J. (2012) · arXiv (Cornell University) · doi:10.48550/arxiv.1205.2631 · Cited by 555
Blackard, J. A., & Dean, D. J. (1999) · Computers and Electronics in Agriculture · doi:10.1016/s0168-1699(99)00046-0 · Cited by 495
Proceedings of the Twenty-Fifth Conference on Uncertainty in Artificial Intelligence
McAllester, D. (2009) · Uncertainty in Artificial Intelligence · Cited by 464
Predicting Parameters in Deep Learning
Shakibi, B. (2014) · University of British Columbia · doi:10.14288/1.0165555 · Cited by 462
Structured Variable Selection with Sparsity-Inducing Norms
Jenatton, R., Audibert, J.-Y., & Bach, F. (2009) · arXiv (Cornell University) · doi:10.48550/arxiv.0904.3523 · Cited by 431
The MNIST Database of Handwritten Digit Images for Machine Learning Research
Deng, L. (2012) · IEEE Signal Processing Magazine · Cited by 397
Decomposition by Partial Linearization: Parallel Optimization of Multi-Agent Systems
Scutari, G., Facchinei, F., Song, P., Palomar, D. P., & Pang, J.-S. (2014) · IEEE Transactions on Signal Processing · doi:10.1109/tsp.2013.2293126 · Cited by 338
On Iteratively Reweighted Algorithms for Nonsmooth Nonconvex Optimization in Computer Vision
Ochs, P., Dosovitskiy, A., Brox, T., & Pock, T. (2015) · SIAM Journal on Imaging Sciences · doi:10.1137/140971518 · Cited by 207
Heterogeneous Feature Selection With Multi-Modal Deep Neural Networks and Sparse Group LASSO
Zhao, L., Hu, Q., & Wang, W. (2015) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2015.2477058 · Cited by 136
Prediction of telephone calls load using Echo State Network with exogenous variables
Bianchi, F. M., Scardapane, S., Uncini, A., Rizzi, A., & Sadeghian, A. (2015) · Neural Networks · doi:10.1016/j.neunet.2015.08.010 · Cited by 87
Self-citationA Simple Neural Network Pruning Algorithm with Application to Filter Synthesis
Suzuki, K., Horiba, I., & Sugie, N. (2001) · Neural Processing Letters · doi:10.1023/a:1009639214138 · Cited by 84
On parallelizability of stochastic gradient descent for speech DNNS
Seide, F. T. B., Fu, H., Droppo, J., Li, G., & Yu, D. (2014) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2014.6853593 · Cited by 78
Distributed semi-supervised support vector machines
Scardapane, S., Fierimonte, R., Lorenzo, P. D., Panella, M., & Uncini, A. (2016) · Neural Networks · doi:10.1016/j.neunet.2016.04.007 · Cited by 57
Self-citationMorozov, Ivanov and Tikhonov Regularization Based LS-SVMs
Pelckmans, K., Suykens, J. A. K., & Moor, B. D. (2004) · Lecture notes in computer science · doi:10.1007/978-3-540-30499-9_189 · Cited by 18
Cited by
Learning Efficient Convolutional Networks through Network Slimming
Liu, Z., Li, J., Shen, Z., Huang, G., Yan, S., & Zhang, C. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.298 · Cited by 2,761
Hoefler, T., Alistarh, D., Ben-Nun, T., Dryden, N., & Peste, A. (2021) · ETH Zurich · doi:10.3929/ethz-b-000519202 · Cited by 423
Bayesian Compression for Deep Learning
Louizos, C., Ullrich, K., & Welling, M. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1705.08665 · Cited by 312
Variational Dropout Sparsifies Deep Neural Networks
Molchanov, D., Ashukha, A., & Vetrov, D. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1701.05369 · Cited by 311
Erion, G., Janizek, J. D., Sturmfels, P., Lundberg, S. M., & Lee, S.-I. (2021) · Nature Machine Intelligence · doi:10.1038/s42256-021-00343-w · Cited by 300
Metadata
- Authors
- Scardapane, S., Comminiello, D., Hussain, A., & Uncini, A.
- Venue
- Neurocomputing
- Publisher
- Elsevier BV
- Published
- 2017-06
- Volume
- 241
- Pages
- 81-89
- ISSN
- 0925-2312 (ISSN-L)
- OpenAlex
- W2460144244
- Cited by
- 457
- References
- 61
- FWCI
- 36.00
- Type
- journal article
- Language
- en
- Open access
- bronze
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.