The DOI matches its Crossref record; 92 of the 92 references it cites were found in a registry or an index.
Optimization with Sparsity-Inducing Penalties
Open accessAbstract
Sparse estimation methods are aimed at using or obtaining parsimonious representations of data or models. They were first dedicated to linear variable selection but numerous extensions have now emerged such as structured sparsity or kernel selection. It turns out that many of the related estimation problems can be cast as convex optimization problems by regularizing the empirical risk with appropriate nonsmooth norms. The goal of this monograph is to present from a general perspective optimization tools and techniques dedicated to such sparsity-inducing penalties. We cover proximal methods, block-coordinate descent, reweighted l2-penalized techniques, working-set and homotopy methods, as well as non-convex formulations and extensions, and provide an extensive set of experiments to compare various algorithms from a computational point of view.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Boyd, S., & Vandenberghe, L. (2004) · doi:10.1017/cbo9780511804441 · Cited by 56,601
Regression shrinkage and selection via the lasso
Tibshirani, R. (1996) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.2517-6161.1996.tb02080.x · Cited by 51,482
Greedy function approximation: A gradient boosting machine
Friedman, J. H. (2001) · The Annals of Statistics · doi:10.1214/aos/1013203451 · Cited by 30,731
Regularization and Variable Selection Via the Elastic Net
Zou, H., & Hastie, T. (2005) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.1467-9868.2005.00503.x · Cited by 20,957
Boyd, S., Parikh, N., Chu, E., Peleato, B., & Eckstein, J. (2011) · Foundations and Trends® in Machine Learning · doi:10.1561/2200000016 · Cited by 17,075
(2021) · A First Course in Statistical Programming with R · doi:10.1017/9781108993456.011 · Cited by 13,724
Boyd, S. (2010) · now publishers, Inc. eBooks · doi:10.1561/9781601984616 · Cited by 12,677
A Fast Iterative Shrinkage-Thresholding Algorithm for Linear Inverse Problems
Beck, A., & Teboulle, M. (2009) · SIAM Journal on Imaging Sciences · doi:10.1137/080716542 · Cited by 11,952
Bertsekas, D. P. (1997) · Journal of the Operational Research Society · doi:10.1038/sj.jors.2600425 · Cited by 11,726
$rm K$-SVD: An Algorithm for Designing Overcomplete Dictionaries for Sparse Representation
Aharon, M., Elad, M., & Bruckstein, A. (2006) · IEEE Transactions on Signal Processing · doi:10.1109/tsp.2006.881199 · Cited by 9,451
Robust Face Recognition via Sparse Representation
Wright, J., Yang, A. Y., Ganesh, A., Sastry, S. S., & Ma, Y. (2009) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2008.79 · Cited by 9,388
Variable Selection via Nonconcave Penalized Likelihood and its Oracle Properties
Fan, J., & Li, R. (2001) · Journal of the American Statistical Association · doi:10.1198/016214501753382273 · Cited by 8,900
(2010) · Cited by 8,261
Model Selection and Estimation in Regression with Grouped Variables
Yuan, M., & Lin, Y. (2006) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.1467-9868.2005.00532.x · Cited by 7,261
Atomic Decomposition by Basis Pursuit
Chen, S. S., Donoho, D. L., & Saunders, M. A. (1998) · SIAM Journal on Scientific Computing · doi:10.1137/s1064827596304010 · Cited by 6,845
Kernel Methods for Pattern Analysis
Shawe-Taylor, J., & Cristianini, N. (2004) · Cambridge University Press eBooks · doi:10.1017/cbo9780511809682 · Cited by 6,587
Emergence of simple-cell receptive field properties by learning a sparse code for natural images
Olshausen, B. A., & Field, D. J. (1996) · Nature · doi:10.1038/381607a0 · Cited by 5,694
Image Denoising Via Sparse and Redundant Representations Over Learned Dictionaries
Elad, M., & Aharon, M. (2006) · IEEE Transactions on Image Processing · doi:10.1109/tip.2006.881969 · Cited by 5,321
Hierarchical Clustering Schemes
Johnson, S. C. (1967) · Psychometrika · doi:10.1007/bf02289588 · Cited by 4,897
Enhancing Sparsity by Reweighted ℓ 1 Minimization
Candès, E. J., Wakin, M. B., & Boyd, S. P. (2008) · Journal of Fourier Analysis and Applications · doi:10.1007/s00041-008-9045-x · Cited by 4,863
Adapting to Unknown Smoothness via Wavelet Shrinkage
Donoho, D. L., & Johnstone, I. M. (1995) · Journal of the American Statistical Association · doi:10.1080/01621459.1995.10476626 · Cited by 4,630
Greed is Good: Algorithmic Results for Sparse Approximation
Tropp, J. A. (2004) · IEEE Transactions on Information Theory · doi:10.1109/tit.2004.834793 · Cited by 3,635
Introductory Lectures on Convex Optimization: A Basic Course
Nesterov, Y. E. (2014) · Medical Entomology and Zoology · Cited by 3,557
Beckman, R. J. (1987) · Technometrics · doi:10.1080/00401706.1987.10488195 · Cited by 2,835
Sparse Approximate Solutions to Linear Systems
Natarajan, B. K. (1995) · SIAM Journal on Computing · doi:10.1137/s0097539792240406 · Cited by 2,783
Sparsity and Smoothness Via the Fused Lasso
Tibshirani, R., Saunders, M., Rosset, S., Zhu, J., & Knight, K. (2005) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.1467-9868.2005.00490.x · Cited by 2,724
Signal Recovery by Proximal Forward-Backward Splitting
Combettes, P. L., & Wajs, V. R. (2005) · Multiscale Modeling & Simulation · doi:10.1137/050626090 · Cited by 2,678
Smooth minimization of non-smooth functions
Nesterov, Y. (2005) · Mathematical Programming · doi:10.1007/s10107-004-0552-5 · Cited by 2,455
Simultaneous analysis of Lasso and Dantzig selector
Bickel, P. J., Ritov, Y., & Tsybakov, A. B. (2009) · The Annals of Statistics · doi:10.1214/08-aos620 · Cited by 2,396
Online Learning for Matrix Factorization and Sparse Coding
Mairal, J., Bach, F., Ponce, J., & Sapiro, G. (2009) · arXiv (Cornell University) · doi:10.48550/arxiv.0908.0050 · Cited by 2,258
Self-citationAdaptive Algorithms and Stochastic Approximations
Benveniste, A., Métivier, M., & Priouret, P. (1990) · Springer eBooks · doi:10.1007/978-3-642-75894-2 · Cited by 2,006
Sparse Reconstruction by Separable Approximation
Wright, S. J., Nowak, R. D., & Figueiredo, M. A. T. (2009) · IEEE Transactions on Signal Processing · doi:10.1109/tsp.2009.2016892 · Cited by 1,854
Lectures on Stochastic Programming: Modeling and Theory
Shapiro, A., Dentcheva, D., & Ruszczyński, A. (2009) · Cited by 1,735
Sparse Bayesian Learning for Basis Selection
Wipf, D. P., & Rao, B. D. (2004) · IEEE Transactions on Signal Processing · doi:10.1109/tsp.2004.831016 · Cited by 1,559
Convex multi-task feature learning
Argyriou, A., Evgeniou, T., & Pontil, M. (2008) · Machine Learning · doi:10.1007/s10994-007-5040-8 · Cited by 1,528
Multiple kernel learning, conic duality, and the SMO algorithm
Bach, F. R., Lanckriet, G. R. G., & Jordan, M. I. (2004) · Twenty-first international conference on Machine learning - ICML '04 · doi:10.1145/1015330.1015424 · Cited by 1,453
Self-citationArgyriou, A., Evgeniou, T., & Pontil, M. (2007) · Advances in Neural Information Processing Systems 19 · doi:10.7551/mitpress/7503.003.0010 · Cited by 1,333
Iteratively reweighted least squares minimization for sparse recovery
Daubechies, I., DeVore, R., Fornasier, M., & Güntürk, C. S. (2010) · Communications on Pure and Applied Mathematics · doi:10.1002/cpa.20303 · Cited by 1,295
Method of optimal directions for frame design
Engan, K., Aase, S. O., & Hakon Husoy, J. (1999) · 1999 IEEE International Conference on Acoustics, Speech, and Signal Processing. Proceedings. ICASSP99 (Cat. No.99CH36258) · doi:10.1109/icassp.1999.760624 · Cited by 1,253
Efficiency of Coordinate Descent Methods on Huge-Scale Optimization Problems
Nesterov, Y. (2012) · SIAM Journal on Optimization · doi:10.1137/100802001 · Cited by 1,187
A unified framework for high-dimensional analysis of M-estimators with decomposable regularizers
Negahban, S. N. (2011) · Cited by 1,099
Penalized Regressions: The Bridge versus the Lasso
Fu, W. J. (1998) · Journal of Computational and Graphical Statistics · doi:10.1080/10618600.1998.10474784 · Cited by 1,092
Spike and slab variable selection: Frequentist and Bayesian strategies
Ishwaran, H., & Rao, J. S. (2005) · The Annals of Statistics · doi:10.1214/009053604000001147 · Cited by 1,033
On Model Selection Consistency of Lasso
ZhaoPeng, & Yubin, Y. (2006) · Journal of Machine Learning Research · doi:10.5555/1248547.1248637 · Cited by 996
Augmented Lagrangian and Operator-Splitting Methods in Nonlinear Mechanics.
Oden, J. T., Glowinski, R., & Tallec, P. L. (1992) · Mathematics of Computation · doi:10.2307/2153048 · Cited by 972
Maximum-Margin Matrix Factorization
Srebro, N., Rennie, J. D. M., & Jaakkola, T. (2004) · Cited by 962
A rank minimization heuristic with application to minimum order system approximation
Fazel, M., Hindi, H., & Boyd, S. P. (2001) · Proceedings of the 2001 American Control Conference. (Cat. No.01CH37148) · doi:10.1109/acc.2001.945730 · Cited by 956
Sparse multinomial logistic regression: fast algorithms and generalization bounds
Krishnapuram, B., Carin, L., Figueiredo, M. A. T., & Hartemink, A. J. (2005) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2005.127 · Cited by 877
Gradient methods for minimizing composite objective function
Nesterov, Y. E. (2007) · Cited by 867
Large-Scale Bayesian Logistic Regression for Text Categorization
Genkin, A., Lewis, D. D., & Madigan, D. (2007) · Technometrics · doi:10.1198/004017007000000245 · Cited by 808
Coordinate descent algorithms for lasso penalized regressionPreprint
Wu, T. T., & Lange, K. L. (2008) · Cited by 740
A coordinate gradient descent method for nonsmooth separable minimization
Tseng, P., & Yun, S. (2009) · Mathematical Programming · doi:10.1007/s10107-007-0170-0 · Cited by 728
Osborne, M. R., Presnell, B., & Turlach, B. A. (2000) · Journal of Computational and Graphical Statistics · doi:10.1080/10618600.2000.10474883 · Cited by 696
A statistical framework for genomic data fusion
Lanckriet, G. R. G., De Bie, T., Cristianini, N., Jordan, M. I., & Noble, W. S. (2004) · Bioinformatics · doi:10.1093/bioinformatics/bth294 · Cited by 671
Consistency of the group Lasso and multiple kernel learning
Bach, F. (2007) · arXiv (Cornell University) · doi:10.48550/arxiv.0707.3390 · Cited by 669
Self-citationThe composite absolute penalties family for grouped and hierarchical variable selection
Zhao, P., Rocha, G., & Yu, B. (2009) · The Annals of Statistics · doi:10.1214/07-aos584 · Cited by 631
A note on the group lasso and a sparse group lasso
Friedman, J. H., Hastie, T. J., & Tibshirani, R. J. (2010) · arXiv (Cornell University) · doi:10.48550/arxiv.1001.0736 · Cited by 619
Dual Averaging Method for Regularized Stochastic Learning and Online Optimization
Xiao, L. (2009) · Cited by 574
An Interior-Point Method for Large-Scale '1-Regularized Logistic Regression
Koh, K., Kim, S.-J., & Boyd, S. (2007) · Cited by 549
Ravikumar, P., Lafferty, J., Liu, H., & Wasserman, L. (2009) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.1467-9868.2009.00718.x · Cited by 535
Huang, J., & Zhang, T. (2010) · The Annals of Statistics · doi:10.1214/09-aos778 · Cited by 521
Fonctions convexes duales et points proximaux dans un espace hilbertien
Moreau, J. J. (1962) · HAL (Le Centre pour la Communication Scientifique Directe) · Cited by 501
Tseng, P. (1991) · SIAM Journal on Control and Optimization · doi:10.1137/0329006 · Cited by 467
Structured Variable Selection with Sparsity-Inducing Norms
Jenatton, R., Audibert, J.-Y., & Bach, F. (2009) · arXiv (Cornell University) · doi:10.48550/arxiv.0904.3523 · Cited by 431
Self-citationConvex Analysis and Nonlinear Optimization
Borwein, J. M., & Lewis, A. S. (2000) · Springer eBooks · doi:10.1007/978-1-4757-9859-3 · Cited by 394
Simultaneous Variable Selection
Turlach, B. A., Venables, W. N., & Wright, S. J. (2005) · Technometrics · doi:10.1198/004017005000000139 · Cited by 386
Tree-Guided Group Lasso for Multi-Task Regression with Structured Sparsity
Kim, S., & Xing, E. P. (2018) · Figshare · doi:10.1184/r1/6476414 · Cited by 374
Bottou, L., & Cun, Y. L. (2003) · Cited by 359
Fast Optimization Methods for L1 Regularization: A Comparative Study and Two New Approaches
Schmidt, M., Fung, G., & Rosales, R. (2007) · Lecture Notes in Computer Science · doi:10.1007/978-3-540-74958-5_28 · Cited by 353
Total Variation Minimization and a Class of Binary MRF Models
Chambolle, A. (2005) · Lecture Notes in Computer Science · doi:10.1007/11585978_10 · Cited by 324
A new view of automatic relevance determination
Wipf, D., & Nagarajan, S. S. (2008) · Cited by 312
Stochastic methods for l 1 regularized loss minimization
Shalev-Shwartz, S., & Tewari, A. (2009) · Proceedings of the 26th Annual International Conference on Machine Learning · doi:10.1145/1553374.1553493 · Cited by 298
Structured Sparse Principal Component Analysis
Jenatton, R., Obozinski, G., & Bach, F. (2009) · arXiv (Cornell University) · doi:10.48550/arxiv.0909.1440 · Cited by 277
Self-citationBayesian Inference and Optimal Design for the Sparse Linear Model
Seeger, M. (2008) · Infoscience (Ecole Polytechnique Fédérale de Lausanne) · doi:10.5555/1390681.1390707 · Cited by 277
Proximal Methods for Hierarchical Sparse Coding
Jenatton, R., Mairal, J., Obozinski, G., & Bach, F. (2010) · arXiv (Cornell University) · doi:10.48550/arxiv.1009.2139 · Cited by 272
Self-citationDimension Reduction: A Guided Tour
Burges, C. J. C. (2009) · Foundations and Trends® in Machine Learning · doi:10.1561/2200000002 · Cited by 271
Liu, H., Palatucci, M., & Zhang, J. (2009) · Proceedings of the 26th Annual International Conference on Machine Learning · doi:10.1145/1553374.1553458 · Cited by 216
A New Approach to Collaborative Filtering: Operator Estimation with Spectral Regularization
Abernethy, J., Bach, F., Evgeniou, T., & Vert, J.-P. (2008) · arXiv · doi:10.48550/arxiv.0802.1430 · Cited by 210 · cited 1×
“Using the trace norm as a convex surrogate for rank has many applications in control theory [48], matrix completion [1, 130], multi-task learning [109], or multi-label classification [4], where low-rank priors are adapted.”
Self-citationAn O(n) algorithm for quadratic knapsack problems
Brucker, P. (1984) · Operations Research Letters · doi:10.1016/0167-6377(84)90010-5 · Cited by 206
Learning with structured sparsity
Huang, J., Zhang, T., & Metaxas, D. (2009) · Proceedings of the 26th Annual International Conference on Machine Learning · doi:10.1145/1553374.1553429 · Cited by 182
Consistency of trace norm minimization
Bach, F. (2007) · arXiv (Cornell University) · doi:10.48550/arxiv.0710.2848 · Cited by 182
Self-citationTaking Advantage of Sparsity in Multi-Task LearningPreprint
Lounici, K., Pontil, M., Tsybakov, A. B., & van de Geer, S. (2009) · arXiv (Cornell University) · doi:10.48550/arxiv.0903.1468 · Cited by 178
Forward sequential algorithms for best basis selection
Cotter, S. F., Rao, B. D., Kreutz-Delgado, K., & Adler, J. (1999) · IEE Proceedings - Vision Image and Signal Processing · doi:10.1049/ip-vis:19990445 · Cited by 152
Convex Sparse Matrix FactorizationsPreprint
Bach, F., Mairal, J., & Ponce, J. (2008) · arXiv (Cornell University) · doi:10.48550/arxiv.0812.1869 · Cited by 129
Self-citationSparse Probabilistic Projections
Archambeau, C., & Bach, F. (2009) · Cited by 126
Self-citationSparse Bayesian nonparametric regression
Caron, F., & Arnaud Doucet, B. (2008) · International Conference on Machine Learning · doi:10.1145/1390156.1390168 · Cited by 94
Solving Structured Sparsity Regularization with Proximal Methods
Mosci, S., Rosasco, L., Santoro, M., Verri, A., & Villa, S. (2010) · Lecture notes in computer science · doi:10.1007/978-3-642-15883-4_27 · Cited by 86
Structured Sparsity in Structured Prediction
Martins, A. F. T., Smith, N. A., Aguiar, P. M. Q., & Figueiredo, M. A. T. (2018) · Figshare · doi:10.1184/r1/6473714 · Cited by 66
Variable Sparsity Kernel Learning
Aflalo, J., Ben‐Tal, A., Bhattacharyya, C., et al. (2011) · NOT FOUND REPOSITORY (Indian Institute of Science Bangalore) · doi:10.5555/1953048.1953065 · Cited by 43
Sparse coding for machine learning, image processing and computer vision
Mairal, J. (2010) · theses.fr (ABES) · doi:10.70675/72faf70fze619z44adzb071z7e7ca54a5a67 · Cited by 36
Self-citationMulti-scale Mining of fMRI Data with Hierarchical Structured Sparsity
Jenatton, R., Gramfort, A., Michel, V., Obozinski, G., Bach, F., & Thirion, B. (2011) · International Workshop on Pattern Recognition in NeuroImaging · doi:10.1109/prni.2011.15 · Cited by 29
Self-citationRegularizers for Structured SparsityPreprint
Micchelli, C. A., Morales, J. M., & Pontil, M. (2010) · arXiv (Cornell University) · doi:10.48550/arxiv.1010.0556 · Cited by 6
Cited by
Optimization Methods for Large-Scale Machine Learning
Bottou, L., Curtis, F. E., & Nocedal, J. (2018) · SIAM Review · doi:10.1137/16m1080173 · Cited by 3,128
Learning to learn by gradient descent by gradient descent
Andrychowicz, M., Denil, M., Gomez, S., Hoffman, M. W., Pfau, D., Schaul, T., Shillingford, B., & de Freitas, N. (2016) · arXiv · doi:10.48550/arxiv.1606.04474 · Cited by 1,486
FastText.zip: Compressing text classification models
Joulin, A., Grave, E., Bojanowski, P., Douze, M., Jégou, H., & Mikolov, T. (2016) · arXiv · doi:10.48550/arxiv.1612.03651 · Cited by 887
Group Sparse Beamforming for Green Cloud-RAN
(2014) · IEEE Transactions on Wireless Communications · doi:10.1109/twc.2014.040214.131770 · Cited by 468
Komodakis, N., & Pesquet, J.-C. (2015) · IEEE Signal Processing Magazine · doi:10.1109/msp.2014.2377273 · Cited by 419
Metadata
- Authors
- Bach, F., Jenatton, R., Mairal, J., & Obozinski, G.
- Venue
- Foundations and Trends® in Machine Learning
- Publisher
- Emerald
- Published
- 2025-12-17
- Volume
- 4
- Issue
- 1
- Pages
- 1-106
- ISSN
- 1935-8237 (ISSN-L)
- 1935-8245
- OpenAlex
- W2952139899
- Cited by
- 730
- References
- 92
- FWCI
- 58.13
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.
