Coordinate descent algorithms for lasso penalized regression
Open accessThe DOI matches its Crossref record; 27 of the 27 references it cites were found in a registry or an index.
Abstract
Imposition of a lasso penalty shrinks parameter estimates toward zero and performs continuous model selection. Lasso penalized regression is capable of handling linear regression problems where the number of predictors far exceeds the number of cases. This paper tests two exceptionally fast algorithms for estimating regression coefficients with a lasso penalty. The previously known ℓ2 algorithm is based on cyclic coordinate descent. Our new ℓ1 algorithm is based on greedy coordinate descent and Edgeworth’s algorithm for ordinary ℓ1 regression. Each algorithm relies on a tuning constant that can be chosen by cross-validation. In some regression problems it is natural to group parameters and penalize parameters group by group rather than separately. If the group penalty is proportional to the Euclidean norm of the parameters of the group, then it is possible to majorize the norm and reduce parameter estimation to ℓ2 regression with a lasso penalty. Thus, the existing algorithm can be extended to novel settings. Each of the algorithms discussed is tested via either simulated or real data or both. The Appendix proves that a greedy form of the ℓ2 algorithm converges to the minimum value of the objective function.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Regression shrinkage and selection via the lasso
Tibshirani, R. (1996) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.2517-6161.1996.tb02080.x · Cited by 51,482
Regularization and Variable Selection Via the Elastic Net
Zou, H., & Hastie, T. (2005) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.1467-9868.2005.00503.x · Cited by 20,957
Model Selection and Estimation in Regression with Grouped Variables
Yuan, M., & Lin, Y. (2006) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.1467-9868.2005.00532.x · Cited by 7,261
Atomic Decomposition by Basis Pursuit
Chen, S. S., Donoho, D. L., & Saunders, M. A. (1998) · SIAM Journal on Scientific Computing · doi:10.1137/s1064827596304010 · Cited by 6,845
An iterative thresholding algorithm for linear inverse problems with a sparsity constraint.
Daubechies, I., Defrise, M., & De Mol, C. (2004) · Communications on Pure and Applied Mathematics · doi:10.1002/cpa.20042 · Cited by 4,958
Convergence of a Block Coordinate Descent Method for Nondifferentiable Minimization
Tseng, P. (2001) · Journal of Optimization Theory and Applications · doi:10.1023/a:1017501703105 · Cited by 1,989
Pathwise coordinate optimization
Friedman, J., Hastie, T., Höfling, H., & Tibshirani, R. (2007) · The Annals of Applied Statistics · doi:10.1214/07-aoas131 · Cited by 1,895
Hunter, D. R., & Lange, K. (2004) · The American Statistician · doi:10.1198/0003130042836 · Cited by 1,783
Self-citationThe Dantzig selector: Statistical estimation when p is much larger than n
Candes, E., & Tao, T. (2007) · The Annals of Statistics · doi:10.1214/009053606000001523 · Cited by 1,747
Penalized Regressions: The Bridge versus the Lasso
Fu, W. J. (1998) · Journal of Computational and Graphical Statistics · doi:10.1080/10618600.1998.10474784 · Cited by 1,092
Cited by
Regression Shrinkage and Selection via The Lasso: A Retrospective
Tibshirani, R. (2011) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.1467-9868.2011.00771.x · Cited by 3,589
Challenges of Big Data analysis
Fan, J., Han, F., & Liu, H. (2014) · National Science Review · doi:10.1093/nsr/nwt032 · Cited by 1,482
Structure-Preserving Color Normalization and Sparse Stain Separation for Histological Images
Vahadane, A., Peng, T., Sethi, A., Albarqouni, S., Wang, L., Baust, M., Steiger, K., Schlitter, A. M., Esposito, I., & Navab, N. (2016) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2016.2529665 · Cited by 825
Genome-wide association analysis by lasso penalized logistic regression
Wu, T. T., Chen, Y. F., Hastie, T., Sobel, E., & Lange, K. (2009) · Bioinformatics · doi:10.1093/bioinformatics/btp041 · Cited by 785
Self-citationBreheny, P., & Huang, J. (2011) · The Annals of Applied Statistics · doi:10.1214/10-aoas388 · Cited by 671
Richtárik, P., & Takáč, M. (2014) · Mathematical Programming · doi:10.1007/s10107-012-0614-z · Cited by 654
Prioritizing GWAS Results: A Review of Statistical Methods and Recommendations for Their Application
Cantor, R. M., Lange, K., & Sinsheimer, J. S. (2010) · The American Journal of Human Genetics · doi:10.1016/j.ajhg.2009.11.017 · Cited by 622
Self-citationNetwork-constrained regularization and variable selection for analysis of genomic data
Li, C., & Li, H. (2008) · Bioinformatics · doi:10.1093/bioinformatics/btn081 · Cited by 599
Distributed Sparse Linear Regression
Mateos, G., Bazerque, J. A., & Giannakis, G. B. (2010) · IEEE Transactions on Signal Processing · doi:10.1109/tsp.2010.2055862 · Cited by 550
Structured Sparsity through Convex Optimization
Bach, F., Jenatton, R., Mairal, J., & Obozinski, G. (2012) · Statistical Science · doi:10.1214/12-sts394 · Cited by 315
A Selective Review of Group Selection in High-Dimensional Models
Huang, J., Breheny, P., & Ma, S. (2012) · Statistical Science · doi:10.1214/12-sts392 · Cited by 300
Stochastic methods for l 1 regularized loss minimization
Shalev-Shwartz, S., & Tewari, A. (2009) · Proceedings of the 26th Annual International Conference on Machine Learning · doi:10.1145/1553374.1553493 · Cited by 298
Breheny, P., & Huang, J. (2015) · Statistics and Computing · doi:10.1007/s11222-013-9424-2 · Cited by 282
Deep Air Learning: Interpolation, Prediction, and Feature Analysis of Fine-Grained Air Quality
Qi, Z., Wang, T., Song, G., Hu, W., Li, X., & Zhang, Z. (2018) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2018.2823740 · Cited by 274
Online Adaptive Estimation of Sparse Signals: Where RLS Meets the $\ell_1$-Norm
Angelosante, D., Bazerque, J. A., & Giannakis, G. B. (2010) · IEEE Transactions on Signal Processing · doi:10.1109/tsp.2010.2046897 · Cited by 272
Newtonized Orthogonal Matching Pursuit: Frequency Estimation Over the Continuum
Mamandipoor, B., Ramasamy, D., & Madhow, U. (2016) · IEEE Transactions on Signal Processing · doi:10.1109/tsp.2016.2580523 · Cited by 262
Liu, H., Palatucci, M., & Zhang, J. (2009) · Proceedings of the 26th Annual International Conference on Machine Learning · doi:10.1145/1553374.1553458 · Cited by 216
“… Basis Discovery Han Liu hanliu@cs.cmu.edu Machine Learning Department, School of Computer Science, Carnegie Mellon University, PA 15213 USA Mark Palatucci mpalatuc@cs.cmu.edu School of Computer Science, Carnegie Mellon University, 5000 Forbes Ave., Pittsburgh, PA 15213 USA Jian Zhang…”
BackgroundMethodAssociation screening of common and rare genetic variants by penalized regression
Zhou, H., Sehl, M. E., Sinsheimer, J. S., & Lange, K. (2010) · Bioinformatics · doi:10.1093/bioinformatics/btq448 · Cited by 137
Self-citationMendel: the Swiss army knife of genetic analysis programs
Lange, K., Papp, J. C., Sinsheimer, J. S., Sripracha, R., Zhou, H., & Sobel, E. M. (2013) · Bioinformatics · doi:10.1093/bioinformatics/btt187 · Cited by 109
Self-citationChung, W., Chen, J., Turman, C., Lindstrom, S., Zhu, Z., Loh, P.-R., Kraft, P., & Liang, L. (2019) · Nature Communications · doi:10.1038/s41467-019-08535-0 · Cited by 60
Mu, D., Jiang, J., & Zhang, C. (2013) · Energies · doi:10.3390/en6052583 · Cited by 17
“Recent advances in sparse linear model identification have shown that the cyclic coordinate descent (CCD) algorithm provides an efficient means of solving the penalized least squares (LS) problem [28,29] and can be implemented online in a recursive fashion [30,31].”
Background
Metadata
- Authors
- Wu, T. T., & Lange, K.
- Venue
- The Annals of Applied Statistics
- Publisher
- Institute of Mathematical Statistics
- Published
- 2008-03-01
- Volume
- 2
- Issue
- 1
- Pages
- 224-244
- ISSN
- 1932-6157 (ISSN-L)
- OpenAlex
- W2043478128
- Cited by
- 563
- References
- 27
- FWCI
- 19.25
- Type
- journal article
- Language
- en
- Open access
- bronze
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.