Joint covariate selection and joint subspace selection for multiple classification problems
Open accessThe DOI matches its Crossref record; 39 of the 39 references it cites were found in a registry or an index.
Abstract
We address the problem of recovering a common set of covariates that are relevant simultaneously to several classification problems. By penalizing the sum of ℓ 2 norms of the blocks of coefficients associated with each covariate across different classification problems, similar sparsity patterns in all models are encouraged. To take computational advantage of the sparsity of solutions at high regularization levels, we propose a blockwise path-following scheme that approximately traces the regularization path. As the regularization coefficient decreases, the algorithm maintains and updates concurrently a growing set of covariates that are simultaneously active for all problems. We also show how to use random projections to extend this approach to the problem of joint subspace selection, where multiple predictors are found in a common low-dimensional subspace. We present theoretical results showing that this random projection approach converges to the solution yielded by trace-norm regularization. Finally, we present a variety of experimental results exploring joint covariate selection and joint subspace selection, comparing the path-following approach to competing algorithms in terms of prediction accuracy and running time.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Boyd, S., & Vandenberghe, L. (2004) · doi:10.1017/cbo9780511804441 · Cited by 56,601
Regression shrinkage and selection via the lasso
Tibshirani, R. (1996) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.2517-6161.1996.tb02080.x · Cited by 51,482
The Elements of Statistical Learning
Hastie, T., Friedman, J., & Tibshirani, R. (2001) · Springer Series in Statistics · doi:10.1007/978-0-387-21606-5 · Cited by 24,162
Advances in neural information processing systems 7
Cios, K., & Shields, M. (1997) · Neurocomputing · doi:10.1016/s0925-2312(97)90018-7 · Cited by 22,059
(2005) · Technometrics · doi:10.1198/tech.2005.s303 · Cited by 17,923
Efron, B., Hastie, T., Johnstone, I., & Tibshirani, R. (2004) · The Annals of Statistics · doi:10.1214/009053604000000067 · Cited by 9,357
Model Selection and Estimation in Regression with Grouped Variables
Yuan, M., & Lin, Y. (2006) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.1467-9868.2005.00532.x · Cited by 7,261 · cited 1×
“This approach is a particular case of a general methodology in which block norms are used to define groupings of variables in regression and classification problems (Bach et al. 2004; Yuan and Lin 2006; Park and Hastie 2006; Meier et al. 2008; Kim et al. 2006; Zhao et al. 2008).”
MethodDraper, N. R., & Smith, H. (1998) · Wiley Series in Probability and Statistics · doi:10.1002/9781118625590 · Cited by 5,024
Gunst, R. F. (1999) · Technometrics · doi:10.1080/00401706.1999.10485680 · Cited by 4,002
Khan, J., Wei, J. S., Ringnér, M., Saal, L. H., Ladanyi, M., Westermann, F., Berthold, F., Schwab, M., Antonescu, C. R., Peterson, C., et al. (2001) · Nature Medicine · doi:10.1038/89044 · Cited by 2,644 · cited 1×
“6, consider a DNA microarray analysis problem in which the covariates are levels of gene expression and the responses are phenotypes or cellular processes (Khan et al. 2001).”
Background
Cited by
Online Learning for Matrix Factorization and Sparse Coding
Mairal, J., Bach, F., Ponce, J., & Sapiro, G. (2009) · arXiv (Cornell University) · doi:10.48550/arxiv.0908.0050 · Cited by 2,258
Cross-Stitch Networks for Multi-task Learning
Misra, I., Shrivastava, A., Gupta, A., & Hebert, M. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.433 · Cited by 1,397
An overview of multi-task learning
Zhang, Y., & Yang, Q. (2018) · National Science Review · doi:10.1093/nsr/nwx105 · Cited by 1,117
Optimization for Machine Learning
(2011) · The MIT Press eBooks · doi:10.7551/mitpress/8996.001.0001 · Cited by 875
Group lasso with overlap and graph lasso
Jacob, L., Obozinski, G., & Vert, J.-P. (2009) · Proceedings of the 26th Annual International Conference on Machine Learning · doi:10.1145/1553374.1553431 · Cited by 833
Self-citationA Simpler Approach to Matrix Completion
Recht, B. (2009) · arXiv (Cornell University) · doi:10.48550/arxiv.0910.0651 · Cited by 673
Elastic Net Regularization Paths for All Generalized Linear Models
Tay, J. K., Narasimhan, B., & Hastie, T. (2023) · Journal of Statistical Software · doi:10.18637/jss.v106.i01 · Cited by 653
Deep Learning and Its Applications in Biomedicine
Cao, C., Liu, F., Tan, H., Song, D., Shu, W., Li, W., Zhou, Y., Bo, X., & Xie, Z. (2018) · Genomics, Proteomics & Bioinformatics · doi:10.1016/j.gpb.2017.07.003 · Cited by 624
Optimization with Sparsity-Inducing Penalties
Bach, F. (2011) · now publishers, Inc. eBooks · doi:10.1561/9781601985118 · Cited by 608
An accelerated gradient method for trace norm minimization
Ji, S., & Ye, J. (2009) · Proceedings of the 26th Annual International Conference on Machine Learning · doi:10.1145/1553374.1553434 · Cited by 517
An Overview of Low-Rank Matrix Recovery From Incomplete Observations
Davenport, M. A., & Romberg, J. (2016) · IEEE Journal of Selected Topics in Signal Processing · doi:10.1109/jstsp.2016.2539100 · Cited by 439
Structured Variable Selection with Sparsity-Inducing Norms
Jenatton, R., Audibert, J.-Y., & Bach, F. (2009) · arXiv (Cornell University) · doi:10.48550/arxiv.0904.3523 · Cited by 431
Kernels for Vector-Valued Functions: A Review
Álvarez, M. A., Rosasco, L., & Lawrence, N. D. (2012) · Foundations and Trends® in Machine Learning · doi:10.1561/2200000036 · Cited by 415
Tree-Guided Group Lasso for Multi-Task Regression with Structured Sparsity
Kim, S., & Xing, E. P. (2018) · Figshare · doi:10.1184/r1/6476414 · Cited by 374
Visual Classification With Multitask Joint Sparse Representation
Yuan, X.-T., Liu, X., & Yan, S. (2012) · IEEE Transactions on Image Processing · doi:10.1109/tip.2012.2205006 · Cited by 364
Support union recovery in high-dimensional multivariate regression
Obozinski, G., Wainwright, M. J., & Jordan, M. I. (2011) · The Annals of Statistics · doi:10.1214/09-aos776 · Cited by 350
Self-citationFeature Selection Based on Structured Sparsity: A Comprehensive Study
Gui, J., Sun, Z., Ji, S., Tao, D., & Tan, T. (2017) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2016.2551724 · Cited by 335
Multi-target regression via input space expansion: treating targets as inputs
Spyromitros-Xioufis, E., Tsoumakas, G., Groves, W., & Vlahavas, I. (2016) · Machine Learning · doi:10.1007/s10994-016-5546-z · Cited by 326
Sparse Modeling for Image and Vision Processing
Mairal, J. (2014) · now publishers, Inc. eBooks · doi:10.1561/9781680830095 · Cited by 322
Structured Sparsity through Convex Optimization
Bach, F., Jenatton, R., Mairal, J., & Obozinski, G. (2012) · Statistical Science · doi:10.1214/12-sts394 · Cited by 315
Self-citationVisual classification with multi-task joint sparse representation
Yuan, X.-T., & Yan, S. (2010) · 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2010.5539967 · Cited by 303
Structured Sparse Principal Component Analysis
Jenatton, R., Obozinski, G., & Bach, F. (2009) · arXiv (Cornell University) · doi:10.48550/arxiv.0909.1440 · Cited by 277
Self-citationA brief review on multi-task learning
Thung, K.-H., & Wee, C.-Y. (2018) · Multimedia Tools and Applications · doi:10.1007/s11042-018-6463-x · Cited by 263
Yang, Q., Zhang, Y., Dai, W., & Pan, S. J. (2020) · Cambridge University Press eBooks · doi:10.1017/9781139061773 · Cited by 254
Feature Selection for Multimedia Analysis by Sharing Information Among Multiple Tasks
Yang, Y., Ma, Z., Hauptmann, A. G., & Sebe, N. (2013) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2012.2237023 · Cited by 202
“As reported in [27], the shared structure of multiple variables can be expressed by a low rank matrix of them.”
BackgroundSAR Target Recognition via Joint Sparse Representation of Monogenic Signal
Dong, G., Kuang, G., Wang, N., Zhao, L., & Lu, J. (2015) · IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing · doi:10.1109/jstars.2015.2436694 · Cited by 177
Jie, B., Liu, M., & Shen, D. (2018) · Medical Image Analysis · doi:10.1016/j.media.2018.03.013 · Cited by 172
Hyper-connectivity of functional networks for brain disease diagnosis
Jie, B., Wee, C.-Y., Shen, D., & Zhang, D. (2016) · Medical Image Analysis · doi:10.1016/j.media.2016.03.003 · Cited by 160
“Compared with the conventional single-task feature selection methods, multi-task feature selection can utilize related auxiliary information among tasks and hence often leads to better learning model (Argyriou et al., 2008; Obozinski et al., 2010).”
Initiative, T. A. D. N., Zu, C., Jie, B., Liu, M., Chen, S., Shen, D., & Zhang, D. (2016) · Brain Imaging and Behavior · doi:10.1007/s11682-015-9480-7 · Cited by 95
Burn scar detection using cloudy MODIS images via low-rank and sparsity-based models
Dao, M., Kwan, C., Ayhan, B., & Tran, T. D. (2016) · 2016 IEEE Global Conference on Signal and Information Processing (GlobalSIP) · doi:10.1109/globalsip.2016.7905827 · Cited by 42
“This model is called joint sparse representation (JSR) and has been successfully applied in many applications, such as hyperspectral chemical plume detection [11], visual data classification [12], or multi-sensor classification [13].”
MethodA Feature Selection Method for Multimodal Multispectral LiDAR Sensing
Han, Y., Salido-Monzú, D., Butt, J. A., Schweizer, S., & Wieser, A. (2024) · ISPRS Journal of Photogrammetry and Remote Sensing · doi:10.1016/j.isprsjprs.2024.04.022 · Cited by 6
Metadata
- Authors
- Obozinski, G., Taskar, B., & Jordan, M. I.
- Venue
- Statistics and Computing
- Published
- 2009-01-14
- Volume
- 20
- Issue
- 2
- Pages
- 231-252
- ISSN
- 0960-3174 (ISSN-L)
- 1573-1375
- OpenAlex
- W2159514083
- Cited by
- 451
- References
- 39
- FWCI
- 29.52
- Type
- journal article
- Language
- en
- Open access
- hybrid
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.