Generalized Linear Discriminant Analysis: A Unified Framework and Efficient Model Selection
The DOI matches its Crossref record; 37 of the 37 references it cites were found in a registry or an index.
Abstract
High-dimensional data are common in many domains, and dimensionality reduction is the key to cope with the curse-of-dimensionality. Linear discriminant analysis (LDA) is a well-known method for supervised dimensionality reduction. When dealing with high-dimensional and low sample size data, classical LDA suffers from the singularity problem. Over the years, many algorithms have been developed to overcome this problem, and they have been applied successfully in various applications. However, there is a lack of a systematic study of the commonalities and differences of these algorithms, as well as their intrinsic relationships. In this paper, a unified framework for generalized LDA is proposed, which elucidates the properties of various algorithms and their relationships. Based on the proposed framework, we show that the matrix computations involved in LDA-based algorithms can be simplified so that the cross-validation procedure for model selection can be performed efficiently. We conduct extensive experiments using a collection of high-dimensional data sets, including text documents, face images, gene expression data, and gene expression pattern images, to evaluate the proposed theories and algorithms.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Golub, G. (2012) · doi:10.56021/9781421407944 · Cited by 34,505
The Elements of Statistical Learning
Hastie, T., Tibshirani, R., & Friedman, J. (2009) · Springer Series in Statistics · doi:10.1007/978-0-387-84858-7 · Cited by 25,970
THE USE OF MULTIPLE MEASUREMENTS IN TAXONOMIC PROBLEMS
FISHER, R. A. (1936) · Annals of Eugenics · doi:10.1111/j.1469-1809.1936.tb02137.x · Cited by 15,227 · cited 1×
“The commonly used dimensionality reduction methods include supervised approaches such as linear discriminant analysis (LDA) [10], [11], and unsupervised ones such as principal component analysis (PCA) [18], and additional spectral and”
MethodNonlinear Dimensionality Reduction by Locally Linear Embedding
Roweis, S. T., & Saul, L. K. (2000) · Science · doi:10.1126/science.290.5500.2323 · Cited by 14,965 · cited 1×
“manifold learning methods [2], [4], [25], [26], [28].”
A Global Geometric Framework for Nonlinear Dimensionality Reduction
Tenenbaum, J. B., Silva, V. D., & Langford, J. C. (2000) · Science · doi:10.1126/science.290.5500.2319 · Cited by 13,585
The Elements of Statistical Learning: Data Mining, Inference, and Prediction
John Lu, Z. Q. (2010) · Journal of the Royal Statistical Society Series A: Statistics in Society · doi:10.1111/j.1467-985x.2010.00646_6.x · Cited by 12,390
Eigenfaces vs. Fisherfaces: recognition using class specific linear projection
Belhumeur, P. N., Hespanha, J. P., & Kriegman, D. J. (1997) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/34.598228 · Cited by 11,707
Introduction to Statistical Pattern Recognition
(1990) · Elsevier eBooks · doi:10.1016/c2009-0-27872-x · Cited by 11,031
Goodall, C. (1988) · Technometrics · doi:10.1080/00401706.1988.10488412 · Cited by 8,675
Jolliffe, I. T. (1986) · Springer Series in Statistics · doi:10.1007/978-1-4757-1904-8 · Cited by 8,273
Laplacian Eigenmaps for Dimensionality Reduction and Data Representation
Belkin, M., & Niyogi, P. (2003) · Neural Computation · doi:10.1162/089976603321780317 · Cited by 7,621 · cited 1×
“manifold learning methods [2], [4], [25], [26], [28].”
Chiang, L. H., Russell, E. L., & Braatz, R. D. (2001) · Advanced Textbooks in Control and Signal Processing · doi:10.1007/978-1-4471-0347-9_3 · Cited by 5,003
Martinez, A. M., & Kak, A. C. (2001) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/34.908974 · Cited by 3,249
Graph Embedding and Extensions: A General Framework for Dimensionality Reduction
Yan, S., Xu, D., Zhang, B., Zhang, H.-J., Yang, Q., & Lin, S. (2007) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2007.250598 · Cited by 3,031
Adaptive Control Processes: A Guided Tour
Bellman, R. E. (1961) · doi:10.1515/9781400874668 · Cited by 2,802
Martínez, A. M. (1998) · CTIT technical reports series · Cited by 2,778
Prediction of central nervous system embryonal tumour outcome based on gene expression
Pomeroy, S. L., Tamayo, P., Gaasenbeek, M., Sturla, L. M., Angelo, M., McLaughlin, M. E., Kim, J. Y. H., Goumnerova, L. C., Black, P. M., Lau, C., et al. (2002) · Nature · doi:10.1038/415436a · Cited by 2,371
ADVANCES IN NEURAL INFORMATION PROCESSING SYSTEMS 16
Storkey, A. (2004) · neural information processing systems · Cited by 2,085
Multiclass cancer diagnosis using tumor gene expression signatures
Ramaswamy, S., Tamayo, P., Rifkin, R., Mukherjee, S., Yeang, C.-H., Angelo, M., Ladd, C., Reich, M., Latulippe, E., Mesirov, J. P., et al. (2001) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.211566398 · Cited by 2,023
A new LDA-based face recognition system which can solve the small sample size problem
Chen, L.-F., Liao, H.-Y. M., Ko, M.-T., Lin, J.-C., & Yu, G.-J. (2000) · Pattern Recognition · doi:10.1016/s0031-3203(99)00139-9 · Cited by 1,427 · cited 3×
“In recent years, many approaches have been proposed to deal withthesingularityproblem,includingPCA LDA[1],regularized LDA (RLDA) [13], null space LDA (NLDA) [ 6 ], orthogonal centroid method (OCM) [22], uncorrelated LDA (ULDA) [32], orthogonal LDA (OLDA) [32], and LDA/GSVD [17].”
BackgroundMethodInfluentialAdaptive Control Processes: A Guided Tour
Morgenstern, D., & Bellman, R. (1962) · Econometrica · doi:10.2307/1909901 · Cited by 992
Molecular classification of human carcinomas by use of gene expression signatures.
Su, A. I., Welsh, J. B., Sapinoso, L. M., Kern, S. G., Dimitrov, P., Lapp, H., Schultz, P. G., Powell, S. M., Moskaluk, C. A., Frierson, H. F., et al. (2001) · PubMed · Cited by 753
Scheffer, T. (2005) · Encyclopedia of Data Warehousing and Mining · doi:10.4018/978-1-59140-557-3.ch192 · Cited by 624
Regularized linear discriminant analysis and its application in microarrays
Guo, Y., Hastie, T., & Tibshirani, R. (2007) · Biostatistics · doi:10.1093/biostatistics/kxj035 · Cited by 597 · cited 4×
“It has been used widely in many applications involving high-dimensional data [1], [6], [ 13 ], [20], [29], [32].”
MethodInfluentialDiscriminative common vectors for face recognition
Cevikalp, H., Neamtu, M., Wilkes, M., & Barkana, A. (2005) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2005.9 · Cited by 528 · cited 7×
“A similar result has been shown in [5] when all classes in the data set have a common sample size.”
BackgroundMethodResultInfluentialGeometric Representation of High Dimension, Low Sample Size Data
Hall, P., Marron, J. S., & Neeman, A. (2005) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.1467-9868.2005.00510.x · Cited by 483 · cited 1×
“Recent studies on the geometric representation of high-dimensional and small sample size data [ 15 ] show that undermildconditions,thecovariancematrix tendstoascaled identity matrix when the data dimension tends to infinity with the sample size fixed.”
BackgroundA unified framework for subspace face recognition
Wang, X., & Tang, X. (2004) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2004.57 · Cited by 359 · cited 1×
“It has been used widely in many applications involving high-dimensional data [1], [6], [13], [20], [29], [32].”
MethodYe, J. (2005) · Cited by 353
Self-citationSpectral Methods for Dimensionality Reduction
Saul, L. K., Weinberger, K. Q., Sha, F., Ham, J., & Lee, D. D. (2006) · Semi-Supervised Learning · doi:10.7551/mitpress/6173.003.0022 · Cited by 272
An optimal transformation for discriminant and principal component analysis
Duchene, J., & Leclercq, S. (1988) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/34.9121 · Cited by 203 · cited 1×
“Note that discriminant analysis with orthogonal transformation has been studied in [8].”
BackgroundHowland, P., Jeon, M., & Park, H. (2003) · SIAM Journal on Matrix Analysis and Applications · doi:10.1137/s0895479801393666 · Cited by 178 · cited 2×
“The features in the reduced space of ULDA are uncorrelated, whilethetransformation ofOLDAisorthogonal,i.e., . The LDA/GSVD algorithm proposed in [ 17 ] that overcomes the singularity problem via the generalized singular value decomposition (GSVD)[12] also belongs to this family.”
MethodWhere are linear feature extraction methods applicable?
Martínez, A. M., & Zhu, M. (2005) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2005.250 · Cited by 142
Lower Dimensional Representation of Text Data Based on Centroids and Least Squares
Park, H., Jeon, M., & Rosen, J. B. (2003) · BIT Numerical Mathematics · doi:10.1023/a:1026039313770 · Cited by 137
Geometric Methods for Feature Extraction and Dimensional Reduction
Burges, C. J. C. (2006) · Springer eBooks · doi:10.1007/0-387-25465-x_4 · Cited by 88
Semi-Supervised Protein Classification Using Cluster Kernels
Weston, J., Leslie, C. S., Ie, E., & Noble, W. S. (2006) · The MIT Press eBooks · doi:10.7551/mitpress/6173.003.0026 · Cited by 45
Gurunathan, R., Emden, B. V., Panchanathan, S., & Kumar, S. (2004) · BMC Bioinformatics · doi:10.1186/1471-2105-5-202 · Cited by 36 · cited 2×
“Fruitfly [ 14 ], [33] is the gene expression pattern image data set where the first three (correspond to stage ranges 1‐3, 4‐6, and 7‐8) and six (correspond to stage ranges 1‐3, 4‐6, 7‐8, 9‐10, 11‐12, and 13‐16) classes are used in the experiments.”
BackgroundPruning Noisy Bases in Discriminant Analysis
Manli, Z., & Martı́nez, A. M. (2008) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.2007.904040 · Cited by 28
Cited by
Classification in the Presence of Label Noise: A Survey
Frenay, B., & Verleysen, M. (2014) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2013.2292894 · Cited by 1,777
Feature dimensionality reduction: a review
Jia, W., Sun, M., Lian, J., & Hou, S. (2022) · Complex & Intelligent Systems · doi:10.1007/s40747-021-00637-x · Cited by 797
Metadata
- Authors
- Ji, S., & Ye, J.
- Venue
- IEEE Transactions on Neural Networks
- Published
- 2008-10
- Volume
- 19
- Issue
- 10
- Pages
- 1768-1782
- ISSN
- 1045-9227 (ISSN-L)
- 1941-0093
- OpenAlex
- W2169073444
- Cited by
- 186
- References
- 37
- FWCI
- 10.02
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.