The DOI matches its Crossref record; 30 of the 30 references it cites were found in a registry or an index.
High-dimensional classification using features annealed independence rules
Open accessAbstract
Classification using high-dimensional features arises frequently in many contemporary statistical studies such as tumor classification using microarray or other high-throughput data. The impact of dimensionality on classifications is largely poorly understood. In a seminal paper, Bickel and Levina (2004) show that the Fisher discriminant performs poorly due to diverging spectra and they propose to use the independence rule to overcome the problem. We first demonstrate that even for the independence classification rule, classification using all the features can be as bad as the random guessing due to noise accumulation in estimating population centroids in high-dimensional feature space. In fact, we demonstrate further that almost all linear discriminants can perform as bad as the random guessing. Thus, it is paramountly important to select a subset of important features for high-dimensional classification, resulting in Features Annealed Independence Rules (FAIR). The conditions under which all the important features can be selected by the two-sample t-statistic are established. The choice of the optimal number of features, or equivalently, the threshold value of the test statistics are proposed based on an upper bound of the classification error. Simulation studies and real data analysis support our theoretical results and demonstrate convincingly the advantage of our new classification procedure.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Golub, T. R., Slonim, D. K., Tamayo, P., Huard, C., Gaasenbeek, M., Mesirov, J. P., Coller, H., Loh, M. L., Downing, J. R., Caligiuri, M. A., et al. (1999) · Science · doi:10.1126/science.286.5439.531 · Cited by 11,516
Weak Convergence and Empirical Processes
van der Vaart, A. W., & Wellner, J. A. (1996) · Springer Series in Statistics · doi:10.1007/978-1-4757-2545-2 · Cited by 5,014
Sparse Principal Component Analysis
Zou, H., Hastie, T., & Tibshirani, R. (2006) · Journal of Computational and Graphical Statistics · doi:10.1198/106186006x113430 · Cited by 3,152
Diagnosis of multiple cancer types by shrunken centroids of gene expression
Tibshirani, R., Hastie, T., Narasimhan, B., & Chu, G. (2002) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.082099299 · Cited by 2,930
Sure Independence Screening for Ultrahigh Dimensional Feature Space
Fan, J., & Lv, J. (2008) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.1467-9868.2008.00674.x · Cited by 2,711
Self-citationComparison of Discrimination Methods for the Classification of Tumors Using Gene Expression Data
Dudoit, S., Fridlyand, J., & Speed, T. P. (2002) · Journal of the American Statistical Association · doi:10.1198/016214502753479248 · Cited by 2,672
Gene expression correlates of clinical prostate cancer behavior
Singh, D., Febbo, P. G., Ross, K., Jackson, D. G., Manola, J., Ladd, C., Tamayo, P., Renshaw, A. A., D'Amico, A. V., Richie, J. P., et al. (2002) · Cancer Cell · doi:10.1016/s1535-6108(02)00030-2 · Cited by 2,512
Regularized Discriminant Analysis
Friedman, J. H. (1989) · Journal of the American Statistical Association · doi:10.1080/01621459.1989.10478752 · Cited by 2,421
Predicting the clinical status of human breast cancer by using gene expression profiles
West, M., Blanchette, C., Dressman, H., Huang, E., Ishida, S., Spang, R., Zuzan, H., Olson, J. A., Marks, J. R., & Nevins, J. R. (2001) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.201162998 · Cited by 1,318
Gordon, G. J., Jensen, R. V., Hsiao, L.-L., Gullans, S. R., Blumenstock, J. E., Ramaswamy, S., Richards, W. G., Sugarbaker, D. J., & Bueno, R. (2002) · PubMed · Cited by 859
Boosting With the L2Loss: Regression and Classification
Bühlmann, P., & Yu, B. (2003) · Journal of the American Statistical Association · doi:10.1198/016214503000125 · Cited by 845
Prediction by Supervised Principal Components
Bair, E., Hastie, T., Paul, D., & Tibshirani, R. (2006) · Journal of the American Statistical Association · doi:10.1198/016214505000000628 · Cited by 817
Tumor classification by partial least squares using microarray gene expression data
Nguyen, D. V., & Rocke, D. M. (2002) · Bioinformatics · doi:10.1093/bioinformatics/18.1.39 · Cited by 816
Weak Convergence and Empirical Processes
Mikosch, T., der Vaart, A. W. V., & Wellner, J. A. (1997) · Journal of the American Statistical Association · doi:10.2307/2965734 · Cited by 765
Bickel, P. J., & Levina, E. (2004) · Bernoulli · doi:10.3150/bj/1106314847 · Cited by 548
BagBoosting for tumor classification with gene expression data
Dettling, M. (2004) · Bioinformatics · doi:10.1093/bioinformatics/bth447 · Cited by 542
Meinshausen, N. (2007) · Computational Statistics & Data Analysis · doi:10.1016/j.csda.2006.12.019 · Cited by 492
Geometric Representation of High Dimension, Low Sample Size Data
Hall, P., Marron, J. S., & Neeman, A. (2005) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.1467-9868.2005.00510.x · Cited by 483
Persistence in high-dimensional linear predictor selection and the virtue of overparametrization
Greenshtein, E., & Ritov, Y. (2004) · Bernoulli · doi:10.3150/bj/1106314846 · Cited by 362
Statistical challenges with high dimensionality: feature selection in knowledge discovery
Fan, J., & Li, R. (2007) · Proceedings of the International Congress of Mathematicians Madrid, August 22–30, 2006 · doi:10.4171/022-3/31 · Cited by 348
Self-citationBoosting for tumor classification with gene expression data
Dettling, M., & Bühlmann, P. (2003) · Bioinformatics · doi:10.1093/bioinformatics/btf867 · Cited by 318
Distance-Weighted Discrimination
Marron, J. S., Todd, M. J., & Ahn, J. (2007) · Journal of the American Statistical Association · doi:10.1198/016214507000001120 · Cited by 316
Test of Significance Based on Wavelet Thresholding and Neyman's Truncation
Fan, J. (1996) · Journal of the American Statistical Association · doi:10.1080/01621459.1996.10476936 · Cited by 288
Self-citationPLS Dimension Reduction for Classification with Microarray Data
Boulesteix, A.-L. (2004) · Statistical Applications in Genetics and Molecular Biology · doi:10.2202/1544-6115.1075 · Cited by 210
Limit theory for mixing dependent random variables
Lin, Z., & Lu, C. (1996) · Cited by 195
Effective dimension reduction methods for tumor classification using gene expression data
Antoniadis, A., Lambert-Lacroix, S., & Leblanc, F. (2003) · Bioinformatics · doi:10.1093/bioinformatics/btg062 · Cited by 185
To How Many Simultaneous Hypothesis Tests Can Normal, Student'stor Bootstrap Calibration Be Applied?
Fan, J., Hall, P. A., & Yao, Q. (2007) · Journal of the American Statistical Association · doi:10.1198/016214507000000969 · Cited by 84
Self-citationDimension reduction strategies for analyzing global gene expression data with a response
Chiaromonte, F., & Martinelli, J. (2002) · Mathematical Biosciences · doi:10.1016/s0025-5564(01)00106-7 · Cited by 82
Greenshtein, E. (2006) · Cited by 80
Statistical Analysis of DNA Microarray Data in Cancer Research
Fan, J., & Ren, Y. (2006) · Clinical Cancer Research · doi:10.1158/1078-0432.ccr-06-1033 · Cited by 45
Self-citation
Cited by
Sure Independence Screening for Ultrahigh Dimensional Feature Space
Fan, J., & Lv, J. (2008) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.1467-9868.2008.00674.x · Cited by 2,711
Self-citationChallenges of Big Data analysis
Fan, J., Han, F., & Liu, H. (2014) · National Science Review · doi:10.1093/nsr/nwt032 · Cited by 1,482
Self-citationSure independence screening in generalized linear models with NP-dimensionality
Fan, J., & Song, R. (2010) · The Annals of Statistics · doi:10.1214/10-aos798 · Cited by 630
Self-citationHuman–Agent Teaming for Multirobot Control: A Review of Human Factors Issues
Chen, J. Y. C., & Barnes, M. J. (2014) · IEEE Transactions on Human-Machine Systems · doi:10.1109/thms.2013.2293535 · Cited by 522
Nonconcave Penalized Likelihood With NP-Dimensionality
Fan, J., & Lv, J. (2011) · IEEE Transactions on Information Theory · doi:10.1109/tit.2011.2158486 · Cited by 427
Self-citation
Metadata
- Authors
- Fan, J., & Fan, Y.
- Venue
- The Annals of Statistics
- Publisher
- Institute of Mathematical Statistics
- Published
- 2008-12-01
- Volume
- 36
- Issue
- 6
- Pages
- 2605-2637
- ISSN
- 0090-5364 (ISSN-L)
- OpenAlex
- W2049701820
- Cited by
- 480
- References
- 30
- FWCI
- 16.14
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.