The DOI matches its Crossref record; 49 of the 49 references it cites were found in a registry or an index.
Adapting to unknown sparsity by controlling the false discovery rate
Open accessAbstract
We attempt to recover an n-dimensional vector observed in white noise, where n is large and the vector is known to be sparse, but the degree of sparsity is unknown. We consider three different ways of defining sparsity of a vector: using the fraction of nonzero terms; imposing power-law decay bounds on the ordered entries; and controlling the ℓp norm for p small. We obtain a procedure which is asymptotically minimax for ℓr loss, simultaneously throughout a range of such sparsity classes. The optimal procedure is a data-adaptive thresholding scheme, driven by control of the false discovery rate (FDR). FDR control is a relatively recent innovation in simultaneous testing, ensuring that at most a certain expected fraction of the rejected null hypotheses will correspond to false rejections. In our treatment, the FDR control parameter qn also plays a determining role in asymptotic minimaxity. If q=lim qn∈[0,1/2] and also qn>γ/log(n), we get sharp asymptotic minimaxity, simultaneously, over a wide range of sparse parameter spaces and loss functions. On the other hand, q=lim qn∈(1/2,1] forces the risk to exceed the minimax risk by a factor growing with q. To our knowledge, this relation between ideas in simultaneous inference and asymptotic decision theory is new. Our work provides a new perspective on a class of model selection rules which has been introduced recently by several authors. These new rules impose complexity penalization of the form 2⋅log(potential model size/actual model sizes). We exhibit a close connection with FDR-controlling procedures under stringent control of the false discovery rate.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Controlling the False Discovery Rate: A Practical and Powerful Approach to Multiple Testing
Benjamini, Y., & Hochberg, Y. (1995) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.2517-6161.1995.tb02031.x · Cited by 109,482
Self-citationEstimating the Dimension of a Model
Schwarz, G. (1978) · The Annals of Statistics · doi:10.1214/aos/1176344136 · Cited by 47,033
An introduction to probability theory and its applications
(1958) · Journal of the Franklin Institute · doi:10.1016/0016-0032(58)90688-4 · Cited by 29,269
Information Theory and an Extension of the Maximum Likelihood Principle
Akaike, H. (1998) · Springer Series in Statistics · doi:10.1007/978-1-4612-1694-0_15 · Cited by 17,781
The control of the false discovery rate in multiple testing under dependency
Benjamini, Y., & Yekutieli, D. (2001) · The Annals of Statistics · doi:10.1214/aos/1013699998 · Cited by 10,490
Self-citationDe-noising by soft-thresholding
Donoho, D. L. (1995) · IEEE Transactions on Information Theory · doi:10.1109/18.382009 · Cited by 9,619
Self-citationA Direct Approach to False Discovery Rates
Storey, J. D. (2002) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/1467-9868.00346 · Cited by 5,603
Adapting to Unknown Smoothness via Wavelet Shrinkage
Donoho, D. L., & Johnstone, I. M. (1995) · Journal of the American Statistical Association · doi:10.1080/01621459.1995.10476626 · Cited by 4,630
Self-citationRelations between the statistics of natural images and the response properties of cortical cells
Field, D. J. (1987) · Journal of the Optical Society of America A · doi:10.1364/josaa.4.002379 · Cited by 3,241
Adaptive linear step-up procedures that control the false discovery rate
Benjamini, Y., Krieger, A. M., & Yekutieli, D. (2006) · Biometrika · doi:10.1093/biomet/93.3.491 · Cited by 3,228
Self-citation
Cited by
Efron, B., Hastie, T., Johnstone, I., & Tibshirani, R. (2004) · The Annals of Statistics · doi:10.1214/009053604000000067 · Cited by 9,357
Self-citationAdaptive linear step-up procedures that control the false discovery rate
Benjamini, Y., Krieger, A. M., & Yekutieli, D. (2006) · Biometrika · doi:10.1093/biomet/93.3.491 · Cited by 3,228
Self-citationIdentifying differentially expressed genes using false discovery rate controlling procedures
Reiner, A., Yekutieli, D., & Benjamini, Y. (2003) · Bioinformatics · doi:10.1093/bioinformatics/btf877 · Cited by 1,765
Self-citationDonoho, D. L., Tsaig, Y., Drori, I., & Starck, J.-L. (2012) · IEEE Transactions on Information Theory · doi:10.1109/tit.2011.2173241 · Cited by 1,509
Self-citationSparse Partial Least Squares Regression for Simultaneous Dimension Reduction and Variable Selection
Chun, H., & Keleş, S. (2010) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.1467-9868.2009.00723.x · Cited by 908
Covariance regularization by thresholding
Bickel, P. J., & Levina, E. (2008) · The Annals of Statistics · doi:10.1214/08-aos600 · Cited by 893
A Review of Feature Selection Methods for Machine Learning-Based Disease Risk Prediction
Pudjihartono, N., Fadason, T., Kempa-Liehr, A. W., & O'Sullivan, J. M. (2022) · Frontiers in Bioinformatics · doi:10.3389/fbinf.2022.927312 · Cited by 880
Higher criticism for detecting sparse heterogeneous mixtures
Donoho, D., & Jin, J. (2004) · The Annals of Statistics · doi:10.1214/009053604000000265 · Cited by 721
Self-citationDiscovering the False Discovery Rate
Benjamini, Y. (2010) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.1467-9868.2010.00746.x · Cited by 629
Self-citationThe Nonparanormal: Semiparametric Estimation of High Dimensional Undirected Graphs
Liu, H., Lafferty, J., & Wasserman, L. (2018) · doi:10.1184/r1/6610712.v1 · Cited by 621
A two-sample test for high-dimensional data with applications to gene-set testing
Chen, S. X., & Qin, Y.-L. (2010) · The Annals of Statistics · doi:10.1214/09-aos716 · Cited by 574
Operating Characteristics and Extensions of the False Discovery Rate Procedure
Genovese, C., & Wasserman, L. (2002) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/1467-9868.00347 · Cited by 561
Needles and straw in haystacks: Empirical Bayes estimates of possibly sparse sequences
Johnstone, I. M., & Silverman, B. W. (2004) · The Annals of Statistics · doi:10.1214/009053604000000030 · Cited by 456
Self-citationAIC, BIC and Recent Advances in Model Selection
Chakrabarti, A., & Ghosh, J. K. (2011) · Philosophy of Statistics · doi:10.1016/b978-0-444-51862-0.50018-6 · Cited by 362
From Local Kernel to Nonlocal Multiple-Model Image Denoising
Katkovnik, V., Foi, A., Egiazarian, K., & Astola, J. (2010) · International Journal of Computer Vision · doi:10.1007/s11263-009-0272-7 · Cited by 353
Wavelets, Ridgelets, and Curvelets for Poisson Noise Removal
Zhang, B., Fadili, J. M., & Starck, J. L. (2008) · IEEE Transactions on Image Processing · doi:10.1109/tip.2008.924386 · Cited by 349
Empirical Bayes selection of wavelet thresholds
Johnstone, I. M., & Silverman, B. W. (2005) · The Annals of Statistics · doi:10.1214/009053605000000345 · Cited by 340
Self-citationOn the Benjamini–Hochberg method
Ferreira, J. A., & Zwinderman, A. H. (2006) · The Annals of Statistics · doi:10.1214/009053606000000425 · Cited by 318
Wavelet Estimators in Nonparametric Regression: A Comparative Simulation Study
Antoniadis, A., Bigot, J., & Sapatinas, T. (2001) · Journal of Statistical Software · doi:10.18637/jss.v006.i06 · Cited by 301
Minimal Penalties for Gaussian Model Selection
Birgé, L., & Massart, P. (2007) · Probability Theory and Related Fields · doi:10.1007/s00440-006-0011-8 · Cited by 299
A stochastic process approach to false discovery control
Genovese, C., & Wasserman, L. (2004) · The Annals of Statistics · doi:10.1214/009053604000000283 · Cited by 291
Sparse PCA: Optimal rates and adaptive estimation
Cai, T. T., Ma, Z., & Wu, Y. (2013) · The Annals of Statistics · doi:10.1214/13-aos1178 · Cited by 283
Bayesian linear regression with sparse priors
Castillo, I., Schmidt-Hieber, J., & van der Vaart, A. (2015) · The Annals of Statistics · doi:10.1214/15-aos1334 · Cited by 277
SLOPE—Adaptive variable selection via convex optimization
Bogdan, M., van den Berg, E., Sabatti, C., Su, W., & Candès, E. J. (2015) · The Annals of Applied Statistics · doi:10.1214/15-aoas842 · Cited by 277
Technical trading revisited: False discoveries, persistence tests, and transaction costs
Bajgrowicz, P., & Scaillet, O. (2012) · Journal of Financial Economics · doi:10.1016/j.jfineco.2012.06.001 · Cited by 253
Some Results on False Discovery Rate in Stepwise multiple testing procedures
Sarkar, S. K. (2002) · The Annals of Statistics · doi:10.1214/aos/1015362192 · Cited by 268
Sparsity oracle inequalities for the Lasso
Bunea, F., Tsybakov, A., & Wegkamp, M. (2007) · Electronic Journal of Statistics · doi:10.1214/07-ejs008 · Cited by 253
Xie, Y., Pan, W., & Khodursky, A. B. (2005) · Bioinformatics · doi:10.1093/bioinformatics/bti685 · Cited by 125
Metadata
- Authors
- Abramovich, F., Benjamini, Y., Donoho, D. L., & Johnstone, I. M.
- Venue
- The Annals of Statistics
- Publisher
- Institute of Mathematical Statistics
- Published
- 2006-04-01
- Volume
- 34
- Issue
- 2
- Pages
- 584-653
- ISSN
- 0090-5364 (ISSN-L)
- OpenAlex
- W2133264613
- Cited by
- 455
- References
- 49
- FWCI
- 29.59
- Type
- journal article
- Language
- en
- Open access
- bronze
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.