The DOI matches its Crossref record; 49 of the 49 references it cites were found in a registry or an index.
Adapting to unknown sparsity by controlling the false discovery rate
Open accessAbstract
We attempt to recover an n-dimensional vector observed in white noise, where n is large and the vector is known to be sparse, but the degree of sparsity is unknown. We consider three different ways of defining sparsity of a vector: using the fraction of nonzero terms; imposing power-law decay bounds on the ordered entries; and controlling the ℓp norm for p small. We obtain a procedure which is asymptotically minimax for ℓr loss, simultaneously throughout a range of such sparsity classes. The optimal procedure is a data-adaptive thresholding scheme, driven by control of the false discovery rate (FDR). FDR control is a relatively recent innovation in simultaneous testing, ensuring that at most a certain expected fraction of the rejected null hypotheses will correspond to false rejections. In our treatment, the FDR control parameter qn also plays a determining role in asymptotic minimaxity. If q=lim qn∈[0,1/2] and also qn>γ/log(n), we get sharp asymptotic minimaxity, simultaneously, over a wide range of sparse parameter spaces and loss functions. On the other hand, q=lim qn∈(1/2,1] forces the risk to exceed the minimax risk by a factor growing with q. To our knowledge, this relation between ideas in simultaneous inference and asymptotic decision theory is new. Our work provides a new perspective on a class of model selection rules which has been introduced recently by several authors. These new rules impose complexity penalization of the form 2⋅log(potential model size/actual model sizes). We exhibit a close connection with FDR-controlling procedures under stringent control of the false discovery rate.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Controlling the False Discovery Rate: A Practical and Powerful Approach to Multiple Testing
Benjamini, Y., & Hochberg, Y. (1995) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.2517-6161.1995.tb02031.x · Cited by 109,482
Self-citationEstimating the Dimension of a Model
Schwarz, G. (1978) · The Annals of Statistics · doi:10.1214/aos/1176344136 · Cited by 47,033
An introduction to probability theory and its applications
(1958) · Journal of the Franklin Institute · doi:10.1016/0016-0032(58)90688-4 · Cited by 29,269
Information Theory and an Extension of the Maximum Likelihood Principle
Akaike, H. (1998) · Springer Series in Statistics · doi:10.1007/978-1-4612-1694-0_15 · Cited by 17,781
The control of the false discovery rate in multiple testing under dependency
Benjamini, Y., & Yekutieli, D. (2001) · The Annals of Statistics · doi:10.1214/aos/1013699998 · Cited by 10,490
Self-citationDe-noising by soft-thresholding
Donoho, D. L. (1995) · IEEE Transactions on Information Theory · doi:10.1109/18.382009 · Cited by 9,619
Self-citationA Direct Approach to False Discovery Rates
Storey, J. D. (2002) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/1467-9868.00346 · Cited by 5,603
Adapting to Unknown Smoothness via Wavelet Shrinkage
Donoho, D. L., & Johnstone, I. M. (1995) · Journal of the American Statistical Association · doi:10.1080/01621459.1995.10476626 · Cited by 4,630
Self-citationRelations between the statistics of natural images and the response properties of cortical cells
Field, D. J. (1987) · Journal of the Optical Society of America A · doi:10.1364/josaa.4.002379 · Cited by 3,241
Adaptive linear step-up procedures that control the false discovery rate
Benjamini, Y., Krieger, A. M., & Yekutieli, D. (2006) · Biometrika · doi:10.1093/biomet/93.3.491 · Cited by 3,228
Self-citationAn improved Bonferroni procedure for multiple tests of significance
SIMES, R. J. (1986) · Biometrika · doi:10.1093/biomet/73.3.751 · Cited by 2,396
Multiple Comparison Procedures
Hochberg, Y., & Tamhane, A. C. (1987) · Wiley Series in Probability and Statistics · doi:10.1002/9780470316672 · Cited by 2,389
Convergence of Stochastic Processes.
Robinson, P. M., & Pollard, D. (1985) · Economica · doi:10.2307/2553898 · Cited by 2,184
Multiple Comparison Procedures
(2013) · An Introduction to Statistical Concepts · doi:10.4324/9780203137819-17 · Cited by 2,109
Mallows, C. L. (1995) · Technometrics · doi:10.1080/00401706.1995.10484370 · Cited by 2,083
On the adaptive control of the false discovery rate in multiple testing with independent statistics
Benjamini, Y., & Hochberg, Y. (2000) · Journal of Educational and Behavioral Statistics · doi:10.3102/10769986025001060 · Cited by 2,015
Self-citationShaffer, J. P. (1995) · Annual Review of Psychology · doi:10.1146/annurev.ps.46.020195.003021 · Cited by 1,960
Convergence of Stochastic Processes
Pollard, D. (1984) · Springer Series in Statistics · doi:10.1007/978-1-4612-5254-2 · Cited by 1,841
Wavelet Shrinkage: Asymptopia?
Donoho, D. L., Johnstone, I. M., Kerkyacharian, G., & Picard, D. (1995) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.2517-6161.1995.tb02032.x · Cited by 1,749
Self-citationEmpirical Bayes analysis of a microarray experiment
Efron, B., Tibshirani, R., Storey, J. D., & Tusher, V. (2001) · Journal of the American Statistical Association · doi:10.1198/016214501753382129 · Cited by 1,733
Storey, J. D., Taylor, J. E., & Siegmund, D. (2004) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.1467-9868.2004.00439.x · Cited by 1,388
A stagewise rejective multiple test procedure based on a modified Bonferroni test
HOMMEL, G. (1988) · Biometrika · doi:10.1093/biomet/75.2.383 · Cited by 1,367
Mallows, C. L. (1995) · Technometrics · doi:10.2307/1269729 · Cited by 1,080
The statistics of natural images
Ruderman, D. (1994) · Network: Computation in Neural Systems · doi:10.1088/0954-898x/5/4/006 · Cited by 941
Image compression through wavelet transform coding
DeVore, R. A., Jawerth, B., & Lucier, B. J. (1992) · IEEE Transactions on Information Theory · doi:10.1109/18.119733 · Cited by 910
How Biased is the Apparent Error Rate of a Prediction Rule?
Efron, B. (1986) · Journal of the American Statistical Association · doi:10.1080/01621459.1986.10478291 · Cited by 587
The Risk Inflation Criterion for Multiple Regression
Foster, D. P., & George, E. I. (1994) · The Annals of Statistics · doi:10.1214/aos/1176325766 · Cited by 541
Calibration and empirical Bayes variable selection
George, E. (2000) · Biometrika · doi:10.1093/biomet/87.4.731 · Cited by 520
Statistics of natural images and models
Huang, J., & Mumford, D. (2003) · Proceedings. 1999 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (Cat. No PR00149) · doi:10.1109/cvpr.1999.786990 · Cited by 495
Birgé, L., & Massart, P. (2001) · Journal of the European Mathematical Society · doi:10.1007/s100970100031 · Cited by 479
Needles and straw in haystacks: Empirical Bayes estimates of possibly sparse sequences
Johnstone, I. M., & Silverman, B. W. (2004) · The Annals of Statistics · doi:10.1214/009053604000000030 · Cited by 456
Self-citationMaximum Entropy and the Nearly Black Object
Donoho, D. L., Johnstone, I. M., Hoch, J. C., & Stern, A. S. (1992) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.2517-6161.1992.tb01864.x · Cited by 393
Self-citationKwek, L. C., & Oh, C. H. (1998) · The European Physical Journal C · doi:10.1007/s100520050261 · Cited by 324
A stochastic process approach to false discovery control
Genovese, C., & Wasserman, L. (2004) · The Annals of Statistics · doi:10.1214/009053604000000283 · Cited by 291
Some Results on False Discovery Rate in Stepwise multiple testing procedures
Sarkar, S. K. (2002) · The Annals of Statistics · doi:10.1214/aos/1015362192 · Cited by 268
Ideal Denoising in an orthonormal basis chosen from a library of bases
Donoho, D. L., Johnstone, I. M., & X, E. (. B. (1994) · Cited by 244
Self-citationBenjamini, Y., & Liu, W. (1999) · Journal of Statistical Planning and Inference · doi:10.1016/s0378-3758(99)00040-3 · Cited by 242
Self-citationBayesian Denoising of Visual Images in the Wavelet Domain
Simoncelli, E. P. (1999) · Lecture Notes in Statistics · doi:10.1007/978-1-4612-0567-8_18 · Cited by 236
Minimax risk overl p -balls forl p -error
Donoho, D. L., & Johnstone, I. M. (1994) · Probability Theory and Related Fields · doi:10.1007/bf01199026 · Cited by 218
Self-citationAdaptive thresholding of wavelet coefficients
Abramovich, F., & Benjamini, Y. (1996) · Computational Statistics & Data Analysis · doi:10.1016/0167-9473(96)00003-5 · Cited by 179
Self-citationHow Many Variables Should Be Entered in a Regression Equation?
Breiman, L., & Freedman, D. (1983) · Journal of the American Statistical Association · doi:10.1080/01621459.1983.10477941 · Cited by 164
The Covariance Inflation Criterion for Adaptive Model Selection
Tibshirani, R. J., & Knight, K. (1999) · Journal of the Royal Statistical Society Series B (Statistical Methodology) · doi:10.1111/1467-9868.00191 · Cited by 135
Keselman, H. J., Cribbie, R. A., & Holland, B. S. (1999) · Psychological Methods · doi:10.1037/1082-989x.4.1.58 · Cited by 118
Thresholding of Wavelet Coefficients as Multiple Hypotheses Testing Procedure
Abramovich, F., & Benjamini, Y. (1995) · Lecture notes in statistics · doi:10.1007/978-1-4612-2544-7_1 · Cited by 85
Self-citationA Note on a Method for the Analysis of Significances en masse
Seeger, P. (1968) · Technometrics · doi:10.1080/00401706.1968.10490605 · Cited by 60
Hommel, G., & Bernhard, G. (1993) · Contributions to statistics · doi:10.1007/978-3-642-99766-2_10 · Cited by 58
Minimax Bayes, Asymptotic Minimax and Sparse Wavelet Priors
Johnstone, I. M. (1994) · Springer eBooks · doi:10.1007/978-1-4612-2618-5_23 · Cited by 54
Self-citationLocal asymptotic coding and the minimum description length
Foster, D. P., & Stine, R. A. (1999) · IEEE Transactions on Information Theory · doi:10.1109/18.761287 · Cited by 22
An Information Theoretic Comparison of Model Selection CriteriaPreprint
Foster, D. P., & Stine, R. A. (1997) · RePEc: Research Papers in Economics · Cited by 8
Cited by
Efron, B., Hastie, T., Johnstone, I., & Tibshirani, R. (2004) · The Annals of Statistics · doi:10.1214/009053604000000067 · Cited by 9,357
Self-citationAdaptive linear step-up procedures that control the false discovery rate
Benjamini, Y., Krieger, A. M., & Yekutieli, D. (2006) · Biometrika · doi:10.1093/biomet/93.3.491 · Cited by 3,228
Self-citationIdentifying differentially expressed genes using false discovery rate controlling procedures
Reiner, A., Yekutieli, D., & Benjamini, Y. (2003) · Bioinformatics · doi:10.1093/bioinformatics/btf877 · Cited by 1,765
Self-citationDonoho, D. L., Tsaig, Y., Drori, I., & Starck, J.-L. (2012) · IEEE Transactions on Information Theory · doi:10.1109/tit.2011.2173241 · Cited by 1,509
Self-citationSparse Partial Least Squares Regression for Simultaneous Dimension Reduction and Variable Selection
Chun, H., & Keleş, S. (2010) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.1467-9868.2009.00723.x · Cited by 908
Metadata
- Authors
- Abramovich, F., Benjamini, Y., Donoho, D. L., & Johnstone, I. M.
- Venue
- The Annals of Statistics
- Publisher
- Institute of Mathematical Statistics
- Published
- 2006-04-01
- Volume
- 34
- Issue
- 2
- Pages
- 584-653
- ISSN
- 0090-5364 (ISSN-L)
- OpenAlex
- W2133264613
- Cited by
- 455
- References
- 49
- FWCI
- 29.59
- Type
- journal article
- Language
- en
- Open access
- bronze
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.