Correlation and Large-Scale Simultaneous Significance Testing
Abstract
Large-scale hypothesis testing problems, with hundreds or thousands of test statistics zi to consider at once, have become familiar in current practice. Applications of popular analysis methods, such as false discovery rate techniques, do not require independence of the zi's, but their accuracy can be compromised in high-correlation situations. This article presents computational and theoretical methods for assessing the size and effect of correlation in large-scale testing. A simple theory leads to the identification of a single omnibus measure of correlation for the zi's order statistic. The theory relates to the correct choice of a null distribution for simultaneous significance testing and its effect on inference.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Pre-specified Bonferroni-like corrections on association p-values are not likely to be appropriate given possible correlations between defined groups of rare variants and/or overlapping windows to be tested.
“Pre-specified Bonferroni-like corrections on association p-values are not likely to be appropriate given possible correlations between defined groups of rare variants and/or overlapping windows to be tested.”
From Statistical analysis strategies for association studies involving rare variants (2010) · cites this paper as [99]
Cites
Controlling the False Discovery Rate: A Practical and Powerful Approach to Multiple Testing
Benjamini, Y., & Hochberg, Y. (1995) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.2517-6161.1995.tb02031.x · Cited by 109,482 · see citing works
The control of the false discovery rate in multiple testing under dependency
Benjamini, Y., & Yekutieli, D. (2001) · The Annals of Statistics · doi:10.1214/aos/1013699998 · Cited by 10,490 · see citing works
Bolstad, B. M., Irizarry, R. A., Åstrand, M., & Speed, T. P. (2003) · Bioinformatics · doi:10.1093/bioinformatics/19.2.185 · Cited by 8,185 · see citing works
A Direct Approach to False Discovery Rates
Storey, J. D. (2002) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/1467-9868.00346 · Cited by 5,603 · see citing works
Resampling-Based Multiple Testing: Examples and Methods for p-Value Adjustment
Martín, M. A., Westfall, P. H., & Young, S. (1993) · Biometrics · doi:10.2307/2533464 · Cited by 2,308 · see citing works
Identifying differentially expressed genes using false discovery rate controlling procedures
Reiner, A., Yekutieli, D., & Benjamini, Y. (2003) · Bioinformatics · doi:10.1093/bioinformatics/btf877 · Cited by 1,765 · see citing works
Empirical Bayes analysis of a microarray experiment
Efron, B., Tibshirani, R., Storey, J. D., & Tusher, V. (2001) · Journal of the American Statistical Association · doi:10.1198/016214501753382129 · Cited by 1,733 · see citing works
Self-citationStorey, J. D., Taylor, J., & Siegmund, D. (2003) · Journal of the Royal Statistical Society Series B (Statistical Methodology) · doi:10.1111/j.1467-9868.2004.00439.x · Cited by 1,345 · see citing works
Multiple Comparisons: Theory and Methods
Christensen, R., & Hsu, J. C. (1997) · Technometrics · doi:10.2307/1270920 · Cited by 1,105 · see citing works
Large-Scale Simultaneous Hypothesis Testing: The Choice of a Null Hypothesis
Efron, B. (2004) · Journal of the American Statistical Association · doi:10.1198/016214504000000089 · Cited by 1,005 · see citing works
Self-citationGene-expression profiles in hereditary breast cancer.
Hedenfalk, I., Duggan, D., Chen, Y.-D., Radmacher, M., Bittner, M., Simon, R., Meltzer, P., Gusterson, B., Esteller, M., Raffeld, M., et al. (2001) · New England Journal of Medicine · doi:10.1056/nejm200102223440801 · Cited by 978 · see citing works
Outcome signature genes in breast cancer: is there a unique set?
Ein‐Dor, L., Kela, I., Getz, G. A., Givol, D., & Domany, E. (2004) · Bioinformatics · doi:10.1093/bioinformatics/bth469 · Cited by 792 · see citing works
Empirical Bayes methods and false discovery rates for microarrays
Efron, B., & Tibshirani, R. (2002) · Genetic Epidemiology · doi:10.1002/gepi.1124 · Cited by 684 · see citing works
Self-citationResampling-Based Multiple Testing: Examples and Methods for p-Value Adjustment.
Thomas, G. E., Westfall, P. H., & Young, S. S. (1994) · Journal of the Royal Statistical Society Series D (The Statistician) · doi:10.2307/2348369 · Cited by 515 · see citing works
Resampling-based multiple testing for microarray data analysis
Ge, Y., Dudoit, S., & Speed, T. P. (2003) · Test · doi:10.1007/bf02595811 · Cited by 506 · see citing works
Plots of P-values to evaluate many tests simultaneously
Schweder, T., & Spjøtvoll, E. (1982) · Biometrika · doi:10.1093/biomet/69.3.493 · Cited by 440 · see citing works
Size, power and false discovery rates
Efron, B. (2007) · The Annals of Statistics · doi:10.1214/009053606000001460 · Cited by 436 · see citing works
Self-citationEstimating the Proportion of True Null Hypotheses, with application to DNA Microarray Data
Langaas, M., Lindqvist, B. H., & Ferkingstad, E. (2005) · Journal of the Royal Statistical Society Series B (Statistical Methodology) · doi:10.1111/j.1467-9868.2005.00515.x · Cited by 316 · see citing works
Generalizations of the Familywise Error Rate
Lehmann, E. L., & Romano, J. P. (2011) · Springer eBooks · doi:10.1007/978-1-4614-1412-4_57 · Cited by 253 · see citing works
Multiple Testing of General Contrasts Using Logical Constraints and Correlations
Westfall, P. H. (1997) · Journal of the American Statistical Association · doi:10.1080/01621459.1997.10473627 · Cited by 223 · see citing works
Using specially designed exponential families for density estimation
Efron, B., & Tibshirani, R. J. (1996) · The Annals of Statistics · doi:10.1214/aos/1032181161 · Cited by 176 · see citing works
Self-citationCellular Gene Expression upon Human Immunodeficiency Virus Type 1 Infection of CD4+-T-Cell Lines
Wout, A. B. V. ’., Lehrman, G. K., Mikheeva, S. A., OʼKeeffe, G., Katze, M. G., Bumgarner, R. E., Geiss, G. K., & Mullins, J. I. (2002) · Journal of Virology · doi:10.1128/jvi.77.2.1392-1402.2003 · Cited by 176 · see citing works
Multiple Testing. Part I. Single-Step Procedures for Control of General Type I Error Rates
Dudoit, S., van der Laan, M. J., & Pollard, K. S. (2004) · Statistical Applications in Genetics and Molecular Biology · doi:10.2202/1544-6115.1040 · Cited by 167 · see citing works
Detecting Differentially Expressed Genes in Microarrays Using Bayesian Model Selection
Ishwaran, H., & Rao, J. S. (2003) · Journal of the American Statistical Association · doi:10.1198/016214503000224 · Cited by 165 · see citing works
Variance of the Number of False Discoveries
Owen, A. B. (2005) · Journal of the Royal Statistical Society Series B (Statistical Methodology) · doi:10.1111/j.1467-9868.2005.00509.x · Cited by 142 · see citing works
The effects of normalization on the correlation structure of microarray data
Qiu, X., Brooks, A. I., Klebanov, L. B., & Yakovlev, A. (2005) · BMC Bioinformatics · doi:10.1186/1471-2105-6-120 · Cited by 111 · see citing works
The Factor Analytic Approach to Simultaneous Inference in the General Linear Model
Hsu, J. C. (1992) · Journal of Computational and Graphical Statistics · doi:10.1080/10618600.1992.10477011 · Cited by 108 · see citing works
Qiu, X., Klebanov, L. B., & Yakovlev, A. (2005) · Statistical Applications in Genetics and Molecular Biology · doi:10.2202/1544-6115.1157 · Cited by 108 · see citing works
Efron, B. (2010) · Cambridge University Press eBooks · doi:10.1017/cbo9780511761362.006 · Cited by 99 · see citing works
Self-citationBias in the estimation of false discovery rate in microarray studies
Pawitan, Y., Murthy, K. R. K., Michiels, S., et al. (2005) · Computer applications in the biosciences · doi:10.1093/bioinformatics/bti626 · Cited by 84 · see citing works
American Statistical Association
Leone, F. C. (2005) · Encyclopedia of Statistical Sciences · doi:10.1002/0471667196.ess0036.pub2 · Cited by 70 · see citing works
Gentle, J. E. (2010) · Elsevier eBooks · doi:10.1016/b978-0-08-044894-7.01316-6 · Cited by 56 · see citing works
Quantile-Function Based Null Distribution in Resampling Based Multiple Testing
van der Laan, M. J., & Hubbard, A. (2006) · Statistical Applications in Genetics and Molecular Biology · doi:10.2202/1544-6115.1199 · Cited by 27 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
The ENCODE Project Consortium (2007) · Nature · doi:10.1038/nature05874 · Cited by 5,223 · see citing works
Capturing Heterogeneity in Gene Expression Studies by Surrogate Variable Analysis
Leek, J. T., & Storey, J. D. (2007) · PLoS Genetics · doi:10.1371/journal.pgen.0030161 · Cited by 1,930 · see citing works
Assigning Significance to Peptides Identified by Tandem Mass Spectrometry Using Decoy Databases
Käll, L., Storey, J. D., MacCoss, M. J., & Noble, W. S. (2008) · Journal of Proteome Research · doi:10.1021/pr700600n · Cited by 607 · see citing works
Statistical analysis strategies for association studies involving rare variants
Bansal, V., Libiger, O., Torkamani, A., & Schork, N. J. (2010) · Nature Reviews Genetics · doi:10.1038/nrg2867 · Cited by 455 · see citing works
Sartor, M. A., Tomlinson, C. R., Wesselkamper, S. C., Sivaganesan, S., Leikauf, G. D., & Medvedovic, M. (2006) · BMC Bioinformatics · doi:10.1186/1471-2105-7-538 · Cited by 263 · see citing works
Estimating False Discovery Proportion Under Arbitrary Covariance Dependence
Fan, J., Han, X., & Gu, W. (2012) · Journal of the American Statistical Association · doi:10.1080/01621459.2012.720478 · Cited by 227 · see citing works
Wang, H.-Q., Tuominen, L. K., & Tsai, C.-J. (2011) · Bioinformatics · doi:10.1093/bioinformatics/btq650 · Cited by 223 · see citing works
Effects of dependence in high-dimensional multiple testing problems
Kim, K. I., & van de Wiel, M. A. (2008) · BMC Bioinformatics · doi:10.1186/1471-2105-9-114 · Cited by 220 · see citing works
“The variance of pairwise correlations was previously described as an important parameter in FDR estimation [20].”
InfluentialPaternal adverse childhood experiences: Associations with infant DNA methylation
Merrill, S. M., Moore, S. R., Gladish, N., Giesbrecht, G. F., Dewey, D., Konwar, C., MacIssac, J. L., Kobor, M. S., & Letourneau, N. L. (2021) · Developmental Psychobiology · doi:10.1002/dev.22174 · Cited by 42 · see citing works
Police-Community Relations in Cincinnati
Ridgeway, G., Schell, T., Gifford, B., Saunders, J., Turner, S., Riley, K., & Dixon, T. (2009) · RAND Corporation eBooks · doi:10.7249/mg853 · Cited by 33 · see citing works
- Authors
- Efron, B.
- Affiliations
- Stanford University (US)
- Corresponding author
- Efron, Bradley
- Venue
- Journal of the American Statistical Association
- Publisher
- Informa UK Limited
- Published
- 2007-03
- Volume
- 102
- Issue
- 477
- Pages
- 93-103
- ISSN
- 0162-1459 (ISSN-L)
- 1537-274X
- OpenAlex
- W2006145698
- Cited by
- 405
- References
- 33
- Citations by year
- 2026: 6 · 2025: 13 · 2024: 18 · 2023: 19 · 2022: 16 · 2021: 18 · 2020: 16 · 2019: 17 · 2018: 22 · 2017: 18 · 2016: 18 · 2015: 14 · 2014: 32 · 2013: 16 · 2012: 24
- FWCI
- 31.13
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Journal of the American Statistical Association (publishedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- Topics
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods and Inference
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Clinical Trials
- Physical Sciences › Mathematics › Statistics and Probability › Probability and Statistical Research
- Keywords
- multiple comparisons, false discovery rate, null distribution, order statistics
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review