On the adaptive control of the false discovery rate in multiple testing with independent statistics
Abstract
A new approach to problems of multiple significance testing was presented in Benjamini and Hochberg (1995), which calls for controlling the expected ratio of the number of erroneous rejections to the number of rejections–the False Discovery Rate (FDR). The procedure given there was shown to control the FDR for independent test statistics. When some of the hypotheses are in fact false, that procedure is too conservative. We present here an adaptive procedure, where the number of true null hypotheses is estimated first as in Hochberg and Benjamini (1990), and this estimate is used in the procedure of Benjamini and Hochberg (1995). The result is still a simple stepwise procedure, to which we also give a graphical companion. The new procedure is used in several examples drawn from educational and behavioral studies, addressing problems in multi-center studies, subset analysis and meta-analysis. The examples vary in the number of hypotheses tested, and the implication of the new procedure on the conclusions. In a large simulation study of independent test statistics the adaptive procedure is shown to control the FDR and have substantially better power than the previously suggested FDR controlling method, which by itself is more powerful than the traditional family wise error-rate controlling methods. In cases where most of the tested hypotheses are far from being true there is hardly any penalty due to the simultaneous testing of many hypotheses.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
p-values were adjusted for multiple testing by controlling the false discovery rate according to the method of Benjamini and Hochberg.
“p-values were adjusted for multiple testing by controlling the false discovery rate (FDR) according to the method of Benjamini and Hochberg [44].”
From Shared and distinct microRNA profiles between HT22, N2A and SH-SY5Y cell lines and primary mouse hippocampal neurons (2025) · cites this paper as [44]
In mouse and rat hippocampal neurons, miR-103, miR-130a, miR-130b, miR-134, miR-376 and miR-449 are in the group with intermediate expression.
“and miR-103, miR-130a, miR-130b, miR-134, miR-376 and miR-449 are in the group with intermediate expression [44].”
From Shared and distinct microRNA profiles between HT22, N2A and SH-SY5Y cell lines and primary mouse hippocampal neurons (2025) · cites this paper as [44]
Cites
Controlling the False Discovery Rate: A Practical and Powerful Approach to Multiple Testing
Benjamini, Y., & Hochberg, Y. (1995) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.2517-6161.1995.tb02031.x · Cited by 109,482 · see citing works
Self-citationAn improved Bonferroni procedure for multiple tests of significance
SIMES, R. J. (1986) · Biometrika · doi:10.1093/biomet/73.3.751 · Cited by 2,396 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
The control of the false discovery rate in multiple testing under dependency
Benjamini, Y., & Yekutieli, D. (2001) · The Annals of Statistics · doi:10.1214/aos/1013699998 · Cited by 10,490 · see citing works
Self-citationStatistical significance for genomewide studies
Storey, J. D., & Tibshirani, R. (2003) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1530509100 · Cited by 9,692 · see citing works
A Direct Approach to False Discovery Rates
Storey, J. D. (2002) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/1467-9868.00346 · Cited by 5,603 · see citing works
Controlling the false discovery rate in behavior genetics research
Benjamini, Y., Drai, D., Elmer, G., Kafkafi, N., & Golani, I. (2001) · Behavioural Brain Research · doi:10.1016/s0166-4328(01)00297-2 · Cited by 4,146 · see citing works
Self-citationThe positive false discovery rate: a Bayesian interpretation and the q-value
Storey, J. D. (2003) · The Annals of Statistics · doi:10.1214/aos/1074290335 · Cited by 2,204 · see citing works
Bretz, F., Hothorn, T., & Westfall, P. (2016) · doi:10.1201/9781420010909 · Cited by 1,233 · see citing works
Mao, S., Zhang, M., Liu, J., & Zhu, W. (2015) · Scientific Reports · doi:10.1038/srep16116 · Cited by 532 · see citing works
Laffel, L. M., Kanapka, L. G., Beck, R. W., Bergamo, K., Clements, M. A., Criego, A., DeSalvo, D. J., Goland, R., Hood, K., Liljenquist, D., et al. (2020) · JAMA · doi:10.1001/jama.2020.6940 · Cited by 443 · see citing works
Frazier, T. W., Georgiades, S., Bishop, S. L., & Hardan, A. Y. (2014) · Journal of the American Academy of Child & Adolescent Psychiatry · doi:10.1016/j.jaac.2013.12.004 · Cited by 409 · see citing works
Genetics of brain fiber architecture and intellectual performance
Chiang, M.-C., Barysheva, M., Shattuck, D. W., Lee, A. D., Madsen, S. K., Avedissian, C., Klunder, A. D., Toga, A. W., McMahon, K. L., de Zubicaray, G. I., et al. (2009) · The Journal of Neuroscience · doi:10.1523/jneurosci.4184-08.2009 · Cited by 406 · see citing works
Lainhart, J. E., Bigler, E. D., Bocian, M., Coon, H., Dinh, E., Dawson, G., Deutsch, C. K., Dunn, M., Estes, A., Tager‐Flusberg, H., et al. (2006) · American Journal of Medical Genetics Part A · doi:10.1002/ajmg.a.31465 · Cited by 381 · see citing works
Pratley, R. E., Kanapka, L. G., Rickels, M. R., Ahmann, A., Aleppo, G., Beck, R., Bhargava, A., Bode, B. W., Carlson, A., Chaytor, N. S., et al. (2020) · JAMA · doi:10.1001/jama.2020.6928 · Cited by 375 · see citing works
Bergenstal, R. M., Nimri, R., Beck, R. W., Criego, A., Laffel, L., Schatz, D., Battelino, T., Danne, T., Weinzimer, S. A., Sibayan, J., et al. (2021) · The Lancet · doi:10.1016/s0140-6736(20)32514-9 · Cited by 344 · see citing works
Hosseini, S., Wilk, E., Michaelsen-Preusse, K., Gerhauser, I., Baumgärtner, W., Geffers, R., Schughart, K., & Korte, M. (2018) · The Journal of Neuroscience · doi:10.1523/jneurosci.1740-17.2018 · Cited by 236 · see citing works
Benjamini, Y., & Liu, W. (1999) · Journal of Statistical Planning and Inference · doi:10.1016/s0378-3758(99)00040-3 · Cited by 242 · see citing works
Self-citationClinical and neural effects of six-week administration of oxytocin on core symptoms of autism
Watanabe, T., Kuroda, M., Kuwabara, H., Aoki, Y., Iwashiro, N., Tatsunobu, N., Takao, H., Nippashi, Y., Kawakubo, Y., Kunimatsu, A., et al. (2015) · Brain · doi:10.1093/brain/awv249 · Cited by 222 · see citing works
Compton, S. N., Peris, T. S., Almirall, D., Birmaher, B., Sherrill, J., Kendall, P. C., March, J. S., Gosch, E. A., Ginsburg, G. S., Rynn, M. A., et al. (2014) · Journal of Consulting and Clinical Psychology · doi:10.1037/a0035458 · Cited by 226 · see citing works
Anisimova, M., & Yang, Z. (2007) · Molecular Biology and Evolution · doi:10.1093/molbev/msm042 · Cited by 213 · see citing works
Hardan, A. Y., Libove, R. A., Keshavan, M. S., Melhem, N. M., & Minshew, N. J. (2009) · Biological Psychiatry · doi:10.1016/j.biopsych.2009.04.024 · Cited by 173 · see citing works
Large-scale brain network dynamics supporting adolescent cognitive control
Dwyer, D. B., Harrison, B. J., Yücel, M., Whittle, S., Zalesky, A., Pantelis, C., Allen, N. B., & Fornito, A. (2014) · The Journal of Neuroscience · doi:10.1523/jneurosci.1634-14.2014 · Cited by 143 · see citing works
A Clinical Trial of In-Home CBT for Depressed Mothers in Home Visitation
Ammerman, R. T., Putnam, F. W., Altaye, M., Stevens, J., Teeters, A. R., & Van Ginkel, J. B. (2013) · Behavior Therapy · doi:10.1016/j.beth.2013.01.002 · Cited by 123 · see citing works
Sanchez‐Roige, S., Fontanillas, P., Elson, S. L., The 23andMe Research Team, Gray, J. C., de Wit, H., Davis, L. K., MacKillop, J., & Palmer, A. A. (2019) · Addiction Biology · doi:10.1111/adb.12574 · Cited by 101 · see citing works
Nozawa, T., Sakaki, K., Ikeda, S., Jeong, H., Yamazaki, S., Kawata, K. H. D. S., Kawata, N. Y. D. S., Sasaki, Y., Kulason, K., Hirano, K., et al. (2019) · Scientific Reports · doi:10.1038/s41598-019-49257-z · Cited by 95 · see citing works
Wolchik, S. A., Sandler, I. N., Tein, J.-Y., Mahrer, N. E., Millsap, R. E., Winslow, E., Vélez, C., Porter, M. M., Luecken, L. J., & Reed, A. (2013) · Journal of Consulting and Clinical Psychology · doi:10.1037/a0033235 · Cited by 91 · see citing works
Panigrahy, A., Schmithorst, V. J., Wisnowski, J. L., Watson, C. G., Bellinger, D. C., Newburger, J. W., & Rivkin, M. J. (2015) · NeuroImage: Clinical · doi:10.1016/j.nicl.2015.01.013 · Cited by 86 · see citing works
Caracausi, M., Ghini, V., Locatelli, C., Mericio, M., Piovesan, A., Antonaros, F., Pelleri, M. C., Vitale, L., Vacca, R. A., Bedetti, F., et al. (2018) · Scientific Reports · doi:10.1038/s41598-018-20834-y · Cited by 84 · see citing works
Scholtens, D. M., Bain, J. R., Reisetter, A. C., Muehlbauer, M. J., Nodzenski, M., Stevens, R. D., Ilkayeva, O., Lowe, L. P., Metzger, B. E., Newgard, C. B., et al. (2016) · Diabetes · doi:10.2337/db15-1748 · Cited by 69 · see citing works
Walker, D. M., Goetz, B. M., & Gore, A. C. (2014) · Molecular Endocrinology · doi:10.1210/me.2013-1270 · Cited by 73 · see citing works
Miller, K. M., Kanapka, L. G., Rickels, M. R., Ahmann, A. J., Aleppo, G., Ang, L., Bhargava, A., Bode, B. W., Carlson, A., Chaytor, N. S., et al. (2022) · Diabetes Technology & Therapeutics · doi:10.1089/dia.2021.0503 · Cited by 73 · see citing works
Salanova, M., Gelfi, C., Moriggi, M., Vasso, M., Viganò, A., Minafra, L., Bonifacio, G., Schiffl, G., Gutsmann, M., Felsenberg, D., et al. (2014) · The FASEB Journal · doi:10.1096/fj.14-252825 · Cited by 68 · see citing works
Cox, S. R., Harris, M. A., Ritchie, S. J., Buchanan, C. R., Valdés Hernández, M. C., Corley, J., Taylor, A. M., Madole, J. W., Harris, S. E., Whalley, H. C., et al. (2021) · Molecular Psychiatry · doi:10.1038/s41380-020-00975-1 · Cited by 60 · see citing works
Strunz, T., Lauwen, S., Kiel, C., International AMD Genomics Consortium (IAMDGC), Fritsche, L. G., Igl, W., Bailey, J. N. C., Grassmann, F., Sengupta, S., Bragg-Gresham, J. L., et al. (2020) · Scientific Reports · doi:10.1038/s41598-020-58510-9 · Cited by 60 · see citing works
Prenatal cocaine effects on brain structure in early infancy
Grewen, K., Burchinal, M., Vachet, C., Gouttard, S., Gilmore, J. H., Lin, W., Johns, J., Elam, M., & Gerig, G. (2014) · NeuroImage · doi:10.1016/j.neuroimage.2014.06.070 · Cited by 65 · see citing works
Vercammen, A., Knegtering, H., Bruggeman, R., & Aleman, A. (2011) · Schizophrenia Bulletin · doi:10.1093/schbul/sbq007 · Cited by 63 · see citing works
Functional brain connectivity in ex utero premature infants compared to in utero fetuses
De Asis-Cruz, J., Kapse, K., Basu, S. K., Said, M., Scheinost, D., Murnick, J., Chang, T., du Plessis, A., & Limperopoulos, C. (2020) · NeuroImage · doi:10.1016/j.neuroimage.2020.117043 · Cited by 63 · see citing works
Karcher, N. R., Michelini, G., Kotov, R., & Barch, D. M. (2021) · Biological Psychiatry: Cognitive Neuroscience and Neuroimaging · doi:10.1016/j.bpsc.2020.09.008 · Cited by 55 · see citing works
Bionic Pancreas Research Group, Messer, L. H., Buckingham, B. A., Cogen, F., Daniels, M., Forlenza, G., Jafri, R. Z., Mauras, N., Muir, A., Wadwa, R. P., et al. (2022) · Diabetes Technology & Therapeutics · doi:10.1089/dia.2022.0201.pub · Cited by 59 · see citing works
Gupta, A., Mayer, E. A., Acosta, J. R., Hamadani, K., Torgerson, C., van Horn, J. D., Chang, L., Naliboff, B., Tillisch, K., & Labus, J. S. (2017) · Neurobiology of Stress · doi:10.1016/j.ynstr.2017.02.003 · Cited by 56 · see citing works
Chien, H., Lin, H., Lai, M., Gau, S. S., & Tseng, W. I. (2015) · Autism Research · doi:10.1002/aur.1457 · Cited by 54 · see citing works
Morphological brain measures of cortico‐limbic inhibition related to resilience
Gupta, A., Love, A., Kilpatrick, L. A., Labus, J. S., Bhatt, R., Chang, L., Tillisch, K., Naliboff, B., & Mayer, E. A. (2017) · Journal of Neuroscience Research · doi:10.1002/jnr.24007 · Cited by 54 · see citing works
Kwong, K. S., Holland, B., & Cheung, S. H. (2002) · Journal of Statistical Planning and Inference · doi:10.1016/s0378-3758(01)00252-x · Cited by 54 · see citing works
Bionic Pancreas Research Group, Beck, R. W., Russell, S. J., Damiano, E. R., El-Khatib, F. H., Ruedy, K. J., Balliro, C., Li, Z., & Calhoun, P. (2022) · Diabetes Technology & Therapeutics · doi:10.1089/dia.2022.0167 · Cited by 40 · see citing works
Bionic Pancreas Research Group, Kruger, D., Kass, A., Lonier, J., Pettus, J., Raskin, P., Salam, M., Trikudanathan, S., Zhou, K., Russell, S. J., et al. (2022) · Diabetes Technology & Therapeutics · doi:10.1089/dia.2022.0200 · Cited by 38 · see citing works
Levy, C. J., Raghinaru, D., Kudva, Y. C., Pandit, K., Blevins, T., Casaubon, L., Desjardins, D., Levister, C. M., O’Malley, G., Reid, C., et al. (2024) · Clinical Diabetes · doi:10.2337/cd23-0025 · Cited by 37 · see citing works
Intermittent Fasting Implementation and Association with Eating Disorder Symptomatology
Cuccolo, K., Kramer, R., Petros, T., & Thoennes, M. (2022) · Eating Disorders · doi:10.1080/10640266.2021.1922145 · Cited by 33 · see citing works
Mindfulness, Obsessive–Compulsive Symptoms, and Executive Dysfunction
Crowe, K., & McKay, D. (2016) · Cognitive Therapy and Research · doi:10.1007/s10608-016-9777-x · Cited by 32 · see citing works
Amyloid-beta induced retrograde axonal degeneration in a mouse tauopathy model
Nishioka, C., Liang, H.-F., Barsamian, B., & Sun, S.-W. (2019) · NeuroImage · doi:10.1016/j.neuroimage.2019.01.007 · Cited by 30 · see citing works
Implicit Cognitions as a Behavioral Marker of Suicide Attempts in Adolescents
Millner, A. J., Augenstein, T. M., Visser, K. H., Gallagher, K., Vergara, G. A., D’Angelo, E. J., & Nock, M. K. (2019) · Archives of Suicide Research · doi:10.1080/13811118.2017.1421488 · Cited by 29 · see citing works
Foster, M. E., Anthony, J. L., Zucker, T. A., & Branum-Martin, L. (2019) · Early Childhood Research Quarterly · doi:10.1016/j.ecresq.2018.02.007 · Cited by 26 · see citing works
Thissen, A. J. A. M., Luman, M., Hartman, C., Hoekstra, P., van Lieshout, M., Franke, B., Oosterlaan, J., Rommelse, N. N. J., & Buitelaar, J. K. (2014) · Journal of the American Academy of Child & Adolescent Psychiatry · doi:10.1016/j.jaac.2014.05.015 · Cited by 22 · see citing works
Van Name, M. A., Kanapka, L. G., DiMeglio, L. A., Miller, K. M., Albanese-O’Neill, A., Commissariat, P., Corathers, S. D., Harrington, K. R., Hilliard, M. E., Anderson, B. J., et al. (2023) · Journal of Diabetes Science and Technology · doi:10.1177/19322968221084667 · Cited by 20 · see citing works
Mikkilineni, L., Natrakul, D. A., Lam, N., Manasanch, E. E., Mann, J., Weissler, K. A., Wong, N., Brudno, J. N., Goff, S. L., Yang, J. C., et al. (2024) · Molecular Therapy · doi:10.1016/j.ymthe.2023.12.018 · Cited by 16 · see citing works
Chebyshev’s Inequality for Nonparametric Testing with Small<i>N</i>and<i>α</i>in Microarray Research
Beasley, T. M., Page, G. P., Brand, J. P. L., Gadbury, G. L., Mountz, J. D., & Allison, D. B. (2004) · Journal of the Royal Statistical Society Series C: Applied Statistics · doi:10.1111/j.1467-9876.2004.00428.x · Cited by 15 · see citing works
Integrative Data Analysis and the Study of Global Health
Hussong, A. M., Cole, V. T., Curran, P. J., Bauer, D. J., & Gottfredson, N. C. (2020) · ICSA Book Series in Statistics · doi:10.1007/978-3-030-35260-8_5 · Cited by 13 · see citing works
Child anxiety prevention study: impact on functional outcomes
Pella, J. E., Drake, K. L., Tein, J.-Y., & Ginsburg, G. S. (2017) · Child Psychiatry & Human Development · doi:10.1007/s10578-016-0667-y · Cited by 12 · see citing works
Crowl, S., Coleman, M. B., Chaphiv, A., Jordan, B. T., & Naegle, K. M. (2025) · Cell Systems · doi:10.1016/j.cels.2025.101318 · Cited by 10 · see citing works
Allergic tendencies are associated with larger gray matter volumes
Takeuchi, H., Taki, Y., Nouchi, R., Yokoyama, R., Kotozaki, Y., Nakagawa, S., Sekiguchi, A., Iizuka, K., Yamamoto, Y., Hanawa, S., et al. (2018) · Scientific Reports · doi:10.1038/s41598-018-21985-8 · Cited by 8 · see citing works
Koning, S. M., Kessler, C. L., Canli, T., Duman, E. A., Adam, E. K., Zinbarg, R., Craske, M. G., Stephens, J. E., & Vrshek-Schallhorn, S. (2024) · Psychoneuroendocrinology · doi:10.1016/j.psyneuen.2024.107181 · Cited by 8 · see citing works
Anitua, E., Murias-Freijo, A., Tierno, R., Tejero, R., & Alkhraisat, M. H. (2024) · BMC Oral Health · doi:10.1186/s12903-024-04675-y · Cited by 7 · see citing works
Boyd, M. C., Burdette, J. H., Miller, M. E., Lyday, R. G., Hugenschmidt, C. E., Jack Rejeski, W., Simpson, S. L., Baker, L. D., Tomlinson, C. E., Kritchevsky, S. B., et al. (2024) · GeroScience · doi:10.1007/s11357-024-01251-8 · Cited by 3 · see citing works
Cell signaling pathways discovery from multi-modal data
He, C., Simpson, C., Cossentino, I., Zhang, B., Tkachev, S., Eddins, D. J., Kosters, A., Yang, J., Sheth, S., Levy, T., et al. (2025) · doi:10.1101/2025.02.06.636961 · Cited by 3 · see citing works
Murphy, R., Villegas-Salmerón, J., Sanz-Rodriguez, A., Engel, T., Mooney, C., Henshall, D. C., & Jimenez-Mateos, E. M. (2025) · PLOS One · doi:10.1371/journal.pone.0326401 · Cited by 3 · see citing works
Gruner, N. G., Cullen, J. M., Crosby, J. M., DeCross, S. N., Mathes, B., Garner, L., Gironda, C., Hu, Y., Krompinger, J. W., Elias, J. A., et al. (2024) · Journal of Cognitive Psychotherapy · doi:10.1891/jcp-2022-0033
- Authors
- Benjamini, Y., & Hochberg, Y.
- Affiliations
- Tel Aviv University
- Venue
- Journal of Educational and Behavioral Statistics
- Published
- 2000-03
- Volume
- 25
- Issue
- 1
- Pages
- 60-83
- ISSN
- 1076-9986
- 1935-1054
- Cited by
- 1,349
- References
- 2
- Language
- en
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
