When to adjust alpha during multiple testing: a consideration of disjunction, conjunction, and individual testing
Open accessPublished version of a preprint
Abstract
Scientists often adjust their significance threshold (alpha level) during null hypothesis significance testing in order to take into account multiple testing and multiple comparisons. This alpha adjustment has become particularly relevant in the context of the replication crisis in science. The present article considers the conditions in which this alpha adjustment is appropriate and the conditions in which it is inappropriate. A distinction is drawn between three types of multiple testing: disjunction testing, conjunction testing, and individual testing. It is argued that alpha adjustment is only appropriate in the case of disjunction testing, in which at least one test result must be significant in order to reject the associated joint null hypothesis. Alpha adjustment is inappropriate in the case of conjunction testing, in which all relevant results must be significant in order to reject the joint null hypothesis. Alpha adjustment is also inappropriate in the case of individual testing, in which each individual result must be significant in order to reject each associated individual null hypothesis. The conditions under which each of these three types of multiple testing is warranted are examined. It is concluded that researchers should not automatically (mindlessly) assume that alpha adjustment is necessary during multiple testing. Illustrations are provided in relation to joint studywise hypotheses and joint multiway ANOVAwise hypotheses.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Alpha levels of 0.05 without correction for multiple comparisons were applied because the analyses are exploratory and each outcome is best considered as individual tests rather than disjunction tests.
“Alpha levels of 0.05 alpha without correction for multiple comparisons were applied, given that the analyses are exploratory and each outcome is best considered as individual tests rather than disjunction tests.75”
From Acceptance and commitment therapy for mild traumatic brain injury (ACTion-mTBI): a quasiexperimental feasibility study (2025) · cites this paper as [75]
The Abstract and Conclusions support not correcting alpha for multiple comparisons when outcomes are tested individually rather than as a disjunction test. But the Conclusions explicitly reject exploratory status as the reason for deciding whether to adjust alpha. The section ‘Distinguishing Between Alpha Specification and Alpha Adjustment’ also cautions that individual testing does not automatically justify an alpha level of .05. (LLM, conf=0.90)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Controlling the False Discovery Rate: A Practical and Powerful Approach to Multiple Testing
Benjamini, Y., & Hochberg, Y. (1995) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.2517-6161.1995.tb02031.x · Cited by 109,482 · see citing works
Taming Black Swans: Challenges for Sustainable Strategic Crisis Management: Plenary Session
Pelletier, P. (2021) · HAL · doi:10.1111/j.1468 · Cited by 6,488 · see citing works
No adjustments are needed for multiple comparisons.
Rothman, K. J. (1990) · Epidemiology · doi:10.1097/00001648-199001000-00010 · Cited by 5,965 · see citing works
What's wrong with Bonferroni adjustments
Perneger, T. V. (1998) · BMJ · doi:10.1136/bmj.316.7139.1236 · Cited by 5,900 · see citing works
Casella, G., & Berger, R. L. (1993) · Chapman & Hall/CRC eBooks · doi:10.1201/9781003456285 · Cited by 5,835 · see citing works
Simultaneous Statistical Inference
Miller, R. G. (1981) · Springer series in statistics · doi:10.1007/978-1-4613-8122-8 · Cited by 5,608 · see citing works
Fisher, R. A. (1936) · BMJ · doi:10.1136/bmj.1.3923.554-a · Cited by 4,980 · see citing works
Adjusting for multiple testing—when and how?
Bender, R., & Lange, S. (2001) · Journal of Clinical Epidemiology · doi:10.1016/s0895-4356(00)00314-0 · Cited by 3,060 · see citing works
When to use the <scp>B</scp> onferroni correction
Armstrong, R. A. (2014) · Ophthalmic and Physiological Optics · doi:10.1111/opo.12131 · Cited by 3,031 · see citing works
Rectangular Confidence Regions for the Means of Multivariate Normal Distributions
Šidák, Z. (1967) · Journal of the American Statistical Association · doi:10.1080/01621459.1967.10482935 · Cited by 2,904 · see citing works
Redefine statistical significance
Benjamin, D. J., Berger, J. O., Johannesson, M., Nosek, B. A., Wagenmakers, E.-J., Berk, R., Bollen, K. A., Brembs, B., Brown, L., Camerer, C., et al. (2017) · Nature Human Behaviour · doi:10.1038/s41562-017-0189-z · Cited by 2,772 · see citing works
Designing Experiments and Analyzing Data: A Model Comparison Perspective
Maxwell, S., & Delaney, H. D. (1990) · Journal of the American Statistical Association · doi:10.2307/2289744 · Cited by 2,593 · see citing works
Multiple Comparison Procedures
Hochberg, Y., & Tamhane, A. C. (1987) · Wiley Series in Probability and Statistics · doi:10.1002/9780470316672 · Cited by 2,389 · see citing works
Resampling-Based Multiple Testing: Examples and Methods for p-Value Adjustment
Martín, M. A., Westfall, P. H., & Young, S. (1993) · Biometrics · doi:10.2307/2533464 · Cited by 2,308 · see citing works
Multiple Comparison Procedures
Hahs-Vaughn, D., & Lomax, R. (2013) · Routledge eBooks · doi:10.4324/9780203137819-17 · Cited by 2,132 · see citing works
The preregistration revolution
Nosek, B. A., Ebersole, C. R., DeHaven, A. C., & Mellor, D. T. (2018) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1708274114 · Cited by 1,998 · see citing works
Shaffer, J. P. (1995) · Annual Review of Psychology · doi:10.1146/annurev.ps.46.020195.003021 · Cited by 1,960 · see citing works
Meehl, P. E. (1978) · Journal of Consulting and Clinical Psychology · doi:10.1037/0022-006x.46.4.806 · Cited by 1,952 · see citing works
Valid conjunction inference with the minimum statistic
Nichols, T. E., Brett, M., Andersson, J., Wager, T. D., & Poline, J. (2005) · NeuroImage · doi:10.1016/j.neuroimage.2004.12.005 · Cited by 1,885 · see citing works
Things I have learned (so far).
Cohen, J. (1990) · American Psychologist · doi:10.1037/0003-066x.45.12.1304 · Cited by 1,781 · see citing works
Modern Epidemiology 3rd edition
Rothman, K. J., Lash, T. L., & Greenland, S. (1986) · Cited by 1,736 · see citing works
Glickman, M. E., Rao, S. R., & Schultz, M. R. (2014) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2014.03.012 · Cited by 1,561 · see citing works
(1953) · Medical Entomology and Zoology · Cited by 1,321 · see citing works
Do multiple outcome measures require p-value adjustment?
Feise, R. J. (2002) · BMC Medical Research Methodology · doi:10.1186/1471-2288-2-8 · Cited by 1,299 · see citing works
What does research reproducibility mean?
Goodman, S. N., Fanelli, D., & Ioannidis, J. P. A. (2016) · Science Translational Medicine · doi:10.1126/scitranslmed.aaf5027 · Cited by 1,293 · see citing works
Bretz, F., Hothorn, T., & Westfall, P. (2016) · doi:10.1201/9781420010909 · Cited by 1,233 · cited 1× · see citing works
“Berger (1982) proposed this approach as an intersection–union test (Berger, 1982; Berger & Hsu, 1996; Bretz et al., 2011, p. 22).”
BackgroundDesigning Experiments and Analysing Data: A Model Comparison Perspective.
Freeman, G. H., Maxwell, S. E., & Delaney, H. D. (1991) · Biometrics · doi:10.2307/2532173 · Cited by 1,229 · see citing works
How to get statistically significant effects in any ERP experiment (and why you shouldn't)
Luck, S. J., & Gaspelin, N. (2016) · Psychophysiology · doi:10.1111/psyp.12639 · Cited by 1,202 · see citing works
Multiple Comparisons: Theory and Methods
Christensen, R., & Hsu, J. C. (1997) · Technometrics · doi:10.2307/1270920 · Cited by 1,105 · see citing works
Resampling-Based Multiple Testing: Examples and Methods for p-Value Adjustment
Chaubey, Y. P., Westfall, P. H., & Young, S. S. (1993) · Technometrics · doi:10.2307/1270279 · Cited by 943 · see citing works
Bedford, T., & Cooke, R. (2001) · Cambridge University Press eBooks · doi:10.1017/cbo9780511813597.005 · Cited by 724 · see citing works
Registered Reports: A Method to Increase the Credibility of Published Results
Nosek, B. A., & Lakens, D. (2014) · Social Psychology · doi:10.1027/1864-9335/a000192 · Cited by 716 · see citing works
On a Heuristic Method of Test Construction and its use in Multivariate Analysis
Roy, S. (1953) · The Annals of Mathematical Statistics · doi:10.1214/aoms/1177729029 · Cited by 631 · see citing works
Bioequivalence trials, intersection-union tests and equivalence confidence sets
Berger, R. L., & Hsu, J. C. (1996) · Statistical Science · doi:10.1214/ss/1032280304 · Cited by 621 · see citing works
Hsu, J. C. (1996) · Springer eBooks · doi:10.1007/978-1-4899-7180-7 · Cited by 590 · see citing works
The principled control of false positives in neuroimaging
Bennett, C. M., Wolford, G. L., & Miller, M. B. (2009) · Social Cognitive and Affective Neuroscience · doi:10.1093/scan/nsp053 · Cited by 486 · see citing works
Multiple Comparisons and Related Issues in the Interpretation of Epidemiologic Data
Savitz, D. A., & Olshan, A. F. (1995) · American Journal of Epidemiology · doi:10.1093/oxfordjournals.aje.a117737 · Cited by 463 · see citing works
Multiple Testing Procedures with Applications to Genomics
Dudoit, S., & van der Laan, M. J. (2007) · Springer series in statistics · doi:10.1007/978-0-387-49317-6 · Cited by 453 · see citing works
Statistical Issues in Drug Development
Senn, S. (2007) · doi:10.1002/9780470723586 · Cited by 448 · see citing works
Detecting and avoiding likely false‐positive findings – a practical guide
Forstmeier, W., Wagenmakers, E., & Parker, T. H. (2017) · Biological Reviews · doi:10.1111/brv.12315 · Cited by 436 · see citing works
Hidden multiplicity in exploratory multiway ANOVA: Prevalence and remedies
Cramer, A. O. J., van Ravenzwaaij, D., Matzke, D., Steingroever, H., Wetzels, R., Grasman, R. P. P. P., Waldorp, L., & Wagenmakers, E. (2015) · Psychonomic Bulletin & Review · doi:10.3758/s13423-015-0913-5 · Cited by 416 · see citing works
Simultaneous Statistical Inference
Bondar, J. V., Putter, J., & Miller, R. G. (1968) · Technometrics · doi:10.2307/1267066 · Cited by 403 · see citing works
Bennett, C., Miller, M., & Wolford, G. (2009) · NeuroImage · doi:10.1016/s1053-8119(09)71202-9 · Cited by 400 · see citing works
Statistical Issues in Drug Development
Senn, S. J. (2021) · Wiley eBooks · doi:10.1002/9781119238614 · Cited by 384 · see citing works
Designing Experiments and Analyzing Data
Maxwell, S. E., & Delaney, H. D. (2003) · Routledge eBooks · doi:10.4324/9781410609243 · Cited by 379 · see citing works
Multiple hypothesis testing in genomics
Goeman, J., & Solari, A. (2014) · Statistics in Medicine · doi:10.1002/sim.6082 · Cited by 377 · see citing works
Multiplicity in randomised trials I: endpoints and treatments
Schulz, K. F., & Grimes, D. A. (2005) · The Lancet · doi:10.1016/s0140-6736(05)66461-6 · Cited by 366 · see citing works
Streiner, D. L. (2015) · American Journal of Clinical Nutrition · doi:10.3945/ajcn.115.113548 · Cited by 344 · see citing works
Preregistration Is Hard, And Worthwhile
Nosek, B. A., Beck, E. D., Campbell, L., Flake, J. K., Hardwicke, T. E., Mellor, D. T., Veer, A. V. '., & Vazire, S. (2019) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2019.07.009 · Cited by 325 · see citing works
Multiparameter Hypothesis Testing and Acceptance Sampling
Berger, R. L. (1982) · Technometrics · doi:10.1080/00401706.1982.10487790 · Cited by 289 · see citing works
Panagiotou, O. A., Ioannidis, J. P. A., & for the Genome-Wide Significance Project (2012) · International Journal of Epidemiology · doi:10.1093/ije/dyr178 · Cited by 286 · see citing works
Multiparameter Hypothesis Testing and Acceptance Sampling
Berger, R. L. (1982) · Technometrics · doi:10.2307/1267823 · Cited by 285 · see citing works
Resampling based multiple testing examples and methods for p-value adjustment
Westfall, P. H. 1., & Young, S. S. 1. (1993) · Cited by 276 · see citing works
Multiplicity Considerations in the Design and Analysis of Clinical Trials
Cook, R. J., & Farewell, V. T. (1996) · Journal of the Royal Statistical Society Series A (Statistics in Society) · doi:10.2307/2983471 · Cited by 257 · see citing works
Non‐parametric combination and related permutation tests for neuroimaging
Winkler, A. M., Webster, M., Brooks, J. C. W., Tracey, I., Smith, S. M., & Nichols, T. E. (2016) · Human Brain Mapping · doi:10.1002/hbm.23115 · Cited by 237 · see citing works
Practical Guidelines for Multiplicity Adjustment in Clinical Trials
Proschan, M. A., & Waclawiw, M. A. (2000) · Controlled Clinical Trials · doi:10.1016/s0197-2456(00)00106-9 · Cited by 226 · see citing works
Rubin, M. (2017) · Review of General Psychology · doi:10.1037/gpr0000123 · Cited by 205 · see citing works
Self-citationKempson, R. E., & Mead, R. (1990) · Journal of the Royal Statistical Society Series D (The Statistician) · doi:10.2307/2348207 · Cited by 193 · see citing works
Groot, A. D. D. (2014) · Acta Psychologica · doi:10.1016/j.actpsy.2014.02.001 · Cited by 190 · see citing works
Setting an Optimal α That Minimizes Errors in Null Hypothesis Significance Tests
Mudge, J. F., Baker, L. F., Edge, C. B., & Houlahan, J. E. (2012) · PLoS ONE · doi:10.1371/journal.pone.0032734 · Cited by 183 · see citing works
Rubin, M. (2017) · Review of General Psychology · doi:10.1037/gpr0000128 · Cited by 181 · see citing works
Self-citationCorrecting for multiple-testing in multi-arm trials: is it necessary and is it done?
Wason, J. M. S., Stecher, L., & Mander, A. (2014) · Trials · doi:10.1186/1745-6215-15-364 · Cited by 180 · see citing works
Colloquy: Should Familywise Alpha Be Adjusted?.
O’Keefe, D. J. (2003) · Human Communication Research · doi:10.1111/j.1468-2958.2003.tb00846.x · Cited by 155 · see citing works
Traditional multiplicity adjustment methods in clinical trials
Dmitrienko, A. A., & D’Agostino, R. B. (2013) · Statistics in Medicine · doi:10.1002/sim.5990 · Cited by 153 · see citing works
Rules and Judgments in Statistics: Three Examples
Stewart‐Oaten, A. (1995) · Ecology · doi:10.2307/1940736 · Cited by 137 · see citing works
Simultaneous Inference: When Should Hypothesis Testing Problems Be Combined?
Efron, B. (2008) · Cited by 105 · see citing works
Tsai, J., Kasprow, W. J., & Rosenheck, R. A. (2013) · Addictive Behaviors · doi:10.1016/j.addbeh.2013.02.002 · Cited by 102 · see citing works
The Problem of Multiple Testing
Sainani, K. L. (2009) · PM&R · doi:10.1016/j.pmrj.2009.10.004 · Cited by 95 · see citing works
Simultaneous inference: When should hypothesis testing problems be combined?
Efron, B. (2008) · The Annals of Applied Statistics · doi:10.1214/07-aoas141 · Cited by 84 · see citing works
Greenland, S. (2020) · Paediatric and Perinatal Epidemiology · doi:10.1111/ppe.12711 · Cited by 83 · see citing works
Parker, R. A., & Weir, C. J. (2020) · Clinical Trials · doi:10.1177/1740774520941419 · Cited by 82 · see citing works
When to Combine Hypotheses and Adjust for Multiple Tests
Veazie, P. J. (2006) · Health Services Research · doi:10.1111/j.1475-6773.2006.00512.x · Cited by 79 · see citing works
How to deal with multiple endpoints in clinical trials
Neuhäuser, M. (2006) · Fundamental and Clinical Pharmacology · doi:10.1111/j.1472-8206.2006.00437.x · Cited by 77 · see citing works
Jannot, A., Ehret, G., & Perneger, T. (2015) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2015.01.001 · Cited by 76 · see citing works
Errors in Statistical Inference Under Model Misspecification: Evidence, Hypothesis Testing, and AIC
Dennis, B., Ponciano, J. M., Taper, M. L., & Lele, S. R. (2019) · Frontiers in Ecology and Evolution · doi:10.3389/fevo.2019.00372 · Cited by 74 · see citing works
p Value fetishism and use of the Bonferroni adjustment
Morgan, J. F. (2007) · Evidence-Based Mental Health · doi:10.1136/ebmh.10.2.34 · Cited by 73 · see citing works
Matsunaga, M. (2007) · Communication Methods and Measures · doi:10.1080/19312450701641409 · Cited by 72 · see citing works
Joint Hypothesis Testing and Gatekeeping Procedures for Studies with Multiple Endpoints
Mascha, E. J., & Turan, A. (2012) · Anesthesia & Analgesia · doi:10.1213/ane.0b013e3182504435 · Cited by 72 · see citing works
Dataset decay and the problem of sequential analyses on open datasets
Thompson, W. H., Wright, J., Bissett, P. G., & Poldrack, R. A. (2020) · eLife · doi:10.7554/elife.53498 · Cited by 70 · see citing works
An Approach for Addressing the Multiple Testing Problem in Social Policy Impact Evaluations
Schochet, P. Z. (2009) · Evaluation Review · doi:10.1177/0193841x09350590 · Cited by 69 · see citing works
Multiple comparisons theory and methods
Hsu, J. C. (1996) · Cited by 66 · see citing works
LOPSIDED REASONING ON LOPSIDED TESTS AND MULTIPLE COMPARISONS
Hurlbert, S. H., & Lombardi, C. M. (2012) · Australian & New Zealand Journal of Statistics · doi:10.1111/j.1467-842x.2012.00652.x · Cited by 65 · see citing works
Inference to the best explanation: A neglected approach to theory appraisal in psychology
Haig, B. D. (2009) · The American Journal of Psychology · doi:10.2307/27784393 · Cited by 64 · see citing works
Does preregistration improve the credibility of research findings?
Rubin, M. (2020) · The Quantitative Methods for Psychology · doi:10.20982/tqmp.16.4.p376 · Cited by 61 · cited 5× · see citing works
“…materials and data set publicly available (e.g., via the Open Science Framework https:// osf. io/) in order for their audience to confirm that any joint hypothesis that they disjunction tested includes all of the relevant constituent hypotheses (Rubin, 2017b, pp. 272–273; Rubin, 2020, 2022).”
BackgroundInfluentialSelf-citationThe experiment as the unit for computing rates of error.
Ryan, T. A. (1962) · Psychological bulletin · doi:10.1037/h0040562 · Cited by 60 · see citing works
Wu, P., Yang, Q., Wang, K., Zhou, J., Ma, J., Tang, Q., Jin, L. G., Xiao, W., Jiang, A., Jiang, Y., et al. (2017) · Genomics · doi:10.1016/j.ygeno.2017.09.009 · Cited by 55 · see citing works
A note on the inconsistency inherent in the necessity to perform multiple comparisons.
Wilson, W. (1962) · Psychological Bulletin · doi:10.1037/h0040447 · Cited by 49 · see citing works
A Remark on Multiple Comparison Methods
Cox, D. R. (1965) · Technometrics · doi:10.1080/00401706.1965.10490250 · Cited by 49 · see citing works
Rubin, M. (2019) · The British Journal for the Philosophy of Science · doi:10.1093/bjps/axz050 · Cited by 47 · see citing works
Self-citationApproaches to multiplicity issues in complex research in microarray analysis
Yekutieli, D., Benaim, A. R., Benjamini, Y., et al. (2006) · Statistica Neerlandica · doi:10.1111/j.1467-9574.2006.00343.x · Cited by 44 · see citing works
On the Sensible Application of Familywise Alpha Adjustment
Tutzauer, F. (2003) · Human Communication Research · doi:10.1111/j.1468-2958.2003.tb00848.x · Cited by 39 · see citing works
Challenges to multiple testing in clinical trials
Hung, H., & Wang, S. (2010) · Biometrical Journal · doi:10.1002/bimj.200900206 · Cited by 34 · see citing works
Planned Hypothesis Tests Are Not Necessarily Exempt From Multiplicity Adjustment
Frane, A. V. (2015) · DOAJ (DOAJ: Directory of Open Access Journals) · Cited by 33 · see citing works
A Single Contrast Test Procedure
Huberty, C. J., & Morris, J. D. (1988) · Educational and Psychological Measurement · doi:10.1177/0013164488483001 · Cited by 33 · see citing works
On the logic of hypothesis testing in functional imaging
Turkheimer, F., Aston, J. A. D., & Cunningham, V. J. (2004) · European Journal of Nuclear Medicine and Molecular Imaging · doi:10.1007/s00259-003-1387-7 · Cited by 32 · see citing works
Testing the Hypothesis that Matters for Multiple Primary Endpoints
Capizzi, T. P., & Zhang, J. (1996) · Drug Information Journal · doi:10.1177/009286159603000410 · Cited by 32 · see citing works
Rubin, M. (2017) · Review of General Psychology · doi:10.1037/gpr0000135 · Cited by 32 · see citing works
Self-citationWeber, R. (2007) · Communication Methods and Measures · doi:10.1080/19312450701641391 · Cited by 31 · see citing works
Optimal alpha reduces error rates in gene expression studies: a meta-analysis approach
Mudge, J. F., Martyniuk, C. J., & Houlahan, J. E. (2017) · BMC Bioinformatics · doi:10.1186/s12859-017-1728-3 · Cited by 29 · see citing works
Method of balanced adjustment in testing co‐primary endpoints
Kordzakhia, G., Siddiqui, O. I., & Huque, M. F. (2010) · Statistics in Medicine · doi:10.1002/sim.3950 · Cited by 27 · see citing works
Dmitrienko, A. A., Bretz, F., Westfall, P. H., et al. (2009) · Chapman & Hall/CRC eBooks · doi:10.1201/9781584889854-7 · Cited by 27 · see citing works
Sample sizes for trials involving multiple correlated must‐win comparisons
Julious, S. A., & McIntyre, N. E. (2012) · Pharmaceutical Statistics · doi:10.1002/pst.515 · Cited by 26 · see citing works
An Analysis of the Use of Multiple Comparison Corrections in Ophthalmology Research
Stacey, A. W., Pouly, S., & Czyz, C. N. (2012) · Investigative Ophthalmology & Visual Science · doi:10.1167/iovs.11-8730 · Cited by 26 · see citing works
Kromrey, J. D., & Dickinson, W. B. (1995) · The Journal of Applied Behavioral Science · doi:10.1177/0021886395311006 · Cited by 25 · see citing works
Rubin, M. (2019) · Synthese · doi:10.1007/s11229-019-02433-0 · Cited by 23 · see citing works
Self-citationDmitrienko, A. A., Bretz, F., Westfall, P. H., Troendle, J. F., Wiens, B. L., Tamhane, A. C., & Hsu, J. (2009) · Chapman & Hall/CRC biostatistics series · doi:10.1201/9781584889854-c2 · Cited by 23 · see citing works
PROPERTIES OF MULTIPLE INTERSECTION-UNION TESTS FOR MULTIPLE ENDPOINTS IN COMBINATION THERAPY TRIALS
Westfall, P. H., Ho, S., & Prillaman, B. A. (2001) · Journal of Biopharmaceutical Statistics · doi:10.1081/bip-100107653 · Cited by 21 · see citing works
Francis, G., & Thunell, E. (2021) · Psychonomic Bulletin & Review · doi:10.3758/s13423-020-01855-z · Cited by 20 · see citing works
If not multiple comparisons, then what?
Kozak, M., & Powers, S. J. (2017) · Annals of Applied Biology · doi:10.1111/aab.12379 · Cited by 15 · see citing works
Re-assessment of multiple testing strategies for more efficient genome-wide association studies
Otani, T., Noma, H., Nishino, J., & Matsui, S. (2018) · European Journal of Human Genetics · doi:10.1038/s41431-018-0125-3 · Cited by 15 · see citing works
Methods as Tools: A Response to O'Keefe
Hewes, D. E. (2003) · Human Communication Research · doi:10.1093/hcr/29.3.448 · Cited by 12 · see citing works
Lew, M. J. (2019) · Handbook of experimental pharmacology · doi:10.1007/164_2019_286 · Cited by 12 · see citing works
Adjusting for Multiple Comparisons
Drachman, D. A. (2012) · Cited by 11 · see citing works
Ulrich, W. (2005) · DOAJ (DOAJ: Directory of Open Access Journals) · Cited by 10 · see citing works
Multiple Hypotheses Testing Procedures in Clinical Trials and Genomic Studies
Pan, Q. (2013) · Frontiers in Public Health · doi:10.3389/fpubh.2013.00063 · Cited by 10 · see citing works
Multiple Studies and Evidential Defeat
Kotzen, M. (2011) · Noûs · doi:10.1111/j.1468-0068.2010.00824.x · Cited by 9 · see citing works
Adjusting for selection bias in testing multiple families of hypothesesPreprint
Benjamini, Y., & Bogomolov, M. (2011) · arXiv (Cornell University) · doi:10.48550/arxiv.1106.3670 · Cited by 8 · see citing works
Hewes, D. E. (2003) · Human Communication Research · doi:10.1111/j.1468-2958.2003.tb00847.x · Cited by 7 · see citing works
Multiplicity in confirmatory clinical trials: a case study with discussion from a JSM panel
Wang, S., Bretz, F., Dmitrienko, A. A., Hsu, J. C., Hung, H., Koch, G. G., Maurer, W., Offen, W., & O’Neill, R. T. (2015) · Statistics in Medicine · doi:10.1002/sim.6561 · Cited by 6 · see citing works
Mei, S., Karimnezhad, A., Forest, M., Bickel, D. R., & Greenwood, C. M. T. (2017) · PLoS ONE · doi:10.1371/journal.pone.0185174 · Cited by 6 · see citing works
Type I Error Rates and Power of Multiple Hypothesis Testing Procedures in Factorial ANOVA
An, Q., Xu, D., & Brooks, G. P. (2013) · General Linear Model Journal · Cited by 5 · see citing works
Optimal Procedures for Multiple Testing ProblemsPreprint
Rosset, S., Heller, R., Painsky, A., et al. (2018) · arXiv (Cornell University) · Cited by 4 · see citing works
Does preregistration improve the credibility of research findings?
Rubin, M. (2020) · doi:10.23668/psycharchives.4839 · Cited by 3 · see citing works
Self-citationA reckless guide to P-values: local evidence, global errors
Lew, M. J. (2019) · arXiv · doi:10.48550/arxiv.1910.02042 · Cited by 2 · see citing works
Non-Factorial ANOVA: Test Only Substantive and Interpretable Hypotheses.
Rodriguez, M. (1997) · Cited by 1 · see citing works
Multiple Comparisons Texts: Their Utility in Guiding Research Practice
Klockars, A. J. (2003) · Journal of Clinical Child & Adolescent Psychology · doi:10.1207/s15374424jccp3204_15 · Cited by 1 · see citing works
The implications of significance testing based on hypothesiswise and studywise errorPreprint
Rubin, M. (2017) · OSF Preprints (OSF Preprints) · doi:10.17605/osf.io/7yfrv · Cited by 1 · see citing works
Self-citationMassaro, J. M. (2009) · Clinical and Translational Science · doi:10.1016/b978-0-12-373639-0.00003-0 · Cited by 1 · see citing works
Shaffer, J. P. (2005) · Encyclopedia of Statistical Sciences · doi:10.1002/0471667196.ess2452.pub2 · Cited by 1 · see citing works
Optimal and Maximin Procedures for Multiple Testing ProblemsPreprint
Rosset, S., Heller, R., Painsky, A., et al. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1804.10256
Flagged first, then by citations
Research works printed but not linked: 3
- Kim, K., Zakharkin, S. O., Loraine, A., & Allison, D. B. (2004). Picking the most likely candidates for further development: Novel intersection-union tests for addressing multi-component hypotheses in comparative genomics. Proceedings of the American Statistical Association, ASA Section on ENAR Spring Meeting (pp. 1396-1402). http://www.uab.edu/cngi/pdf/2004/JSM%202004%20-IUTs%20Kim%20et%20al.pdf
- Rothman, K. J., Greenland, S., & Lash, T. L. (2008). Modern epidemiology (3 rd ed.). Lippincott Williams & Wilkins.
- Sinclair, J., Taylor, P. J., & Hobbs, S. J. (2013). Alpha level adjustments for multiple dependent variable analyses and their applicability-A review. International Journal of Sports Science Engineering, 7, 17-20.
Cited by
Sorted by citations
Faulkner, J., Prouty, D., Devlin, L., Appleton, D., Roche, M., Below, K., Moffat, J., Snell, D., Williams, M. N., Barker-Collo, S., et al. (2025) · BMJ Open · doi:10.1136/bmjopen-2024-089727 · Cited by 7 · see citing works
- Authors
- Rubin, M.
- Affiliations
- University of Newcastle Australia (AU)
- Corresponding author
- Rubin, Mark
- ORCID
- Rubin, Mark
- Venue
- Synthese
- Published
- 2021-07-06
- Volume
- 199
- Issue
- 3-4
- Pages
- 10969-11000
- ISSN
- 0039-7857 (ISSN-L)
- 1573-0964
- OpenAlex
- W3178039131
- Cited by
- 371
- References
- 130
- Citations by year
- 2026: 69 · 2025: 91 · 2024: 91 · 2023: 68 · 2022: 43 · 2021: 8 · 2020: 1
- FWCI
- 39.15
- Type
- journal article
- Language
- en
- Open access
- green
- doi.org
- Locations
- Synthese (publishedVersion)
- PhilSci-Archive (University of Pittsburgh) (acceptedVersion)
- arXiv (Cornell University) (submittedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- Durham Research Online (Durham University) (submittedVersion)
- Figshare (submittedVersion)
- Durham Research Online (Durham University) (submittedVersion, cc-by)
- PhilPapers (PhilPapers Foundation) (submittedVersion)
- PsyArXiv (OSF Preprints) (submittedVersion, cc-by)
- MetArXiv (OSF Preprints) (submittedVersion, cc-by-nd)
- MetArXiv (OSF Preprints) (submittedVersion)
- PsyArXiv (OSF Preprints) (submittedVersion)
- NOVA (University of Newcastle Australia) (submittedVersion)
- arXiv (Cornell University)
- Psychology Archives (cc-by)
- Topics
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Clinical Trials
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods and Bayesian Inference
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Keywords
- multiple comparisons, individual testing, significance threshold, replication crisis, null hypothesis significance testing
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
