The DOI matches its Crossref record; 15 of the 15 references it cites were found in a registry or an index.
Do p Values Lose Their Meaning in Exploratory Analyses? It Depends How You Define the Familywise Error Rate
Open accessPublished version of a preprint
Abstract
Several researchers have recently argued that p values lose their meaning in exploratory analyses due to an unknown inflation of the alpha level (e.g., Nosek & Lakens, 2014 ; Wagenmakers, 2016 ). For this argument to be tenable, the familywise error rate must be defined in relation to the number of hypotheses that are tested in the same study or article. Under this conceptualization, the familywise error rate is usually unknowable in exploratory analyses because it is usually unclear how many hypotheses have been tested on a spontaneous basis and then omitted from the final research report. In the present article, I argue that it is inappropriate to conceptualize the familywise error rate in relation to the number of hypotheses that are tested. Instead, it is more appropriate to conceptualize familywise error in relation to the number of different tests that are conducted on the same null hypothesis in the same study. Under this conceptualization, alpha-level adjustments in exploratory analyses are (a) less necessary and (b) objectively verifiable. As a result, p values do not lose their meaning in exploratory analyses.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011) · Psychological Science · doi:10.1177/0956797611417632 · Cited by 6,567
No adjustments are needed for multiple comparisons.
Rothman, K. J. (1990) · Epidemiology · doi:10.1097/00001648-199001000-00010 · Cited by 5,965
What's wrong with Bonferroni adjustments
Perneger, T. V. (1998) · BMJ · doi:10.1136/bmj.316.7139.1236 · Cited by 5,900
A manifesto for reproducible science
Munafò, M. R., Nosek, B. A., Bishop, D. V. M., Button, K. S., Chambers, C. D., Percie du Sert, N., Simonsohn, U., Wagenmakers, E.-J., Ware, J. J., & Ioannidis, J. P. A. (2017) · Nature Human Behaviour · doi:10.1038/s41562-016-0021 · Cited by 3,397
When to use the Bonferroni correction
Armstrong, R. A. (2014) · Ophthalmic and Physiological Optics · doi:10.1111/opo.12131 · Cited by 3,031 · cited 1×
“However, many researchers have questioned the usefulness of universal null hypotheses (Armstrong, 2014; Bender & Lange, 2001; Hurlbert & Lombardi, 2012; O’Keefe, 2003; Matsunaga, 2007; Morgan, 2007; Parascandola, 2010; Perneger, 1998; Rothman, 1990; Schulz & Grimes, 2005).”
The preregistration revolution
Nosek, B. A., Ebersole, C. R., DeHaven, A. C., & Mellor, D. T. (2018) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1708274114 · Cited by 1,998
Gergen, K. J. (1973) · Journal of Personality and Social Psychology · doi:10.1037/h0034436 · Cited by 1,845 · cited 1×
“However, this first situation is somewhat hypothetical and may even be regarded as impossible in the social sciences because populations of people change over time and location (e.g., Gergen, 1973; Iso-Ahola, 2017; Schneider, 2015; Serlin, 1987; Stroebe & Strack, 2014).”
BackgroundDo multiple outcome measures require p-value adjustment?
Feise, R. J. (2002) · BMC Medical Research Methodology · doi:10.1186/1471-2288-2-8 · Cited by 1,299
Null hypothesis significance testing: a review of an old and continuing controversy
Nickerson, R. S. (2000) · Psychological Methods · doi:10.1037/1082-989x.5.2.241 · Cited by 1,090
An Agenda for Purely Confirmatory Research
Wagenmakers, E.-J., Wetzels, R., Borsboom, D., van der Maas, H. L. J., & Kievit, R. A. (2012) · Perspectives on Psychological Science · doi:10.1177/1745691612463078 · Cited by 1,042
The Statistical Crisis in Science
Gelman, A., & Loken, E. (2014) · American Scientist · doi:10.1511/2014.111.460 · Cited by 963
A tutorial on sensitivity analyses in clinical trials: the what, why, when and how
Thabane, L., Mbuagbaw, L., Zhang, S., Samaan, Z., Marcucci, M., Ye, C., Thabane, M., Giangregorio, L., Dennis, B., Kosa, D., et al. (2013) · BMC Medical Research Methodology · doi:10.1186/1471-2288-13-92 · Cited by 893
Registered Reports: A Method to Increase the Credibility of Published Results
Nosek, B. A., & Lakens, D. (2014) · Social Psychology · doi:10.1027/1864-9335/a000192 · Cited by 716 · cited 5×
“…out by Nosek, Wagenmakers, and others (Bender & Lange, 2001; Dahl et al., 2008; de Groot, 1956/2014; Forstmeier et al., 2016; Nosek et al., 2017; Nosek & Lakens, 2014; Wagenmakers, 2016) is that although the precise number of tests is known in preregistered confirmatory analyses, it is not…”
BackgroundInfluentialDetecting and avoiding likely false‐positive findings – a practical guide
Forstmeier, W., Wagenmakers, E., & Parker, T. H. (2017) · Biological Reviews · doi:10.1111/brv.12315 · Cited by 436
Multiplicity in randomised trials I: endpoints and treatments
Schulz, K. F., & Grimes, D. A. (2005) · The Lancet · doi:10.1016/s0140-6736(05)66461-6 · Cited by 366
Research works printed but not linked: 21
- 10.7717/peerj.3544 · DOI · Paper page
- 10.1016/S0895-4356(00)00314-0 · DOI · Paper page
- 10.1177/0033294116662659 · DOI
- 10.3758/s13423-015-0913-5 · DOI · Paper page
- 10.1007/s10654-008-9230-x · DOI
- 10.1016/j.actpsy.2014.02.001 · DOI · Paper page
- Forstmeier W., 2016, Biological Reviews of the Cambridge Philosophical Society
- Frane A. V., 2015, Journal of Research Practice, 1, 2
- 10.1111/j.1467-842X.2012.00652.x · DOI
- 10.3389/fpsyg.2017.00879 · DOI
- 10.1080/19312450701641409 · DOI
- 10.1136/ebmh.10.2.34 · DOI
- 10.1111/j.1468-2958.2003.tb00846.x · DOI
- 10.1080/19312450701641383 · DOI
- 10.1093/lpr/mgq005 · DOI
- Schneider J. W., 2015, A mix-up of two different theories: The basis for widespread confusion and numerous misinterpretations. Scientometrics, 102, 411
- 10.1037/0022-0167.34.4.365 · DOI
- 10.1177/1745691616658637 · DOI · Paper page
- 10.1177/1745691613514450 · DOI · Paper page
- 10.1016/j.newideapsych.2017.01.002 · DOI
- 10.1213/ANE.0000000000001107 · DOI
Cited by
Addressing the theory crisis in psychology
Oberauer, K., & Lewandowsky, S. (2019) · Psychonomic Bulletin & Review · doi:10.3758/s13423-019-01645-2 · Cited by 462
Rubin, M. (2021) · Synthese · doi:10.1007/s11229-021-03276-4 · Cited by 371
Self-citationA systematic study of microdosing psychedelics
Polito, V., & Stevenson, R. J. (2019) · PLoS ONE · doi:10.1371/journal.pone.0211023 · Cited by 235
The case for formal methodology in scientific reform
Devezer, B., Navarro, D. J., Vandekerckhove, J., & Buzbas, E. O. (2020) · doi:10.1101/2020.04.26.048306 · Cited by 195
Davis, A. K., So, S., Lancelotta, R., Barsuglia, J. P., & Griffiths, R. R. (2019) · The American Journal of Drug and Alcohol Abuse · doi:10.1080/00952990.2018.1545024 · Cited by 142
Metadata
- Authors
- Rubin, M.
- Venue
- Review of General Psychology
- Publisher
- SAGE Publications
- Published
- 2017-09
- Volume
- 21
- Issue
- 3
- Pages
- 269-275
- ISSN
- 1089-2680 (ISSN-L)
- 1939-1552
- OpenAlex
- W2754878914
- Cited by
- 205
- References
- 15
- FWCI
- 9.42
- Type
- journal article
- Language
- en
- Open access
- green
- Licence
- journals.sagepub.com/page/policies/text-and-data-mining-license (tdm)
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.