The DOI matches its Crossref record; 14 of the 14 references it cites were found in a registry or an index.
Do p Values Lose Their Meaning in Exploratory Analyses? It Depends How You Define the Familywise Error Rate
Open accessPublished version of a preprint
Abstract
Several researchers have recently argued that p values lose their meaning in exploratory analyses due to an unknown inflation of the alpha level (e.g., Nosek & Lakens, 2014 ; Wagenmakers, 2016 ). For this argument to be tenable, the familywise error rate must be defined in relation to the number of hypotheses that are tested in the same study or article. Under this conceptualization, the familywise error rate is usually unknowable in exploratory analyses because it is usually unclear how many hypotheses have been tested on a spontaneous basis and then omitted from the final research report. In the present article, I argue that it is inappropriate to conceptualize the familywise error rate in relation to the number of hypotheses that are tested. Instead, it is more appropriate to conceptualize familywise error in relation to the number of different tests that are conducted on the same null hypothesis in the same study. Under this conceptualization, alpha-level adjustments in exploratory analyses are (a) less necessary and (b) objectively verifiable. As a result, p values do not lose their meaning in exploratory analyses.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011) · Psychological Science · doi:10.1177/0956797611417632 · Cited by 6,567
No adjustments are needed for multiple comparisons.
Rothman, K. J. (1990) · Epidemiology · doi:10.1097/00001648-199001000-00010 · Cited by 5,965
What's wrong with Bonferroni adjustments
Perneger, T. V. (1998) · BMJ · doi:10.1136/bmj.316.7139.1236 · Cited by 5,900
A manifesto for reproducible science
Munafò, M. R., Nosek, B. A., Bishop, D. V. M., Button, K. S., Chambers, C. D., Percie du Sert, N., Simonsohn, U., Wagenmakers, E.-J., Ware, J. J., & Ioannidis, J. P. A. (2017) · Nature Human Behaviour · doi:10.1038/s41562-016-0021 · Cited by 3,397
When to use the Bonferroni correction
Armstrong, R. A. (2014) · Ophthalmic and Physiological Optics · doi:10.1111/opo.12131 · Cited by 3,031 · cited 1×
“However, many researchers have questioned the usefulness of universal null hypotheses (Armstrong, 2014; Bender & Lange, 2001; Hurlbert & Lombardi, 2012; O’Keefe, 2003; Matsunaga, 2007; Morgan, 2007; Parascandola, 2010; Perneger, 1998; Rothman, 1990; Schulz & Grimes, 2005).”
The preregistration revolution
Nosek, B. A., Ebersole, C. R., DeHaven, A. C., & Mellor, D. T. (2018) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1708274114 · Cited by 1,998
Gergen, K. J. (1973) · Journal of Personality and Social Psychology · doi:10.1037/h0034436 · Cited by 1,845
Do multiple outcome measures require p-value adjustment?
Feise, R. J. (2002) · BMC Medical Research Methodology · doi:10.1186/1471-2288-2-8 · Cited by 1,299
Null hypothesis significance testing: a review of an old and continuing controversy
Nickerson, R. S. (2000) · Psychological Methods · doi:10.1037/1082-989x.5.2.241 · Cited by 1,090
An Agenda for Purely Confirmatory Research
Wagenmakers, E.-J., Wetzels, R., Borsboom, D., van der Maas, H. L. J., & Kievit, R. A. (2012) · Perspectives on Psychological Science · doi:10.1177/1745691612463078 · Cited by 1,042
Research works printed but not linked: 22
- 10.7717/peerj.3544 · DOI · Paper page
- 10.1016/S0895-4356(00)00314-0 · DOI · Paper page
- 10.1177/0033294116662659 · DOI
- 10.3758/s13423-015-0913-5 · DOI · Paper page
- 10.1007/s10654-008-9230-x · DOI
- 10.1016/j.actpsy.2014.02.001 · DOI · Paper page
- Forstmeier W., 2016, Biological Reviews of the Cambridge Philosophical Society
- Frane A. V., 2015, Journal of Research Practice, 1, 2
- 10.1511/2014.111.460 · DOI · Paper page
- 10.1111/j.1467-842X.2012.00652.x · DOI
- 10.3389/fpsyg.2017.00879 · DOI
- 10.1080/19312450701641409 · DOI
- 10.1136/ebmh.10.2.34 · DOI
- 10.1111/j.1468-2958.2003.tb00846.x · DOI
- 10.1080/19312450701641383 · DOI
- 10.1093/lpr/mgq005 · DOI
- Schneider J. W., 2015, A mix-up of two different theories: The basis for widespread confusion and numerous misinterpretations. Scientometrics, 102, 411
- 10.1037/0022-0167.34.4.365 · DOI
- 10.1177/1745691616658637 · DOI · Paper page
- 10.1177/1745691613514450 · DOI · Paper page
- 10.1016/j.newideapsych.2017.01.002 · DOI
- 10.1213/ANE.0000000000001107 · DOI
Cited by
Addressing the theory crisis in psychology
Oberauer, K., & Lewandowsky, S. (2019) · Psychonomic Bulletin & Review · doi:10.3758/s13423-019-01645-2 · Cited by 462
Rubin, M. (2021) · Synthese · doi:10.1007/s11229-021-03276-4 · Cited by 371
Self-citationA systematic study of microdosing psychedelics
Polito, V., & Stevenson, R. J. (2019) · PLoS ONE · doi:10.1371/journal.pone.0211023 · Cited by 235
The case for formal methodology in scientific reform
Devezer, B., Navarro, D. J., Vandekerckhove, J., & Buzbas, E. O. (2020) · doi:10.1101/2020.04.26.048306 · Cited by 195
Davis, A. K., So, S., Lancelotta, R., Barsuglia, J. P., & Griffiths, R. R. (2019) · The American Journal of Drug and Alcohol Abuse · doi:10.1080/00952990.2018.1545024 · Cited by 142
Metadata
- Authors
- Rubin, M.
- Venue
- Review of General Psychology
- Publisher
- SAGE Publications
- Published
- 2017-09
- Volume
- 21
- Issue
- 3
- Pages
- 269-275
- ISSN
- 1089-2680 (ISSN-L)
- 1939-1552
- OpenAlex
- W2754878914
- Cited by
- 205
- References
- 14
- FWCI
- 9.42
- Type
- journal article
- Language
- en
- Open access
- green
- Licence
- journals.sagepub.com/page/policies/text-and-data-mining-license (tdm)
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Do p Values Lose Their Meaning in Exploratory Analyses? It Depends How You Define the Familywise Error Rate. https://app.cochat.ai/papers/10.1037/gpr0000123
- The paper
- Rubin, M. (2017). Do p Values Lose Their Meaning in Exploratory Analyses? It Depends How You Define the Familywise Error Rate. Review of General Psychology. https://doi.org/10.1037/gpr0000123
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.