An Evaluation of Four Solutions to the Forking Paths Problem: Adjusted Alpha, Preregistration, Sensitivity Analyses, and Abandoning the Neyman-Pearson Approach
Open accessPublished version of a preprint
The DOI matches its Crossref record; 14 of the 14 references it cites were found in a registry or an index.
Abstract
Gelman and Loken (2013 , 2014 ) proposed that when researchers base their statistical analyses on the idiosyncratic characteristics of a specific sample (e.g., a nonlinear transformation of a variable because it is skewed), they open up alternative analysis paths in potential replications of their study that are based on different samples (i.e., no transformation of the variable because it is not skewed). These alternative analysis paths count as additional (multiple) tests and, consequently, they increase the probability of making a Type I error during hypothesis testing. The present article considers this forking paths problem and evaluates four potential solutions that might be used in psychology and other fields: (a) adjusting the prespecified alpha level, (b) preregistration, (c) sensitivity analyses, and (d) abandoning the Neyman-Pearson approach. It is concluded that although preregistration and sensitivity analyses are effective solutions to p-hacking, they are ineffective against result-neutral forking paths, such as those caused by transforming data. Conversely, although adjusting the alpha level cannot address p-hacking, it can be effective for result-neutral forking paths. Finally, abandoning the Neyman-Pearson approach represents a further solution to the forking paths problem.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Estimating the reproducibility of psychological science
Open Science Collaboration (2015) · Science · doi:10.1126/science.aac4716 · Cited by 8,515
Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011) · Psychological Science · doi:10.1177/0956797611417632 · Cited by 6,567
The ASA Statement on p -Values: Context, Process, and Purpose
Wasserstein, R. L., & Lazar, N. A. (2016) · The American Statistician · doi:10.1080/00031305.2016.1154108 · Cited by 6,306 · cited 3×
“…which the smaller the p value, the less likely it is that the observed data would occur if the null hypothesis was true (Amrhein, Korner-Nievergelt, & Roth, 2017; Biau, Jolles, & Porcher, 2010; Bradley & Brand, 2016; Falissard, 2012; Haig, 2017; Hubbard & Bayarri, 2003; Wasserstein & Lazar, 2016).”
BackgroundResultShaffer, J. P. (1995) · Annual Review of Psychology · doi:10.1146/annurev.ps.46.020195.003021 · Cited by 1,960
A Critical Review and Best‐Practice Recommendations for Control Variable Usage
Bernerth, J. B., & Aguinis, H. (2016) · Personnel Psychology · doi:10.1111/peps.12103 · Cited by 1,731
An Agenda for Purely Confirmatory Research
Wagenmakers, E.-J., Wetzels, R., Borsboom, D., van der Maas, H. L. J., & Kievit, R. A. (2012) · Perspectives on Psychological Science · doi:10.1177/1745691612463078 · Cited by 1,042
The Statistical Crisis in Science
Gelman, A., & Loken, E. (2014) · American Scientist · doi:10.1511/2014.111.460 · Cited by 963
Modern robust statistical methods: An easy way to maximize the accuracy and power of your research.
Erceg-Hurn, D. M., & Mirosevich, V. M. (2008) · American Psychologist · doi:10.1037/0003-066x.63.7.591 · Cited by 959 · cited 1×
“…(e.g., Osborne, 2012), (b) improve the reliability and validity of their measurements (e.g., Schmidt & Hunter, 1999), (c) adjust their statistical tests in order to meet test assumptions (e.g., Erceg-Hurn & Mirosevich, 2008), and (d) control for relevant covariates (e.g., Bernerth & Aguinis, 2016).”
BackgroundBayesian Versus Orthodox Statistics: Which Side Are You On?
Dienes, Z. (2011) · Perspectives on Psychological Science · doi:10.1177/1745691611406920 · Cited by 938 · cited 1×
“Consequently, it is not susceptible to the forking paths problem (Dienes, 2011; see also Bender & Lange, 2001).”
BackgroundA tutorial on sensitivity analyses in clinical trials: the what, why, when and how
Thabane, L., Mbuagbaw, L., Zhang, S., Samaan, Z., Marcucci, M., Ye, C., Thabane, M., Giangregorio, L., Dennis, B., Kosa, D., et al. (2013) · BMC Medical Research Methodology · doi:10.1186/1471-2288-13-92 · Cited by 893 · cited 1×
“A third solution to the forking paths issue is to undertake sensitivity analyses (e.g., Thabane et al., 2013; Wasserstein & Lazar, 2016; Wigboldus & Dotsch, 2016).”
Background
Research works printed but not linked: 28
- 10.7717/peerj.3544 · DOI · Paper page
- 10.1016/S0895-4356(00)00314-0 · DOI · Paper page
- 10.1007/s11999-009-1164-4 · DOI · Paper page
- 10.1177/0033294116662659 · DOI
- 10.1016/j.actpsy.2014.02.001 · DOI · Paper page
- 10.1007/s11999-011-2090-9 · doi:10.1007/s11999-011-2090-9
- Fisher R. A., 1926, Statistical methods for research workers
- Fisher R. A., 1937, The design of experiments
- Fisher R. A., 1955, Journal of the Royal Statistical Society Series B: Methodological, 17, 69, 10.1111/j.2517-6161.1955.tb00180.x · DOI · Paper page
- Fisher R. A., 1956, Statistical methods and scientific inference
- FraneA. V. (2015). Planned hypothesis tests are not necessarily exempt from multiplicity adjustment. Journal of Research Practice, 11, Article P2. Retrieved from http://jrp.icaap.org/index.php/jrp/article/view/514/417
- 10.1177/0013164416667981 · DOI
- Hewes D. E., 2003, Human Communication Research, 29, 448
- 10.1198/0003130031856 · DOI · Paper page
- 10.1177/0959354307086923 · DOI · Paper page
- 10.5735/086.046.0501 · DOI · Paper page
- Jeffreys H., 1961, The theory of probability
- 10.1007/978-94-010-1863-0_14 · DOI · Paper page
- 10.1080/19312450701641409 · DOI
- 10.1017/S030500410001152X · DOI · Paper page
- 10.1037/gpr0000123 · DOI · Paper page
- 10.1016/S0160-2896(99)00024-0 · DOI · Paper page
- 10.1007/s11192-014-1251-5 · DOI
- SilberzahnR., UhlmannE. L., MartinD. P., AnselmiP., AustF., AwtreyE. C. … NosekB. A. (2017). Many analysts, one dataset: Making transparent how variations in analytical choices affect results. Retrieved from https://psyarxiv.com/qkwst
- 10.3389/fpsyg.2016.01444 · DOI · Paper page
- 10.1016/j.jesp.2016.03.004 · DOI · Paper page
- 10.1080/19312450701641391 · DOI
- 10.1007/s11336-015-9445-1 · DOI · Paper page
Cited by
Rubin, M. (2021) · Synthese · doi:10.1007/s11229-021-03276-4 · Cited by 371
Self-citationThe case for formal methodology in scientific reform
Devezer, B., Navarro, D. J., Vandekerckhove, J., & Buzbas, E. O. (2020) · doi:10.1101/2020.04.26.048306 · Cited by 195
The registration continuum in clinical science: A guide toward transparent practices
Benning, S. D., Bachrach, R. L., Smith, E. A., Freeman, A. J., & Wright, A. G. C. (2019) · Journal of Abnormal Psychology · doi:10.1037/abn0000451 · Cited by 77
“The dependencies of various analytic decisions on model diagnostics can even be illustrated to avoid the power penalties required when exploring alternative models without planning (unless the researcher uses Bayesian statistics; Rubin, 2017).”
BackgroundDoes preregistration improve the credibility of research findings?
Rubin, M. (2020) · The Quantitative Methods for Psychology · doi:10.20982/tqmp.16.4.p376 · Cited by 61
“In practice, researchers are unlikely to be interested in this studywise error rate, because the associated studywise hypothesis is not likely to be theoretically meaningful (Rubin, 2017a, 2017b, 2019a).”
BackgroundResultInfluentialSelf-citation
Metadata
- Authors
- Rubin, M.
- Venue
- Review of General Psychology
- Publisher
- SAGE Publications
- Published
- 2017-12
- Volume
- 21
- Issue
- 4
- Pages
- 321-329
- ISSN
- 1089-2680 (ISSN-L)
- 1939-1552
- OpenAlex
- W2771076295
- Cited by
- 36
- References
- 14
- FWCI
- 1.70
- Type
- journal article
- Language
- en
- Open access
- hybrid
- Licence
- journals.sagepub.com/page/policies/text-and-data-mining-license (tdm)
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: An Evaluation of Four Solutions to the Forking Paths Problem: Adjusted Alpha, Preregistration, Sensitivity Analyses, and Abandoning the Neyman-Pearson Approach. https://app.cochat.ai/papers/10.1037/gpr0000135
- The paper
- Rubin, M. (2017). An Evaluation of Four Solutions to the Forking Paths Problem: Adjusted Alpha, Preregistration, Sensitivity Analyses, and Abandoning the Neyman-Pearson Approach. Review of General Psychology. https://doi.org/10.1037/gpr0000135
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.