The DOI matches its Crossref record; 44 of the 44 references it cites were found in a registry or an index.
Increasing Transparency Through a Multiverse Analysis
Abstract
Empirical research inevitably includes constructing a data set by processing raw data into a form ready for statistical analysis. Data processing often involves choices among several reasonable options for excluding, transforming, and coding data. We suggest that instead of performing only one analysis, researchers could perform a multiverse analysis, which involves performing all analyses across the whole set of alternatively processed data sets corresponding to a large set of reasonable scenarios. Using an example focusing on the effect of fertility on religiosity and political attitudes, we show that analyzing a single data set can be misleading and propose a multiverse analysis as an alternative practice. A multiverse analysis offers an idea of how much the conclusions change because of arbitrary choices in data construction and gives pointers as to which choices are most consequential in the fragility of the result.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Kass, R. E., & Raftery, A. E. (1995) · Journal of the American Statistical Association · doi:10.1080/01621459.1995.10476572 · Cited by 12,558
Journal of personality and social psychology
Armstrong, J. S. (1988) · International Journal of Forecasting · doi:10.1016/0169-2070(88)90126-4 · Cited by 10,485
Estimating the reproducibility of psychological science
Open Science Collaboration (2015) · Science · doi:10.1126/science.aac4716 · Cited by 8,515
Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011) · Psychological Science · doi:10.1177/0956797611417632 · Cited by 6,567
Moffitt, R. A. (2009) · American Economic Review · doi:10.1257/aer.99.2.660 · Cited by 4,694
On the practice of dichotomization of quantitative variables.
MacCallum, R. C., Zhang, S., Preacher, K. J., & Rucker, D. D. (2002) · Psychological Methods · doi:10.1037/1082-989x.7.1.19 · Cited by 3,538
Promoting an open research culture
Nosek, B. A., Alter, G., Banks, G. C., Borsboom, D., Bowman, S. D., Breckler, S. J., Buck, S., Chambers, C. D., Chin, G., Christensen, G., et al. (2015) · Science · doi:10.1126/science.aab2374 · Cited by 2,708
Measuring the Prevalence of Questionable Research Practices With Incentives for Truth Telling
John, L. K., Loewenstein, G., & Prelec, D. (2012) · Psychological Science · doi:10.1177/0956797611430953 · Cited by 2,239
Let's Take the Con out of Econometrics
Leamer, E. E. (2026) · Journal of Business Cycle Research · doi:10.1007/s41549-026-00137-2 · Cited by 2,202
(2011) · Cited by 2,025
P-curve: A key to the file-drawer.
Simonsohn, U., Nelson, L. D., & Simmons, J. P. (2014) · Journal of Experimental Psychology: General · doi:10.1037/a0033242 · Cited by 1,500
Why We (Usually) Don't Have to Worry About Multiple Comparisons
Gelman, A., Hill, J., & Yajima, M. (2012) · Journal of Research on Educational Effectiveness · doi:10.1080/19345747.2011.618213 · Cited by 1,274
Self-citationThe statistical sleuth : a course in methods of data analysis
Ramsey, F. L., & Schafer, D. W. (2002) · Cited by 1,166
An Agenda for Purely Confirmatory Research
Wagenmakers, E.-J., Wetzels, R., Borsboom, D., van der Maas, H. L. J., & Kievit, R. A. (2012) · Perspectives on Psychological Science · doi:10.1177/1745691612463078 · Cited by 1,042 · cited 1×
“Preregistration (e.g., Chambers, 2013; Wagenmakers et al., 2012) or blind analysis (e.”
BackgroundWagenmakers, E.-J., Wetzels, R., Borsboom, D., & van der Maas, H. L. J. (2011) · Journal of Personality and Social Psychology · doi:10.1037/a0022790 · Cited by 977
The Statistical Crisis in Science
Gelman, A., & Loken, E. (2014) · American Scientist · doi:10.1511/2014.111.460 · Cited by 963
Self-citationPredictive Inference: An Introduction
Geisser, S. (1993) · Springer eBooks · doi:10.1007/978-1-4899-4467-2 · Cited by 917
Recommendations for Increasing Replicability in Psychology
Asendorpf, J. B., Conner, M., De Fruyt, F., De Houwer, J., Denissen, J. J. A., Fiedler, K., Fiedler, S., Funder, D. C., Kliegl, R., Nosek, B. A., et al. (2013) · European Journal of Personality · doi:10.1002/per.1919 · Cited by 823
Revised standards for statistical evidence
Johnson, V. E. (2013) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1313476110 · Cited by 808
The rules of the game called psychological science
Bakker, M., van Dijk, A., & Wicherts, J. M. (2012) · Perspectives on Psychological Science · doi:10.1177/1745691612459060 · Cited by 800
Understanding The New Statistics: Effect Sizes, Confidence Intervals, and Meta-Analysis
Cumming, G. (2013) · Routledge eBooks · doi:10.4324/9780203807002 · Cited by 790
Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2012) · doi:10.2139/ssrn.2160588 · Cited by 680
Bayes factor approaches for testing interval null hypotheses.
Morey, R. D., & Rouder, J. N. (2011) · Psychological Methods · doi:10.1037/a0024377 · Cited by 582
Registered Reports: A new publishing initiative at Cortex
Chambers, C. D. (2013) · Cortex · doi:10.1016/j.cortex.2012.12.016 · Cited by 570
(1985) · American Economic Review · Cited by 568
(2017) · Routledge eBooks · doi:10.1201/9780203742310 · Cited by 568
The Statistical Sleuth: A Course in Methods of Data Analysis
Kilgo, M. M. (1998) · Journal of Quality Technology · doi:10.1080/00224065.1998.11979882 · Cited by 420
Yong, E. (2012) · Nature · doi:10.1038/485298a · Cited by 405
The Scent of Symmetry A Human Sex Pheromone that Signals Fitness?
Thornhill, R. (1999) · Evolution and Human Behavior · doi:10.1016/s1090-5138(99)00005-7 · Cited by 392
Scientific Utopia: I. Opening Scientific Communication
Nosek, B. A., & Bar-Anan, Y. (2012) · Psychological Inquiry · doi:10.1080/1047840x.2012.692215 · Cited by 368
What to believe: Bayesian methods for data analysis
Kruschke, J. K. (2010) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2010.05.001 · Cited by 342 · cited 1×
“…analysis is valuable, regardless of the inferential framework (frequentist or Bayesian), and regardless of the specific way uncertainty is quantified: a p value, an effect size, a confidence (Cumming, 2013) or credibility (Kruschke, 2010) interval, or a Bayes factor (Morey & Rouder, 2011).”
BackgroundOvulation, Female Competition, and Product Choice: Hormonal Influences on Consumer Behavior
Durante, K. M., Griskevicius, V., Hill, S. E., Perilloux, C., & Li, N. P. (2011) · Journal of Consumer Research · doi:10.1086/656575 · Cited by 309
The Peer Reviewers' Openness Initiative: incentivizing open research practices through peer review
Morey, R. D., Chambers, C. D., Etchells, P. J., Harris, C. R., Hoekstra, R., Lakens, D., Lewandowsky, S., Morey, C. C., Newman, D. P., Schönbrodt, F. D., et al. (2016) · Royal Society Open Science · doi:10.1098/rsos.150547 · Cited by 284
Self-citationPatel, C. J., Burford, B., & Ioannidis, J. P. A. (2015) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2015.05.029 · Cited by 274
Women’s fertility across the cycle increases the short-term attractiveness of creative intelligence
Haselton, M. G., & Miller, G. F. (2006) · Human Nature · doi:10.1007/s12110-006-1020-0 · Cited by 224
Blind analysis: Hide results to seek the truth
MacCoun, R., & Perlmutter, S. (2015) · Nature · doi:10.1038/526187a · Cited by 218 · cited 1×
“Preregistration (e.g., Chambers, 2013; Wagenmakers et al., 2012) or blind analysis (e.g., MacCoun & Perlmutter, 2015) are not useful strategies for deflating the multiverse.”
MethodGangestad, S. W., Haselton, M. G., Welling, L. L. M., Gildersleeve, K., Pillsworth, E. G., Burriss, R. P., Larson, C. M., & Puts, D. A. (2016) · Evolution and Human Behavior · doi:10.1016/j.evolhumbehav.2015.09.001 · Cited by 207
Money, Status, and the Ovulatory Cycle
Durante, K. M., Griskevicius, V., Cantú, S. M., & Simpson, J. A. (2013) · Journal of Marketing Research · doi:10.1509/jmr.11.0327 · Cited by 100
The Fluctuating Female Vote: Politics, Religion, and the Ovulatory Cycle
Durante, K. M., Arsena, A., & Griskevicius, V. (2013) · Psychological Science · doi:10.1177/0956797612466416 · Cited by 95
Ovulation leads women to perceive sexy cads as good dads.
Durante, K. M., Griskevicius, V., Simpson, J. A., Cantú, S. M., & Li, N. P. (2012) · Journal of Personality and Social Psychology · doi:10.1037/a0028498 · Cited by 85
Playing the Field: The Effect of Fertility on Women’s Desire for Variety
Durante, K. M., & Arsena, A. (2014) · Journal of Consumer Research · doi:10.1086/679652 · Cited by 83
Shifts in Masculinity Preferences Across the Menstrual Cycle: Still Not There
Harris, C. R. (2012) · Sex Roles · doi:10.1007/s11199-012-0229-0 · Cited by 56
Harris, C. R., Pashler, H., & Mickes, L. (2014) · Psychological Bulletin · doi:10.1037/a0036478 · Cited by 44
Fertility Can Have Different Effects on Single and Nonsingle Women
Durante, K. M., Arsena, A., & Griskevicius, V. (2014) · Psychological Science · doi:10.1177/0956797614524422 · Cited by 9
Cited by
The JASP guidelines for conducting and reporting a Bayesian analysis
van Doorn, J., van den Bergh, D., Böhm, U., Dablander, F., Derks, K., Draws, T., Etz, A., Evans, N. J., Gronau, Q. F., Haaf, J. M., et al. (2021) · Psychonomic Bulletin & Review · doi:10.3758/s13423-020-01798-5 · Cited by 1,114
Variability in the analysis of a single neuroimaging dataset by many teams
(2020) · doi:10.1038/s41586-020-2314-9 · Cited by 1,097
Thinking Clearly About Correlations and Causation: Graphical Causal Models for Observational Data
Rohrer, J. M. (2018) · Advances in Methods and Practices in Psychological Science · doi:10.1177/2515245917745629 · Cited by 1,002
“Jackson, Thoemmes, Jonkmann, Lüdtke, and Trautwein (2012) were interested in the effects of military training (in comparison with civilian community service) on personality.”
BackgroundMeasurement Schmeasurement: Questionable Measurement Practices and How to Avoid Them
Flake, J. K., & Fried, E. I. (2020) · Advances in Methods and Practices in Psychological Science · doi:10.1177/2515245920952393 · Cited by 969
Wicherts, J. M., Veldkamp, C. L. S., Augusteijn, H. E. M., Bakker, M., van Aert, R. C. M., & van Assen, M. A. L. M. (2016) · Frontiers in Psychology · doi:10.3389/fpsyg.2016.01832 · Cited by 754
Metadata
- Authors
- Steegen, S., Tuerlinckx, F., Gelman, A., & Vanpaemel, W.
- Venue
- Perspectives on Psychological Science
- Publisher
- SAGE Publications
- Published
- 2016-09
- Volume
- 11
- Issue
- 5
- Pages
- 702-712
- ISSN
- 1745-6916 (ISSN-L)
- 1745-6924
- OpenAlex
- W2526501380
- Cited by
- 1,372
- References
- 44
- FWCI
- 53.24
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.