The DOI matches its Crossref record; 49 of the 49 references it cites were found in a registry or an index.
An Agenda for Purely Confirmatory Research
Abstract
The veracity of substantive research claims hinges on the way experimental data are collected and analyzed. In this article, we discuss an uncomfortable fact that threatens the core of psychology's academic enterprise: almost without exception, psychologists do not commit themselves to a method of data analysis before they see the actual data. It then becomes tempting to fine tune the analysis to the data in order to obtain a desired result-a procedure that invalidates the interpretation of the common statistical tests. The extent of the fine tuning varies widely across experiments and experimenters but is almost impossible for reviewers and readers to gauge. To remedy the situation, we propose that researchers preregister their studies and indicate in advance the analyses they intend to conduct. Only these analyses deserve the label "confirmatory," and only for these analyses are the common statistical tests valid. Other analyses can be carried out but these should be labeled "exploratory." We illustrate our proposal with a confirmatory replication attempt of a study on extrasensory perception.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“If hypotheses are included in a preregistration document, then hypothesis generation must have occurred before hypothesis testing (e.g., Centre for Open Science, n. d; ; Nosek et al., 2018; Wagenmakers et al., 2012).”
From Does preregistration improve the credibility of research findings? (2020) · cites this paper as Wagenmakers et al., 2012
Citations
Cites
Kass, R. E., & Raftery, A. E. (1995) · Journal of the American Statistical Association · doi:10.1080/01621459.1995.10476572 · Cited by 12,558
Journal of personality and social psychology
Armstrong, J. (1988) · International Journal of Forecasting · doi:10.1016/0169-2070(88)90126-4 · Cited by 10,544
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · PLoS Medicine · doi:10.1371/journal.pmed.0020124 · Cited by 10,505
Confirmation Bias: A Ubiquitous Phenomenon in Many Guises
Nickerson, R. S. (1998) · Review of General Psychology · doi:10.1037/1089-2680.2.2.175 · Cited by 7,267
Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011) · Psychological Science · doi:10.1177/0956797611417632 · Cited by 6,567
Wildlife Society (2011) · Wildlife Society Bulletin · doi:10.1002/(issn)1938-5463a · Cited by 5,571
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · CHANCE · doi:10.1080/09332480.2005.10722754 · Cited by 4,702 · cited 1×
“confirmatory experiments, wonky statistics, ESP, Bayesian hypothesis test Several years ago, Ioannidis (2005) famously argued that “most published research findings are false.””
Journal of the American Statistical Association
Fildes, R. A. (1989) · International Journal of Forecasting · doi:10.1016/0169-2070(89)90106-4 · Cited by 3,956
Bayesian t tests for accepting and rejecting the null hypothesis
Rouder, J. N., Speckman, P. L., Sun, D., Morey, R. D., & Iverson, G. (2009) · Psychonomic Bulletin & Review · doi:10.3758/pbr.16.2.225 · Cited by 3,688
Raise standards for preclinical cancer research
Begley, C. G., & Ellis, L. M. (2012) · Nature · doi:10.1038/483531a · Cited by 2,979 · cited 1×
“And indeed, recent results from biomedical and cancer research suggest that replication rates are lower than 50%, with some as low as 11% (Begley & Ellis, 2012; Osherovich, 2011; Prinz, Schlange, & Asadullah, 2011).”
ResultCircular analysis in systems neuroscience: the dangers of double dipping
Kriegeskorte, N., Simmons, W. K., Bellgowan, P. S. F., & Baker, C. I. (2009) · Nature Neuroscience · doi:10.1038/nn.2303 · Cited by 2,696
A practical solution to the pervasive problems ofp values
Wagenmakers, E.-J. (2007) · Psychonomic Bulletin & Review · doi:10.3758/bf03194105 · Cited by 2,490
Self-citationMeasuring the Prevalence of Questionable Research Practices With Incentives for Truth Telling
John, L. K., Loewenstein, G., & Prelec, D. (2012) · Psychological Science · doi:10.1177/0956797611430953 · Cited by 2,239
Computing Bayes Factors by Combining Simulation and Asymptotic Approximations
DiCiccio, T., Kass, R., Raftery, A., & Wasserman, L. (1997) · Journal of the American Statistical Association · doi:10.1080/01621459.1997.10474045 · Cited by 2,029
HARKing: Hypothesizing after the results are known
Kerr, N. L. (1998) · Personality and Social Psychology Review · doi:10.1207/s15327957pspr0203_4 · Cited by 1,989
Believe it or not: how much can we rely on published data on potential drug targets?
Prinz, F., Schlange, T., & Asadullah, K. (2011) · Nature Reviews Drug Discovery · doi:10.1038/nrd3439-c1 · Cited by 1,888
Model modifications in covariance structure analysis: The problem of capitalization on chance.
MacCallum, R. C., Roznowski, M., & Necowitz, L. B. (1992) · Psychological Bulletin · doi:10.1037/0033-2909.111.3.490 · Cited by 1,758
Bayesian statistical inference for psychological research.
Edwards, W., Lindman, H., & Savage, L. J. (1963) · Psychological Review · doi:10.1037/h0044139 · Cited by 1,734
Puzzlingly High Correlations in fMRI Studies of Emotion, Personality, and Social Cognition
Vul, E., Harris, C., Winkielman, P., & Pashler, H. (2009) · Perspectives on Psychological Science · doi:10.1111/j.1745-6924.2009.01125.x · Cited by 1,475
Nosek (2012) · doi:10.1177/1745691612459058 · Cited by 1,301
Wagenmakers, E.-J., Wetzels, R., Borsboom, D., & van der Maas, H. L. J. (2011) · Journal of Personality and Social Psychology · doi:10.1037/a0022790 · Cited by 977
Self-citationSterling, T. D. (1959) · Journal of the American Statistical Association · doi:10.1080/01621459.1959.10501497 · Cited by 964 · cited 2×
“For example, one indication that bias influences the research process is that researchers seek to confirm, not falsify, their main hypothesis (Sterling, 1959; Sterling, Rosenbaum, & Weinkam, 1995).”
Statistical Evidence in Experimental Psychology: An Empirical Comparison Using 855 t Tests
Wetzels, R., Matzke, D., Lee, M. D., Rouder, J. N., Iverson, G. J., & Wagenmakers, E.-J. (2011) · Perspectives on Psychological Science · doi:10.1177/1745691611406923 · Cited by 945
Self-citationBayesian Versus Orthodox Statistics: Which Side Are You On?
Dienes, Z. (2011) · Perspectives on Psychological Science · doi:10.1177/1745691611406920 · Cited by 938
Bem, D. J. (2011) · Journal of Personality and Social Psychology · doi:10.1037/a0021524 · Cited by 893
Calibration of ρ Values for Testing Precise Null Hypotheses
Sellke, T., Bayarri, M. J., & Berger, J. O. (2001) · The American Statistician · doi:10.1198/000313001300339950 · Cited by 856
The rules of the game called psychological science
Bakker, M., van Dijk, A., & Wicherts, J. M. (2012) · Perspectives on Psychological Science · doi:10.1177/1745691612459060 · Cited by 800
Extraordinary popular delusions and the madness of crowds.
Mackay, C. (1932) · Noonday Press eBooks · doi:10.1037/14716-000 · Cited by 778
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2019) · CHANCE · doi:10.1080/09332480.2019.1579573 · Cited by 474
Roese, N. J., & Vohs, K. D. (2012) · Perspectives on Psychological Science · doi:10.1177/1745691612454303 · Cited by 457 · cited 1×
“We also know that people fall prey to hindsight bias, the tendency to judge an event as more predictable after it has occurred (Roese & Vohs, 2012).”
Sterling, T. D., Rosenbaum, W. L., & Weinkam, J. J. (1995) · The American Statistician · doi:10.1080/00031305.1995.10476125 · Cited by 453
Yong, E. (2012) · Nature · doi:10.1038/485298a · Cited by 406
The importance of proving the null
Gallistel, C. R. (2009) · Psychological Review · doi:10.1037/a0015251 · Cited by 392
(1974) · Cited by 251
Jasny, B. R., Chin, G. J., Chong, L. D., & VIGNIERI, S. N. (2011) · Science · doi:10.1126/science.334.6060.1225 · Cited by 248
Ritchie, S. J., Wiseman, R. J., & French, C. C. (2012) · PLoS ONE · doi:10.1371/journal.pone.0033423 · Cited by 198
Frank, M. C., & Saxe, R. (2012) · Perspectives on Psychological Science · doi:10.1177/1745691612460686 · Cited by 175
Bayesian Evaluation of Informative Hypotheses
Klugkist, I. G., van Wesel, F., van Well, S., Kolk, A. M. M., Hoijtink, H. J. A., Klugkist, I. G., & Boelen, P. A. (2008) · Springer eBooks · doi:10.1007/978-0-387-09612-4 · Cited by 168
Correcting the past: Failures to replicate psi.
Galak, J., LeBoeuf, R. A., Nelson, L. D., & Simmons, J. P. (2012) · Journal of Personality and Social Psychology · doi:10.1037/a0029709 · Cited by 165
Wetzels, R., Raaijmakers, J. G. W., Jakab, E., & Wagenmakers, E.-J. (2009) · Psychonomic Bulletin & Review · doi:10.3758/pbr.16.4.752 · Cited by 143
Self-citationBayesian Evaluation of Informative Hypotheses
Hoijtink, H. (2013) · PsycEXTRA Dataset · doi:10.1037/e606342013-001 · Cited by 109
Must psychologists change the way they analyze their data?
Bem, D. J., Utts, J., & Johnson, W. O. (2011) · Journal of Personality and Social Psychology · doi:10.1037/a0024777 · Cited by 105
Carpenter, S. (2012) · Science · doi:10.1126/science.335.6076.1558 · Cited by 101
Psychology’s Woes and a Partial Cure: The Value of Replication
Roediger, H. L., III (2012) · APS observer · Cited by 101
Beware the creeping cracks of bias
Sarewitz, D. (2012) · Nature · doi:10.1038/485149a · Cited by 80
Harnessing the Undiscovered Resource of Student Research Projects
Grahe, J. E., Reifman, A. S., Hermann, A. D., Walker, M., Oleson, K. C., Nario‐Redmond, M. R., & Wiebe, R. P. (2012) · Perspectives on Psychological Science · doi:10.1177/1745691612459057 · Cited by 65
The Nine Circles of Scientific Hell
Neuroskeptic (2012) · Perspectives on Psychological Science · doi:10.1177/1745691612459519 · Cited by 65
Bounds for the Frequency of Misleading Bayes Inferences
Kerridge, D. F. (1963) · The Annals of Mathematical Statistics · doi:10.1214/aoms/1177704038 · Cited by 46
Osherovich, L. Z. (2011) · Science-Business eXchange · doi:10.1038/scibx.2011.416 · Cited by 43
Research works printed but not linked: 4
- Feynman R. P., 1974, Engineering & Science, 37, 10
- Goldacre B., 2009, Bad science
- Jeffreys H., 1961, Theory of probability
- Roediger H. L., 2012, APS Observer, 25, 9
Cited by
Sorted by citations
Random effects structure for confirmatory hypothesis testing: Keep it maximal
Barr, D. J., Levy, R., Scheepers, C., & Tily, H. J. (2013) · Journal of Memory and Language · doi:10.1016/j.jml.2012.11.001 · Cited by 10,328
The New Statistics: Why and How
Cumming, G. (2014) · Psychological Science · doi:10.1177/0956797613504966 · Cited by 3,200
Promoting an open research culture
Nosek, B. A., Alter, G., Banks, G. C., Borsboom, D., Bowman, S. D., Breckler, S. J., Buck, S., Chambers, C. D., Chin, G., Christensen, G., et al. (2015) · Science · doi:10.1126/science.aab2374 · Cited by 2,708
Self-citationUsing Bayes to get the most out of non-significant results
Dienes, Z. (2014) · Frontiers in Psychology · doi:10.3389/fpsyg.2014.00781 · Cited by 1,998
The preregistration revolution
Nosek, B. A., Ebersole, C. R., DeHaven, A. C., & Mellor, D. T. (2018) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1708274114 · Cited by 1,998
Metadata
- Authors
- Wagenmakers, E.-J., Wetzels, R., Borsboom, D., van der Maas, H. L. J., & Kievit, R. A.
- Affiliations
- University of Amsterdam (NL)
- Venue
- Perspectives on Psychological Science
- Publisher
- SAGE Publications
- Published
- 2012-11
- Volume
- 7
- Issue
- 6
- Pages
- 632-638
- ISSN
- 1745-6916 (ISSN-L)
- 1745-6924
- OpenAlex
- W2123263696
- Cited by
- 1,042
- References
- 48
- Citations by year
- 2026: 57 · 2025: 41 · 2024: 65 · 2023: 85 · 2022: 102 · 2021: 108 · 2020: 114 · 2019: 98 · 2018: 88 · 2017: 86 · 2016: 75 · 2015: 51 · 2014: 43 · 2013: 28 · 2012: 1
- FWCI
- 44.75
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Perspectives on Psychological Science (publishedVersion)
- PubMed (publishedVersion)
- UvA-DARE (University of Amsterdam) (publishedVersion)
- UvA-DARE (University of Amsterdam) (submittedVersion)
- Data Archiving and Networked Services (DANS) (submittedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Psychology Research and Bibliometrics
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods and Bayesian Inference
- Keywords
- preregistration, confirmatory analysis, exploratory analysis, data snooping, statistical validity, research integrity, extrasensory perception, replication, hypothesis testing
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
