The DOI matches its Crossref record; 49 of the 49 references it cites were found in a registry or an index.
An Agenda for Purely Confirmatory Research
Abstract
The veracity of substantive research claims hinges on the way experimental data are collected and analyzed. In this article, we discuss an uncomfortable fact that threatens the core of psychology's academic enterprise: almost without exception, psychologists do not commit themselves to a method of data analysis before they see the actual data. It then becomes tempting to fine tune the analysis to the data in order to obtain a desired result-a procedure that invalidates the interpretation of the common statistical tests. The extent of the fine tuning varies widely across experiments and experimenters but is almost impossible for reviewers and readers to gauge. To remedy the situation, we propose that researchers preregister their studies and indicate in advance the analyses they intend to conduct. Only these analyses deserve the label "confirmatory," and only for these analyses are the common statistical tests valid. Other analyses can be carried out but these should be labeled "exploratory." We illustrate our proposal with a confirmatory replication attempt of a study on extrasensory perception.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“If hypotheses are included in a preregistration document, then hypothesis generation must have occurred before hypothesis testing (e.g., Centre for Open Science, n. d; ; Nosek et al., 2018; Wagenmakers et al., 2012).”
From Does preregistration improve the credibility of research findings? (2020) · cites this paper as Wagenmakers et al., 2012
Citations
Cites
Kass, R. E., & Raftery, A. E. (1995) · Journal of the American Statistical Association · doi:10.1080/01621459.1995.10476572 · Cited by 12,558
Journal of personality and social psychology
Armstrong, J. (1988) · International Journal of Forecasting · doi:10.1016/0169-2070(88)90126-4 · Cited by 10,544
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · PLoS Medicine · doi:10.1371/journal.pmed.0020124 · Cited by 10,505
Confirmation Bias: A Ubiquitous Phenomenon in Many Guises
Nickerson, R. S. (1998) · Review of General Psychology · doi:10.1037/1089-2680.2.2.175 · Cited by 7,267
Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011) · Psychological Science · doi:10.1177/0956797611417632 · Cited by 6,567
Wildlife Society (2011) · Wildlife Society Bulletin · doi:10.1002/(issn)1938-5463a · Cited by 5,571
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · CHANCE · doi:10.1080/09332480.2005.10722754 · Cited by 4,702 · cited 1×
“confirmatory experiments, wonky statistics, ESP, Bayesian hypothesis test Several years ago, Ioannidis (2005) famously argued that “most published research findings are false.””
Journal of the American Statistical Association
Fildes, R. A. (1989) · International Journal of Forecasting · doi:10.1016/0169-2070(89)90106-4 · Cited by 3,956
Bayesian t tests for accepting and rejecting the null hypothesis
Rouder, J. N., Speckman, P. L., Sun, D., Morey, R. D., & Iverson, G. (2009) · Psychonomic Bulletin & Review · doi:10.3758/pbr.16.2.225 · Cited by 3,688
Raise standards for preclinical cancer research
Begley, C. G., & Ellis, L. M. (2012) · Nature · doi:10.1038/483531a · Cited by 2,979 · cited 1×
“And indeed, recent results from biomedical and cancer research suggest that replication rates are lower than 50%, with some as low as 11% (Begley & Ellis, 2012; Osherovich, 2011; Prinz, Schlange, & Asadullah, 2011).”
Result
Research works printed but not linked: 4
- Feynman R. P., 1974, Engineering & Science, 37, 10
- Goldacre B., 2009, Bad science
- Jeffreys H., 1961, Theory of probability
- Roediger H. L., 2012, APS Observer, 25, 9
Cited by
Sorted by citations
Items 801–900 of 998
Reznick, J. S. (2017) · Cited by 1
Attentional bias: a methodological review
Chew, P. (2015) · ResearchOnline - JCU (James Cook University) · Cited by 1
Are Danes’ Immigration Policy Preferences Based on Accurate Stereotypes?
Kirkegaard, E. O. W., Carl, N., & Bjerrekær, J. D. (2020) · Societies · doi:10.3390/soc10020029 · Cited by 1
The extraordinary resilience of Great Barrier Reef corals, and problems with policy science
Ridd, P. (2017) · ResearchOnline - JCU (James Cook University) · Cited by 1
BAYESOV FAKTOR: OPIS I RAZLOZI ZA UPOTREBU U PSIHOLOŠKIM ISTRAŽIVANJIMA
Lakić, S. (2019) · ГОДИШЊАК ЗА ПСИХОЛОГИЈУ · doi:10.46630/gpsi.18.2019.03 · Cited by 1
アンへドニア（anhedonia）と遅延割引：Lempert & Pizzagalli（2010）の追試
Somatori, K., & 愛彦, 国. (2019) · 心理学評論 · doi:10.24602/sjpr.62.3_231 · Cited by 1
The Reproducibility Crisis in Science Education Studies: Causes and Solutions
Nakamura, D., HARADA, Y., HISASAKA, T., UNZAI, H., & Matsuura, T. (2021) · Journal of Research in Science Education · doi:10.11639/sjst.sp20016 · Cited by 1
Case-Control and Cohort Designs
Kazdin, A. E. (2021) · Cambridge University Press eBooks · doi:10.1017/9781108993647.008 · Cited by 1
Null Hypothesis Significance Testing
Kazdin, A. E. (2021) · Cambridge University Press eBooks · doi:10.1017/9781108993647.014 · Cited by 1
Gerring, J., Mahoney, J., & Elman, C. (2020) · Cambridge University Press eBooks · doi:10.1017/9781108762519.018 · Cited by 1
Gerring, J., & Cojocaru, L. (2020) · Cambridge University Press eBooks · doi:10.1017/9781108762519.005 · Cited by 1
Gerring, J. (2020) · Cambridge University Press eBooks · doi:10.1017/9781108762519.014 · Cited by 1
Gross, N., & Robertson, C. T. (2020) · Cambridge University Press eBooks · doi:10.1017/9781108762519.017 · Cited by 1
Oberski, D. L. (2016) · Oxford University Press eBooks · doi:10.1093/oxfordhb/9780190213299.013.21 · Cited by 1
Common Misconceptions and Misunderstandings in Magic Cut-Off for Significance: P-Value
EDEN, A. B., & İnan, N. G. (2022) · Proceedings of the International Conference on Statistics, Theory and Applications (ICSTA ...) · doi:10.11159/icsta22.125 · Cited by 1
Brinkman, E. H., Beijers, R., Tyborowska, A., Karin Roelofs, A., Kühn, S., Kievit, R., & de Weerth, C. (2024) · Psychoneuroendocrinology · doi:10.1016/j.psyneuen.2023.106789 · Cited by 1
Self-citationGood science, bad science: Questioning research practices in psychological research
Bakker, M. (2014) · UvA-DARE (University of Amsterdam) · Cited by 1
The Influence of Direct and Indirect Speech on Mental RepresentationsPreprint
Eerland, A., Engelen, J. A. A., & Zwaan, R. A. (2017) · doi:10.31234/osf.io/f5jdy · Cited by 1
The Solution to Science's Replication CrisisPreprint
Knuteson, B. (2016) · SSRN Electronic Journal · doi:10.2139/ssrn.2835131 · Cited by 2
Noone, C. (2016) · Arrow@dit (Dublin Institute of Technology) · doi:10.13025/17145 · Cited by 1
Introduction to the special issue – A replication project in personality psychology
Donnellan, M. B., & Lucas, R. E. (2017) · Journal of Research in Personality · doi:10.1016/j.jrp.2017.11.004 · Cited by 1
Research on Research: A Meta-Scientific Study of Problems and Solutions in Psychological SciencePreprint
Nuijten, M. B. (2018) · PsyArXiv (OSF Preprints) · doi:10.31234/osf.io/qtk7e · Cited by 1
Development and Application of a Vignette Measure of Self-Compassion
Redden, E. K. (2019) · The Atrium (University of Guelph) · Cited by 1
Understanding individual differences in learning and consolidating new vocabulary
James, E. (2019) · White Rose eTheses Online (University of Leeds, The University of Sheffield, University of York) · Cited by 1
Is N-Hacking Ever OK? A simulation-based inquiryPreprint
Reinagel, P. (2019) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2019.12.12.868489 · Cited by 1
Socially Thermoregulated Thinking: How Past Experiences Matter in Thinking about Our Loved OnesPreprint
IJzerman, H., Neyroud, L., Courset, R., Schrama, M., Post, J., & Pronk, T. (2017) · doi:10.31234/osf.io/25dmj · Cited by 1
From mother to Mother Nature: The role of interpersonal attachment in human–nature relations
Capaldi, C. A. (2020) · Carleton University's Institutional Repository (MacOdrum Library, Carleton University) · doi:10.22215/etd/2020-13953 · Cited by 1
A call to accompany pre-registration with means to valuable and effective explorationPreprint
Höfler, M., Scherbaum, S., Kanske, P., & Miller, R. E. (2021) · doi:10.31234/osf.io/psfqw · Cited by 1
Jankowsky, K., Steger, D., & Schroeders, U. (2022) · doi:10.31234/osf.io/sa37x · Cited by 1
Do owners know how impulsive their dogs are?Preprint
Stevens, J. R., Mathias, M., Herridge, M., Hughes-Duvall, K., Wolff, L. M., & Yohe, M. (2022) · doi:10.31234/osf.io/hyvdq · Cited by 1
Open Science in Psychophysiology: An Overview of Challenges and Emerging SolutionsPreprint
Garrett-Ruffin, S., Hindash, A. C., Kaczkurkin, A. N., Mears, R. P., Muriel, S. M., Paul, K., Pavlov, Y. G., & Keil, A. (2020) · doi:10.31234/osf.io/ryspq · Cited by 1
Rausch, M., Hellmann, S., & Zehetleitner, M. (2017) · doi:10.31234/osf.io/dmwa4 · Cited by 1
The Science of Technology and Human Behavior: Standards, Old and NewPreprint
Elson, M., & Przybylski, A. K. (2017) · PsyArXiv (OSF Preprints) · doi:10.31234/osf.io/u58mq · Cited by 2
Variable Control and Conceptual Change: A Large-Scale Quantitative Study in Elementary SchoolPreprint
Edelsbrunner, P. A., Schalk, L., Schumacher, R., & Stern, E. (2016) · PsyArXiv (OSF Preprints) · doi:10.31234/osf.io/escqa · Cited by 1
Is Comfort Food Really Good for the Soul? A Replication of Troisi and Gabriel’s (2011) Study 2Preprint
Ong, L., Leung, A. K. ‐., & IJzerman, H. (2017) · doi:10.31234/osf.io/jn9pz · Cited by 1
How preschoolers react to norm violations is associated with culturePreprint
Gampe, A., & Daum, M. M. (2017) · doi:10.31219/osf.io/cm9ag · Cited by 1
Publishing open, reproducible research with undergraduatesPreprint
Strand, J. F., & Brown, V. A. (2019) · doi:10.31234/osf.io/f7kuy · Cited by 1
Ramsey_modestyPreprint
Ramsey, R. (2020) · doi:10.31234/osf.io/hf5sv · Cited by 1
Klau, S., Schönbrodt, F. D., Patel, C. J., Ioannidis, J. P. A., Boulesteix, A., & Hoffmann, S. (2020) · doi:10.31234/osf.io/c7v8b · Cited by 1
Computational optimization of associative learning experimentsPreprint
Melinščak, F., & Bach, D. R. (2019) · OSF Preprints (OSF Preprints) · doi:10.31219/osf.io/cgpmh · Cited by 4
Aarts, A. A. (2022) · SSRN Electronic Journal · doi:10.2139/ssrn.4176917 · Cited by 2
Can Our Claims be Verified? Open Science Practices in Information Systems ResearchPreprint
Parry, D. A. (2022) · MetArXiv (OSF Preprints) · doi:10.31222/osf.io/u295h · Cited by 1
Rethinking replication in empirical legal researchPreprint
Chin, J. M., & Holcombe, A. O. (2022) · doi:10.31222/osf.io/4up65 · Cited by 1
Ubels, J., Hernández-Villafuerte, K., & Schlander, M. (2022) · medRxiv · doi:10.1101/2022.10.05.22280720 · Cited by 1
The Process of Replication Target Selection in Psychology: What to Consider?Preprint
Pittelkow, M.-M., Field, S. M., Isager, P. M., Veer, A. V. '., Anderson, T., Cole, S., Dominik, T., Giner‐Sorolla, R., Gök, S., Heyman, T., et al. (2022) · PsyArXiv (OSF Preprints) · doi:10.31234/osf.io/4pw63 · Cited by 1
Atılgan, H., Koi, J. J. X., Wong, E., Laakso, I., Matilainen, N., Pasqualotto, A., Tanaka, S., Chen, S. A., & Kitada, R. (2022) · doi:10.31234/osf.io/gcf5z · Cited by 1
Margoni, F., & Walkinshaw, N. (2023) · doi:10.31234/osf.io/xzuy5 · Cited by 1
Reproducible research practices and transparency across linguisticsPreprint
Bochyńska, A., Keeble, L., Halfacre, C. A. E., Casillas, J. V., Champagne, I.-A., Chen, K., Röthlisberger, M., Buchanan, E. M., & Roettger, T. B. (2023) · MetArXiv (OSF Preprints) · doi:10.31222/osf.io/rcews · Cited by 2
Hirschhorn, R., & Schönberg, T. (2023) · Elsevier eBooks · doi:10.1016/b978-0-12-820480-1.00014-0 · Cited by 1
The Open Empirical Cycle for Hypothesis Evaluation in PsychologyPreprint
Hoijtink, H., Bruin, J. D., Duken, S. B., Flores-Dourojeanni, J. P., Frankenhuis, W. E., & Lissa, C. J. V. (2023) · PsyArXiv (OSF Preprints) · doi:10.31234/osf.io/wsxbh · Cited by 3
dynConfiR: The R package for sequential sampling models of decision confidencePreprint
Hellmann, S., Zehetleitner, M., & Rausch, M. (2024) · PsyArXiv (OSF Preprints) · doi:10.31234/osf.io/e354s · Cited by 1
A review of pre-registration in autism research journalsPreprint
Poole, D., Linden, A. H., Sedgewick, F., & Hobson, H. (2024) · doi:10.31222/osf.io/9382m · Cited by 1
Noise Versus Signal: What Can One Conclude When a Classic Finding Fails to Replicate?
Cyrus-Lai, W., Tierney, W., & Uhlmann, E. L. (2024) · Advances in Methods and Practices in Psychological Science · doi:10.1177/25152459241268293 · Cited by 2
Katsimpokis, D., van Odenhoven, A. E. C., van Erp, M. A. J. M., Wenzel, H. H. B., van der Aa, M. A., van Swieten, M. M. H., Smedts, D., & Piek, J. M. J. (2025) · medRxiv · doi:10.1101/2025.03.04.25321571 · Cited by 1
de Zubicaray, G. I. (2026) · Language Cognition and Neuroscience · doi:10.1080/23273798.2026.2683465 · Cited by 1
Tobin, C., & LeBel, E. P. (2014) · SSRN Electronic Journal · doi:10.2139/ssrn.2459994 · Cited by 1
Reproducible Protein NMR Data Analysis
Fenwick, M. (2014) · OpenCommons - UConn (University of Connecticut)
Incentivizing Transparency in Psychological Science: Enter ScienpostingPreprint
Gullo, M., & O'Gorman, J. G. (2013) · SSRN Electronic Journal · doi:10.2139/ssrn.2258209
The hierarchical organization of impulse control: Implications for decision making
Coutlee, C. G. (2014) · DukeSpace (Duke University)
LeBel, E. P., & Campbell, L. (2013)
Boekel, W., Wagenmakers, E., Belay, L., Verhagen, J., Brown, S., & Forstmann, B. U. (2014) · F1000Research · doi:10.7490/f1000research.1096084.1
Self-citationC. S. Peirce on the Crisis of Confidence and the “No More Bets” HeuristicPreprint
Vandekerckhove, J., & Wagenmakers, E. (2016) · doi:10.15200/winn.146611.14253
Self-citationTitle: Efficiently Exploring Multilevel Data with Recursive Partitioning Authors and Affiliations:
Martin, D. P., von Oertzen, T., & Rimm‐Kaufman, S. E. (2015)
Tierney, W., Schweinsberg, M., Jordan, J., Kennedy, D. M., Qureshi, I., Sommer, S. A., Thornley, N., Madan, N., Vianello, M., Awtrey, E. C., et al. (2016) · Kölner Universitäts PublikationsServer (Universität zu Köln)
The Effects of Conscious and Primed Learning Goals on the Performance of a Complex Task
McRuer, G. J. K. (2015) · TSpace (University of Toronto)
Veldkamp, C. L. S., Hartgerink, C. H. J., van Assen, M. A. L. M., & Wicherts, J. M. (2016) · PsyArXiv (OSF Preprints) · doi:10.31234/osf.io/uk6ju
Preprint of Too good to be false: Nonsignificant results revisitedPreprint
Hartgerink, C. H. J., Wicherts, J. M., & van Assen, M. A. L. M. (2016) · OSF Preprints (OSF Preprints) · doi:10.31219/osf.io/rkumy
Research Integrity in Public Health
Gorman, D. M. (2018) · Oxford University Press eBooks · doi:10.1093/obo/9780199756797-0189
Preregistered replication of Stoeber (2012): Dyadic perfectionism in romantic relationshipsPreprint
Williams, M. N., & Gill, C. (2020) · doi:10.31234/osf.io/2br58
Am I Luckier Than You? Individual Differences in Precognition Performance
Zdrenka, M. (2018) · Open Access Victoria University of Wellington | Te Herenga Waka (Figshare) · doi:10.26686/wgtn.19314362
Sauer, S., & Sülzenbrück, S. (2018) · Arbeitswelten der Zukunft · doi:10.1007/978-3-658-23397-6_11
Replication and preregistration
Krause, M. S. (2019) · Quality & Quantity · doi:10.1007/s11135-019-00877-7
Mellenbergh, G. J. (2019) · Springer eBooks · doi:10.1007/978-3-030-12272-0_12
Author response: Open exploration
Thompson, W. H., Wright, J., & Bissett, P. G. (2019) · eLife · doi:10.7554/elife.52157.sa2
Decision letter: Open exploration
(2019) · eLife · doi:10.7554/elife.52157.sa1
Oswald, C. (2020) · Carolina Digital Repository (University of North Carolina at Chapel Hill) · doi:10.17615/68ye-rt64
Shedding light on cognitive control
Skau, S. (2020) · Gothenburg University Publications Electronic Archive (Gothenburg University)
Towards a motivational alternative to the strength model of self-control
Lees, D. A. (2016)
The Solution to Science's Replication CrisisPreprint
Knuteson, B. (2016) · RePEc: Research Papers in Economics · doi:10.48550/arxiv.1609.03223
McVay, M. A., Scotti, K. B., Seoane, M. C., Donahue, M. L., & Scherer, L. D. (2020) · doi:10.31234/osf.io/nx4fk
Statistical consequences of staging exploration and confirmation
Pashler, H., & Harris, C. R. (2021) · Methods in Psychology · doi:10.1016/j.metip.2021.100078
Leaps of logic: quantifying the behavioural relevance of hippocampal neurogenesisPreprint
Lazic, S. E., Fuß, J., & Gass, P. (2014) · arXiv (Cornell University)
Kazdin, A. E. (2021) · Cambridge University Press eBooks · doi:10.1017/9781108993647.001
Construct and Data-Evaluation Validity
Kazdin, A. E. (2021) · Cambridge University Press eBooks · doi:10.1017/9781108993647.004
Single-Case Experimental Research Designs
Kazdin, A. E. (2021) · Cambridge University Press eBooks · doi:10.1017/9781108993647.009
Kazdin, A. E. (2021) · Cambridge University Press eBooks · doi:10.1017/9781108993647.007
Cautions, Negative Effects, and Replication
Kazdin, A. E. (2021) · Cambridge University Press eBooks · doi:10.1017/9781108993647.016
Selecting Measures for Research
Kazdin, A. E. (2021) · Cambridge University Press eBooks · doi:10.1017/9781108993647.011
Kazdin, A. E. (2021) · Cambridge University Press eBooks · doi:10.1017/9781108993647.021
Kazdin, A. E. (2021) · Cambridge University Press eBooks · doi:10.1017/9781108993647.013
Kazdin, A. E. (2021) · Cambridge University Press eBooks · doi:10.1017/9781108993647.022
Ideas that Begin the Research Process
Kazdin, A. E. (2021) · Cambridge University Press eBooks · doi:10.1017/9781108993647.005
Kazdin, A. E. (2021) · Cambridge University Press eBooks · doi:10.1017/9781108993647.020
Kazdin, A. E. (2021) · Cambridge University Press eBooks · doi:10.1017/9781108993647.025
Presenting and Analyzing the Data
Kazdin, A. E. (2021) · Cambridge University Press eBooks · doi:10.1017/9781108993647.015
Exploratory Only: A Tool for Large-Scale Exploratory AnalysesPreprint
Kim, J. (2021) · doi:10.31234/osf.io/37fvm · Cited by 1
Kazdin, A. E. (2021) · Cambridge University Press eBooks · doi:10.1017/9781108993647.024
Kazdin, A. E. (2021) · Cambridge University Press eBooks · doi:10.1017/9781108993647.018
Metadata
- Authors
- Wagenmakers, E.-J., Wetzels, R., Borsboom, D., van der Maas, H. L. J., & Kievit, R. A.
- Affiliations
- University of Amsterdam (NL)
- Venue
- Perspectives on Psychological Science
- Publisher
- SAGE Publications
- Published
- 2012-11
- Volume
- 7
- Issue
- 6
- Pages
- 632-638
- ISSN
- 1745-6916 (ISSN-L)
- 1745-6924
- OpenAlex
- W2123263696
- Cited by
- 1,042
- References
- 48
- Citations by year
- 2026: 57 · 2025: 41 · 2024: 65 · 2023: 85 · 2022: 102 · 2021: 108 · 2020: 114 · 2019: 98 · 2018: 88 · 2017: 86 · 2016: 75 · 2015: 51 · 2014: 43 · 2013: 28 · 2012: 1
- FWCI
- 44.75
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Perspectives on Psychological Science (publishedVersion)
- PubMed (publishedVersion)
- UvA-DARE (University of Amsterdam) (publishedVersion)
- UvA-DARE (University of Amsterdam) (submittedVersion)
- Data Archiving and Networked Services (DANS) (submittedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Psychology Research and Bibliometrics
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods and Bayesian Inference
- Keywords
- preregistration, confirmatory analysis, exploratory analysis, data snooping, statistical validity, research integrity, extrasensory perception, replication, hypothesis testing
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
