An Agenda for Purely Confirmatory Research

Wagenmakers, Eric-Jan; Wetzels, Ruud; Borsboom, Denny; van der Maas, Han L. J.; Kievit, Rogier A.

doi:10.1177/1745691612463078

Cited by 1,042Meta-analysis and systematic reviewsPsychology Research and BibliometricsStatistical Methods and Bayesian Inference

An Agenda for Purely Confirmatory Research

Wagenmakers, Eric-Jan, Wetzels, Ruud, Borsboom, Denny, van der Maas, Han L. J., Kievit, Rogier A.·2012·Perspectives on Psychological Science

DOI 10.1177/1745691612463078

Open this paper in CoChat — free account

Abstract

The veracity of substantive research claims hinges on the way experimental data are collected and analyzed. In this article, we discuss an uncomfortable fact that threatens the core of psychology's academic enterprise: almost without exception, psychologists do not commit themselves to a method of data analysis before they see the actual data. It then becomes tempting to fine tune the analysis to the data in order to obtain a desired result-a procedure that invalidates the interpretation of the common statistical tests. The extent of the fine tuning varies widely across experiments and experimenters but is almost impossible for reviewers and readers to gauge. To remedy the situation, we propose that researchers preregister their studies and indicate in advance the analyses they intend to conduct. Only these analyses deserve the label "confirmatory," and only for these analyses are the common statistical tests valid. Other analyses can be carried out but these should be labeled "exploratory." We illustrate our proposal with a confirmatory replication attempt of a study on extrasensory perception.

Integrity signals

Registry

Crossref · 10.1177/1745691612463078

Cites

48 of 48 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Claims the paper makes

This paper’s full text isn’t openly available, so its claims and self-check weren’t run. The registry and citation checks still apply.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

If hypotheses are included in a preregistration document, then hypothesis generation must have occurred before hypothesis testing.
“If hypotheses are included in a preregistration document, then hypothesis generation must have occurred before hypothesis testing (e.g., Centre for Open Science, n. d; ; Nosek et al., 2018; Wagenmakers et al., 2012).”
From Does preregistration improve the credibility of research findings? (2020) · cites this paper as Wagenmakers et al., 2012

Bayes factors
Kass, R. E., & Raftery, A. E. (1995) · Journal of the American Statistical Association · doi:10.1080/01621459.1995.10476572 · Cited by 12,558 · see citing works
Journal of personality and social psychology
Armstrong, J. (1988) · International Journal of Forecasting · doi:10.1016/0169-2070(88)90126-4 · Cited by 10,544 · see citing works
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · PLoS Medicine · doi:10.1371/journal.pmed.0020124 · Cited by 10,505 · see citing works
Confirmation Bias: A Ubiquitous Phenomenon in Many Guises
Nickerson, R. S. (1998) · Review of General Psychology · doi:10.1037/1089-2680.2.2.175 · Cited by 7,267 · see citing works
False-Positive Psychology: Undisclosed Flexibility in Data Collection and Analysis Allows Presenting Anything as Significant
Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011) · Psychological Science · doi:10.1177/0956797611417632 · Cited by 6,567 · see citing works
Wildlife Society Bulletin
Wildlife Society (2011) · Wildlife Society Bulletin · doi:10.1002/(issn)1938-5463a · Cited by 5,581 · see citing works
Journal of the American Statistical Association
Fildes, R. A. (1989) · International Journal of Forecasting · doi:10.1016/0169-2070(89)90106-4 · Cited by 3,706 · see citing works
Bayesian t tests for accepting and rejecting the null hypothesis
Rouder, J. N., Speckman, P. L., Sun, D., Morey, R. D., & Iverson, G. (2009) · Psychonomic Bulletin & Review · doi:10.3758/pbr.16.2.225 · Cited by 3,688 · see citing works
Raise standards for preclinical cancer research
Begley, C. G., & Ellis, L. M. (2012) · Nature · doi:10.1038/483531a · Cited by 2,979 · cited 1× · see citing works
“And indeed, recent results from biomedical and cancer research suggest that replication rates are lower than 50%, with some as low as 11% (Begley & Ellis, 2012; Osherovich, 2011; Prinz, Schlange, & Asadullah, 2011).”
Result
Circular analysis in systems neuroscience: the dangers of double dipping
Kriegeskorte, N., Simmons, W. K., Bellgowan, P. S. F., & Baker, C. I. (2009) · Nature Neuroscience · doi:10.1038/nn.2303 · Cited by 2,696 · see citing works
A practical solution to the pervasive problems ofp values
Wagenmakers, E.-J. (2007) · Psychonomic Bulletin & Review · doi:10.3758/bf03194105 · Cited by 2,490 · see citing works
Self-citation
Measuring the Prevalence of Questionable Research Practices With Incentives for Truth Telling
John, L. K., Loewenstein, G., & Prelec, D. (2012) · Psychological Science · doi:10.1177/0956797611430953 · Cited by 2,239 · see citing works
Computing Bayes Factors by Combining Simulation and Asymptotic Approximations
DiCiccio, T., Kass, R., Raftery, A., & Wasserman, L. (1997) · Journal of the American Statistical Association · doi:10.1080/01621459.1997.10474045 · Cited by 2,029 · see citing works
HARKing: Hypothesizing after the results are known
Kerr, N. L. (1998) · Personality and Social Psychology Review · doi:10.1207/s15327957pspr0203_4 · Cited by 1,989 · see citing works
Believe it or not: how much can we rely on published data on potential drug targets?
Prinz, F., Schlange, T., & Asadullah, K. (2011) · Nature Reviews Drug Discovery · doi:10.1038/nrd3439-c1 · Cited by 1,888 · see citing works
Bayesian statistical inference for psychological research.
Edwards, W., Lindman, H., & Savage, L. J. (1963) · Psychological Review · doi:10.1037/h0044139 · Cited by 1,734 · see citing works
Model modifications in covariance structure analysis: The problem of capitalization on chance.
MacCallum, R. C., Roznowski, M., & Necowitz, L. B. (1992) · Psychological Bulletin · doi:10.1037/0033-2909.111.3.490 · Cited by 1,676 · see citing works
Puzzlingly High Correlations in fMRI Studies of Emotion, Personality, and Social Cognition
Vul, E., Harris, C., Winkielman, P., & Pashler, H. (2009) · Perspectives on Psychological Science · doi:10.1111/j.1745-6924.2009.01125.x · Cited by 1,475 · see citing works
Untitled work
Nosek (2012) · doi:10.1177/1745691612459058 · Cited by 1,301 · see citing works
Why psychologists must change the way they analyze their data: The case of psi: Comment on Bem (2011).
Wagenmakers, E.-J., Wetzels, R., Borsboom, D., & van der Maas, H. L. J. (2011) · Journal of Personality and Social Psychology · doi:10.1037/a0022790 · Cited by 977 · see citing works
Self-citation
Publication Decisions and their Possible Effects on Inferences Drawn from Tests of Significance—or Vice Versa
Sterling, T. D. (1959) · Journal of the American Statistical Association · doi:10.1080/01621459.1959.10501497 · Cited by 964 · cited 2× · see citing works
“For example, one indication that bias influences the research process is that researchers seek to confirm, not falsify, their main hypothesis (Sterling, 1959; Sterling, Rosenbaum, & Weinkam, 1995).”
Statistical Evidence in Experimental Psychology: An Empirical Comparison Using 855 <i>t</i> Tests
Wetzels, R., Matzke, D., Lee, M. D., Rouder, J. N., Iverson, G. J., & Wagenmakers, E.-J. (2011) · Perspectives on Psychological Science · doi:10.1177/1745691611406923 · Cited by 945 · see citing works
Self-citation
Bayesian Versus Orthodox Statistics: Which Side Are You On?
Dienes, Z. (2011) · Perspectives on Psychological Science · doi:10.1177/1745691611406920 · Cited by 938 · see citing works
Feeling the future: Experimental evidence for anomalous retroactive influences on cognition and affect.
Bem, D. J. (2011) · Journal of Personality and Social Psychology · doi:10.1037/a0021524 · Cited by 893 · see citing works
Calibration of ρ Values for Testing Precise Null Hypotheses
Sellke, T., Bayarri, M. J., & Berger, J. O. (2001) · The American Statistician · doi:10.1198/000313001300339950 · Cited by 856 · see citing works
The rules of the game called psychological science
Bakker, M., van Dijk, A., & Wicherts, J. M. (2012) · Perspectives on Psychological Science · doi:10.1177/1745691612459060 · Cited by 800 · see citing works
Extraordinary popular delusions and the madness of crowds.
Mackay, C. (1932) · Noonday Press eBooks · doi:10.1037/14716-000 · Cited by 778 · see citing works
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2019) · CHANCE · doi:10.1080/09332480.2019.1579573 · Cited by 473 · see citing works
Hindsight Bias
Roese, N. J., & Vohs, K. D. (2012) · Perspectives on Psychological Science · doi:10.1177/1745691612454303 · Cited by 457 · cited 1× · see citing works
“We also know that people fall prey to hindsight bias, the tendency to judge an event as more predictable after it has occurred (Roese & Vohs, 2012).”
Publication Decisions Revisited: The Effect of the Outcome of Statistical Tests on the Decision to Publish and Vice Versa
Sterling, T. D., Rosenbaum, W. L., & Weinkam, J. J. (1995) · The American Statistician · doi:10.1080/00031305.1995.10476125 · Cited by 453 · see citing works
Replication studies: Bad copy
Yong, E. (2012) · Nature · doi:10.1038/485298a · Cited by 406 · see citing works
The importance of proving the null
Gallistel, C. R. (2009) · Psychological Review · doi:10.1037/a0015251 · Cited by 392 · see citing works
Untitled work
(1974) · Cited by 251 · see citing works
Again, and Again, and Again …
Jasny, B. R., Chin, G. J., Chong, L. D., & VIGNIERI, S. N. (2011) · Science · doi:10.1126/science.334.6060.1225 · Cited by 248 · see citing works
Failing the Future: Three Unsuccessful Attempts to Replicate Bem's ‘Retroactive Facilitation of Recall’ Effect
Ritchie, S. J., Wiseman, R. J., & French, C. C. (2012) · PLoS ONE · doi:10.1371/journal.pone.0033423 · Cited by 184 · see citing works
Teaching Replication
Frank, M. C., & Saxe, R. (2012) · Perspectives on Psychological Science · doi:10.1177/1745691612460686 · Cited by 175 · see citing works
Bayesian Evaluation of Informative Hypotheses
Klugkist, I. G., van Wesel, F., van Well, S., Kolk, A. M. M., Hoijtink, H. J. A., Klugkist, I. G., & Boelen, P. A. (2008) · Springer eBooks · doi:10.1007/978-0-387-09612-4 · Cited by 168 · see citing works
Correcting the past: Failures to replicate psi.
Galak, J., LeBoeuf, R. A., Nelson, L. D., & Simmons, J. P. (2012) · Journal of Personality and Social Psychology · doi:10.1037/a0029709 · Cited by 148 · see citing works
How to quantify support for and against the null hypothesis: A flexible WinBUGS implementation of a default Bayesian t test
Wetzels, R., Raaijmakers, J. G. W., Jakab, E., & Wagenmakers, E.-J. (2009) · Psychonomic Bulletin & Review · doi:10.3758/pbr.16.4.752 · Cited by 143 · see citing works
Self-citation
Bayesian Evaluation of Informative Hypotheses
Hoijtink, H. (2013) · PsycEXTRA Dataset · doi:10.1037/e606342013-001 · Cited by 108 · see citing works
Must psychologists change the way they analyze their data?
Bem, D. J., Utts, J., & Johnson, W. O. (2011) · Journal of Personality and Social Psychology · doi:10.1037/a0024777 · Cited by 105 · see citing works
Psychology's Bold Initiative
Carpenter, S. (2012) · Science · doi:10.1126/science.335.6076.1558 · Cited by 101 · see citing works
Psychology’s Woes and a Partial Cure: The Value of Replication
Roediger, H. L., III (2012) · APS observer · Cited by 101 · see citing works
Beware the creeping cracks of bias
Sarewitz, D. (2012) · Nature · doi:10.1038/485149a · Cited by 78 · see citing works
The Nine Circles of Scientific Hell
Neuroskeptic (2012) · Perspectives on Psychological Science · doi:10.1177/1745691612459519 · Cited by 65 · see citing works
Harnessing the Undiscovered Resource of Student Research Projects
Grahe, J. E., Reifman, A. S., Hermann, A. D., Walker, M., Oleson, K. C., Nario‐Redmond, M. R., & Wiebe, R. P. (2012) · Perspectives on Psychological Science · doi:10.1177/1745691612459057 · Cited by 64 · see citing works
Bounds for the Frequency of Misleading Bayes Inferences
Kerridge, D. F. (1963) · The Annals of Mathematical Statistics · doi:10.1214/aoms/1177704038 · Cited by 46 · see citing works
Hedging against academic risk
Osherovich, L. Z. (2011) · Science-Business eXchange · doi:10.1038/scibx.2011.416 · Cited by 43 · see citing works

Flagged first, then by citations

Sorted by citations

Random effects structure for confirmatory hypothesis testing: Keep it maximal
Barr, D. J., Levy, R., Scheepers, C., & Tily, H. J. (2013) · Journal of Memory and Language · doi:10.1016/j.jml.2012.11.001 · Cited by 10,328 · see citing works
The New Statistics: Why and How
Cumming, G. (2014) · Psychological Science · doi:10.1177/0956797613504966 · Cited by 3,200 · see citing works
Promoting an open research culture
Nosek, B. A., Alter, G., Banks, G. C., Borsboom, D., Bowman, S. D., Breckler, S. J., Buck, S., Chambers, C. D., Chin, G., Christensen, G., et al. (2015) · Science · doi:10.1126/science.aab2374 · Cited by 2,708 · see citing works
Self-citation
Using Bayes to get the most out of non-significant results
Dienes, Z. (2014) · Frontiers in Psychology · doi:10.3389/fpsyg.2014.00781 · Cited by 1,998 · see citing works
Choosing Prediction Over Explanation in Psychology: Lessons From Machine Learning
Yarkoni, T., & Westfall, J. (2017) · Perspectives on Psychological Science · doi:10.1177/1745691617693393 · Cited by 1,930 · see citing works
Bayesian inference for psychology. Part I: Theoretical advantages and practical ramifications
Wagenmakers, E., Marsman, M., Jamil, T., Ly, A., Verhagen, J., Love, J., Selker, R., Gronau, Q. F., Šmíra, M., Epskamp, S., et al. (2017) · Psychonomic Bulletin & Review · doi:10.3758/s13423-017-1343-3 · Cited by 1,652 · see citing works
Self-citation
The preregistration revolution
Nosek, B. A., Ebersole, C. R., DeHaven, A. C., & Mellor, D. T. (2018) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1708274114 · Cited by 1,998 · see citing works
Increasing Transparency Through a Multiverse Analysis
Steegen, S., Tuerlinckx, F., Gelman, A., & Vanpaemel, W. (2016) · Perspectives on Psychological Science · doi:10.1177/1745691616658637 · Cited by 1,388 · see citing works
Untitled work
(2013) · Cited by 1,351 · see citing works
Self-citation
Many Labs 2: Investigating variation in replicability across samples and settings
Klein, R. A., Vianello, M., Hasselman, F., Adams, B. G., Adams, R. B., Alper, S., Aveyard, M., Axt, J. R., Babalola, M. T., Bahník, Š., et al. (2018) · Advances in Methods and Practices in Psychological Science · doi:10.1177/2515245918810225 · Cited by 944 · see citing works
Recommendations for Increasing Replicability in Psychology
Asendorpf, J. B., Conner, M. T., Fruyt, F. D., Houwer, J. D., Denissen, J. J. A., Fiedler, K., Fiedler, S., Funder, D. C., Kliegl, R. M., Nosek, B. A., et al. (2013) · European Journal of Personality · doi:10.1002/per.1919 · Cited by 818 · see citing works
Many Analysts, One Data Set: Making Transparent How Variations in Analytic Choices Affect Results
Silberzahn, R., Uhlmann, E. L., Martin, D. P., Anselmi, P., Aust, F., Awtrey, E. C., Bahník, Š., Bai, F., Bannard, C., Bonnier, E., et al. (2018) · Advances in Methods and Practices in Psychological Science · doi:10.1177/2515245917747646 · Cited by 773 · see citing works
Self-citation
The meaningfulness of effect sizes in psychological research: differences between sub-disciplines and the impact of potential biases
Schäfer, T., & Schwarz, M. A. (2019) · Frontiers in Psychology · doi:10.3389/fpsyg.2019.00813 · Cited by 774 · see citing works
Replicability, Robustness, and Reproducibility in Psychological Science
Nosek, B. A., Hardwicke, T. E., Moshontz, H., Allard, A., Corker, K. S., Dreber, A., Fidler, F., Hilgard, J., Kline Struhl, M., Nuijten, M. B., et al. (2022) · Annual Review of Psychology · doi:10.1146/annurev-psych-020821-114157 · Cited by 770 · see citing works
Registered Reports: A Method to Increase the Credibility of Published Results
Nosek, B. A., & Lakens, D. (2014) · Social Psychology · doi:10.1027/1864-9335/a000192 · Cited by 716 · see citing works
Degrees of Freedom in Planning, Running, Analyzing, and Reporting Psychological Studies: A Checklist to Avoid p-Hacking
Wicherts, J. M., Veldkamp, C. L. S., Augusteijn, H. E. M., Bakker, M., van Aert, R. C. M., & van Assen, M. A. L. M. (2016) · Frontiers in Psychology · doi:10.3389/fpsyg.2016.01832 · Cited by 754 · see citing works
Psychology, Science, and Knowledge Construction: Broadening Perspectives from the Replication Crisis
Shrout, P. E., & Rodgers, J. L. (2018) · Annual Review of Psychology · doi:10.1146/annurev-psych-122216-011845 · Cited by 740 · see citing works
The Replication Recipe: What makes for a convincing replication?
Brandt, M. J., IJzerman, H., Dijksterhuis, A., Farach, F. J., Geller, J., Giner‐Sorolla, R., Grange, J. A., Perugini, M., Spies, J. R., & Veer, A. V. '. (2013) · Journal of Experimental Social Psychology · doi:10.1016/j.jesp.2013.10.005 · Cited by 703 · see citing works
Teenagers, screens and social media: a narrative review of reviews and key studies
Orben, A. (2020) · Social Psychiatry and Psychiatric Epidemiology · doi:10.1007/s00127-019-01825-4 · Cited by 706 · see citing works
A simple introduction to Markov Chain Monte–Carlo sampling
van Ravenzwaaij, D., Cassey, P., & Brown, S. D. (2016) · Psychonomic Bulletin & Review · doi:10.3758/s13423-016-1015-8 · Cited by 622 · see citing works
Facts Are More Important Than Novelty: Replication in the Education Sciences
Makel, M. C., & Plucker, J. A. (2014) · Educational Researcher · doi:10.3102/0013189x14545513 · Cited by 609 · see citing works
Crowdsourcing Consumer Research
Goodman, J. K., & Paolacci, G. (2017) · Journal of Consumer Research · doi:10.1093/jcr/ucx047 · Cited by 594 · see citing works
Sequential hypothesis testing with Bayes factors: Efficiently testing mean differences.
Schönbrodt, F. D., Wagenmakers, E., Zehetleitner, M., & Perugini, M. (2015) · Psychological Methods · doi:10.1037/met0000061 · Cited by 562 · see citing works
Self-citation
Why Science Is Not Necessarily Self-Correcting
Ioannidis, J. P. A. (2012) · Perspectives on Psychological Science · doi:10.1177/1745691612464056 · Cited by 563 · see citing works
The Academic, Economic and Societal Impacts of Open Access: An Evidence-Based Review
Tennant, J. P., Waldner, F., Jacques, D. C., Masuzzo, P., Collister, L. B., & Hartgerink, C. H. J. (2016) · F1000Research · doi:10.12688/f1000research.8460.3 · Cited by 560 · see citing works
Psychology's Renaissance
Nelson, L. D., Simmons, J., & Simonsohn, U. (2018) · Annual Review of Psychology · doi:10.1146/annurev-psych-122216-011836 · Cited by 548 · see citing works
Small Effects: The Indispensable Foundation for a Cumulative Psychological Science
Götz, F. M., Gosling, S. D., & Rentfrow, P. J. (2021) · Perspectives on Psychological Science · doi:10.1177/1745691620984483 · Cited by 553 · see citing works
Optional stopping: No problem for Bayesians
Rouder, J. N. (2014) · Psychonomic Bulletin & Review · doi:10.3758/s13423-014-0595-4 · Cited by 492 · see citing works
Publication and other reporting biases in cognitive sciences: detection, prevalence, and prevention
Ioannidis, J. P. A., Munafò, M. R., Fusar-Poli, P., Nosek, B. A., & David, S. P. (2014) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2014.02.010 · Cited by 471 · see citing works
Addressing the theory crisis in psychology
Oberauer, K., & Lewandowsky, S. (2019) · Psychonomic Bulletin & Review · doi:10.3758/s13423-019-01645-2 · Cited by 462 · see citing works
Good Practices for Real‐World Data Studies of Treatment and/or Comparative Effectiveness: Recommendations from the Joint ISPOR‐ISPE Special Task Force on Real‐World Evidence in Health Care Decision Making
Berger, M. L., Sox, H. C., Willke, R. J., Brixner, D. L., Eichler, H., Goettsch, W., Madigan, D., Makady, A., Schneeweiß, S., Tarricone, R., et al. (2017) · Value in Health · doi:10.1016/j.jval.2017.08.3019 · Cited by 457 · see citing works
Pre-registration in social psychology—A discussion and suggested template
Veer, A. V. '., & Giner‐Sorolla, R. (2016) · Journal of Experimental Social Psychology · doi:10.1016/j.jesp.2016.03.004 · Cited by 436 · see citing works
Detecting and avoiding likely false‐positive findings – a practical guide
Forstmeier, W., Wagenmakers, E., & Parker, T. H. (2017) · Biological Reviews · doi:10.1111/brv.12315 · Cited by 436 · see citing works
Self-citation
Registered Replication Report: Strack, Martin, & Stepper (1988)
Wagenmakers, E.-J., Beek, T., Dijkhoff, L., Gronau, Q. F., Acosta, A., Adams, R. B., Albohn, D. N., Allard, E. S., Benning, S. D., Blouin-Hudon, E.-M., et al. (2016) · Perspectives on Psychological Science · doi:10.1177/1745691616674458 · Cited by 430 · see citing works
Self-citation
Hidden multiplicity in exploratory multiway ANOVA: Prevalence and remedies
Cramer, A. O. J., van Ravenzwaaij, D., Matzke, D., Steingroever, H., Wetzels, R., Grasman, R. P. P. P., Waldorp, L., & Wagenmakers, E. (2015) · Psychonomic Bulletin & Review · doi:10.3758/s13423-015-0913-5 · Cited by 416 · see citing works
Self-citation
Diffusion Models in Experimental Psychology
Voß, A., Nagler, M., & Lerche, V. (2013) · Experimental Psychology (formerly Zeitschrift für Experimentelle Psychologie) · doi:10.1027/1618-3169/a000218 · Cited by 412 · see citing works
Does stereotype threat influence performance of girls in stereotyped domains? A meta-analysis
Flore, P. C., & Wicherts, J. M. (2015) · Journal of School Psychology · doi:10.1016/j.jsp.2014.10.002 · Cited by 410 · see citing works
The Nature–Nurture Debates
Eagly, A. H., & Wood, W. (2013) · Perspectives on Psychological Science · doi:10.1177/1745691613484767 · Cited by 424 · see citing works
How Bayes factors change scientific practice
Dienes, Z. (2016) · Journal of Mathematical Psychology · doi:10.1016/j.jmp.2015.10.003 · Cited by 364 · see citing works
Exploratory Research
Swedberg, R. (2020) · Cambridge University Press eBooks · doi:10.1017/9781108762519.002 · Cited by 408 · see citing works
Why Hypothesis Testers Should Spend Less Time Testing Hypotheses
Scheel, A. M., Tiokhin, L., Isager, P. M., & Lakens, D. (2021) · Perspectives on Psychological Science · doi:10.1177/1745691620966795 · Cited by 382 · see citing works
“One influential proposal was to separate confirmatory (hypothesis-testing) and exploratory (hypothesis-generating) research using preregistration (Wagenmakers et al., 2012).”
Background
Ten simple rules for predictive modeling of individual differences in neuroimaging
Scheinost, D., Noble, S., Horien, C., Greene, A. S., Lake, E. M., Salehi, M., Gao, S., Shen, X., O'Connor, D., Barron, D. S., et al. (2019) · NeuroImage · doi:10.1016/j.neuroimage.2019.02.057 · Cited by 381 · see citing works
A systematic review of Bayesian articles in psychology: The last 25 years.
van de Schoot, R., Winter, S. D., Ryan, O., Zondervan-Zwijnenburg, M. A. J., & Depaoli, S. A. (2017) · Psychological Methods · doi:10.1037/met0000100 · Cited by 376 · see citing works
Questionable research practices in ecology and evolution
Fraser, H., Parker, T., Nakagawa, S., Barnett, A., & Fidler, F. (2018) · PLOS ONE · doi:10.1371/journal.pone.0200303 · Cited by 346 · see citing works
Good practices for real‐world data studies of treatment and/or comparative effectiveness: Recommendations from the joint ISPOR‐ISPE Special Task Force on real‐world evidence in health care decision making
Berger, M. L., Sox, H. C., Willke, R. J., Brixner, D. L., Eichler, H., Goettsch, W., Madigan, D., Makady, A., Schneeweiß, S., Tarricone, R., et al. (2017) · Pharmacoepidemiology and Drug Safety · doi:10.1002/pds.4297 · Cited by 347 · see citing works
A weak scientific basis for gaming disorder: Let us err on the side of caution
van Rooij, A. J., Ferguson, C. J., Colder Carras, M., Kardefelt-Winther, D., Shi, J., Aarseth, E., Bean, A. M., Bergmark, K. H., Brus, A., Coulson, M., et al. (2018) · Journal of Behavioral Addictions · doi:10.1556/2006.7.2018.19 · Cited by 346 · see citing works
A Bayesian Perspective on the Reproducibility Project: Psychology
Etz, A., & Vandekerckhove, J. (2016) · PLOS ONE · doi:10.1371/journal.pone.0149794 · Cited by 295 · see citing works
“[34] for more detail and see Matzke et al.”
Preregistration Is Hard, And Worthwhile
Nosek, B. A., Beck, E. D., Campbell, L., Flake, J. K., Hardwicke, T. E., Mellor, D. T., Veer, A. V. '., & Vazire, S. (2019) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2019.07.009 · Cited by 325 · see citing works
The Sisyphean Cycle of Technology Panics
Orben, A. (2020) · Perspectives on Psychological Science · doi:10.1177/1745691620919372 · Cited by 316 · see citing works
Replication in Social Science
Jeremy Freese, A., & Peterson, D. (2017) · Annual Review of Sociology · doi:10.1146/annurev-soc-060116-053450 · Cited by 323 · see citing works
Challenges for assessing replicability in preclinical cancer biology
Errington, T. M., Denis, A., Perfito, N., Iorns, E., & Nosek, B. A. (2021) · eLife · doi:10.7554/elife.67995 · Cited by 323 · see citing works
Linear mixed-effects models for within-participant psychology experiments: an introductory tutorial and free, graphical user interface (LMMgui)
Magezi, D. A. (2015) · Frontiers in Psychology · doi:10.3389/fpsyg.2015.00002 · Cited by 291 · see citing works
Bayesian tests to quantify the result of a replication attempt.
Verhagen, J., & Wagenmakers, E. (2014) · Journal of Experimental Psychology General · doi:10.1037/a0036731 · Cited by 287 · see citing works
Self-citation
Evaluating Recommender Systems: Survey and Framework
Zangerle, E., & Bauer, C. (2022) · ACM Computing Surveys · doi:10.1145/3556536 · Cited by 288 · see citing works
Qualitative Research Methods
Kazdin, A. E. (2021) · Cambridge University Press eBooks · doi:10.1017/9781108993647.010 · Cited by 279 · see citing works
Bayesian benefits with JASP
Marsman, M., & Wagenmakers, E.-J. (2017) · European Journal of Developmental Psychology · doi:10.1080/17405629.2016.1259614 · Cited by 267 · see citing works
Self-citation
Sailing From the Seas of Chaos Into the Corridor of Stability: Practical Recommendations to Increase the Informational Value of Studies
Lakens, D., & Evers, E. R. K. (2014) · Perspectives on Psychological Science · doi:10.1177/1745691614528520 · Cited by 266 · see citing works
Statistical Rituals: The Replication Delusion and How We Got There
Gigerenzer, G. (2018) · Advances in Methods and Practices in Psychological Science · doi:10.1177/2515245918771329 · Cited by 263 · see citing works
An open investigation of the reproducibility of cancer biology research
Errington, T. M., Iorns, E., Gunn, W., Tan, F. E., Lomax, J., & Nosek, B. A. (2014) · eLife · doi:10.7554/elife.04333 · Cited by 312 · see citing works
Priming Intelligent Behavior: An Elusive Phenomenon
Shanks, D. R., Newell, B. R., Lee, E. H., Balakrishnan, D., Ekelund, L., Cenac, Z., Kavvadia, F., & Moore, C. (2013) · PLoS ONE · doi:10.1371/journal.pone.0056515 · Cited by 240 · see citing works
Research Design in Clinical Psychology
Kazdin, A. E. (2021) · doi:10.1017/9781108993647 · Cited by 240 · see citing works
Exploratory data analysis as a foundation of inductive research
Jebb, A. T., Parrigon, S., & Woo, S. E. (2016) · Human Resource Management Review · doi:10.1016/j.hrmr.2016.08.003 · Cited by 239 · see citing works
A tutorial on testing hypotheses using the Bayes factor.
Hoijtink, H. J. A., Mulder, J., Lissa, C. J. V., & Gu, X. (2019) · Psychological Methods · doi:10.1037/met0000201 · Cited by 237 · see citing works
How Many Participants Do I Need to Test an Interaction? Conducting an Appropriate Power Analysis and Achieving Sufficient Power to Detect an Interaction
Sommet, N., Weissman, D. L., Cheutin, N., & Elliot, A. J. (2023) · Advances in Methods and Practices in Psychological Science · doi:10.1177/25152459231178728 · Cited by 274 · see citing works
Distinguishing between exploratory and confirmatory preclinical research will improve translation
Kimmelman, J., Mogil, J. S., & Dirnagl, U. (2014) · PLoS Biology · doi:10.1371/journal.pbio.1001863 · Cited by 236 · see citing works
A Practical Guide for Transparency in Psychological Science
Klein, O., Hardwicke, T. E., Aust, F., Breuer, J., Danielsson, H., Mohr, A. H., IJzerman, H., Nilsonne, G., Vanpaemel, W., & Frank, M. C. (2018) · Collabra Psychology · doi:10.1525/collabra.158 · Cited by 233 · see citing works
The P600-as-P3 hypothesis revisited: Single-trial analyses reveal that the late EEG positivity following linguistically deviant material is reaction time aligned
Sassenhagen, J., Schlesewsky, M., & Bornkessel‐Schlesewsky, I. (2014) · Brain and Language · doi:10.1016/j.bandl.2014.07.010 · Cited by 220 · see citing works
The three ghosts of medical AI: Can the black-box present deliver?
Quinn, T. P., Jacobs, S., Senadeera, M., Le, V., & Coghlan, S. (2021) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2021.102158 · Cited by 222 · see citing works
Beyond differences in means: robust graphical methods to compare two groups in neuroscience
Rousselet, G. A., Pernet, C. R., & Wilcox, R. R. (2017) · European Journal of Neuroscience · doi:10.1111/ejn.13610 · Cited by 215 · see citing works
A Short (Personal) Future History of Revolution 2.0
Spellman, B. A. (2015) · Perspectives on Psychological Science · doi:10.1177/1745691615609918 · Cited by 213 · see citing works
Modeling psychopathology: From data models to formal theories.
Haslbeck, J. M. B., Ryan, O., Robinaugh, D. J., Waldorp, L., & Borsboom, D. (2021) · Psychological Methods · doi:10.1037/met0000303 · Cited by 222 · see citing works
Self-citation
Do p Values Lose Their Meaning in Exploratory Analyses? It Depends How You Define the Familywise Error Rate
Rubin, M. (2017) · Review of General Psychology · doi:10.1037/gpr0000123 · Cited by 205 · see citing works
Recommendations for Increasing the Transparency of Analysis of Preexisting Data Sets
Weston, S. J., Ritchie, S. J., Rohrer, J. M., & Przybylski, A. K. (2019) · Advances in Methods and Practices in Psychological Science · doi:10.1177/2515245919848684 · Cited by 206 · see citing works
Best research practices in psychology: Illustrating epistemological and pragmatic considerations with the case of relationship science.
Finkel, E. J., Eastwick, P. W., & Reis, H. T. (2015) · Journal of Personality and Social Psychology · doi:10.1037/pspi0000007 · Cited by 201 · see citing works
Transparency in Ecology and Evolution: Real Problems, Real Solutions
Parker, T. H., Forstmeier, W., Koricheva, J., Fidler, F., Hadfield, J. D., Chee, Y. E., Kelly, C. D., Gurevitch, J., & Nakagawa, S. (2016) · Trends in Ecology & Evolution · doi:10.1016/j.tree.2016.07.002 · Cited by 201 · see citing works
Safeguard Power as a Protection Against Imprecise Power Estimates
Perugini, M., Gallucci, M., & Costantini, G. (2014) · Perspectives on Psychological Science · doi:10.1177/1745691614528519 · Cited by 203 · see citing works
The case for formal methodology in scientific reform
Devezer, B., Navarro, D., Vandekerckhove, J., & Buzbas, E. O. (2021) · Royal Society Open Science · doi:10.1098/rsos.200805 · Cited by 195 · see citing works
An empirical assessment of transparency and reproducibility-related research practices in the social sciences (2014–2017)
Hardwicke, T. E., Wallach, J. D., Kidwell, M. C., Bendixen, T., Crüwell, S., & Ioannidis, J. P. A. (2020) · Royal Society Open Science · doi:10.1098/rsos.190806 · Cited by 217 · see citing works
Meta-Analysis of Menstrual Cycle Effects on Women’s Mate Preferences
Wood, W., Kressel, L. M., Joshi, P. D., & Louie, B. E. (2014) · Emotion Review · doi:10.1177/1754073914523073 · Cited by 193 · see citing works
The meaning of “significance” for different types of research [translated and annotated by Eric-Jan Wagenmakers, Denny Borsboom, Josine Verhagen, Rogier Kievit, Marjan Bakker, Angelique Cramer, Dora Matzke, Don Mellenbergh, and Han L. J. van der Maas]
Groot, A. D. D. (2014) · Acta Psychologica · doi:10.1016/j.actpsy.2014.02.001 · Cited by 190 · see citing works
Estimating the Prevalence of Transparency and Reproducibility-Related Research Practices in Psychology (2014–2017)
Hardwicke, T. E., Thibault, R. T., Kosie, J. E., Wallach, J. D., Kidwell, M. C., & Ioannidis, J. P. A. (2022) · Perspectives on Psychological Science · doi:10.1177/1745691620979806 · Cited by 187 · see citing works
The academic, economic and societal impacts of Open Access: an evidence-based review
Tennant, J., Waldner, F., Jacques, D., Masuzzo, P., Collister, L., & Hartgerink, C. H. J. (2016) · F1000Research · doi:10.12688/f1000research.8460.1 · Cited by 182 · see citing works
When Does HARKing Hurt? Identifying When Different Types of Undisclosed Post Hoc Hypothesizing Harm Scientific Progress
Rubin, M. (2017) · Review of General Psychology · doi:10.1037/gpr0000128 · Cited by 181 · see citing works
Reducing bias, increasing transparency and calibrating confidence with preregistration
Hardwicke, T. E., & Wagenmakers, E. (2023) · Nature Human Behaviour · doi:10.1038/s41562-022-01497-2 · Cited by 190 · see citing works
Self-citation
Protecting against researcher bias in secondary data analysis: challenges and potential solutions
Baldwin, J. R., Pingault, J.-B., Schoeler, T., Sallis, H. M., & Munafò, M. R. (2022) · European Journal of Epidemiology · doi:10.1007/s10654-021-00839-0 · Cited by 178 · see citing works
“I Know Things They Don’t Know!”: The Role of Need for Uniqueness in Belief in Conspiracy Theories
Lantian, A., Muller, D., Nurra, C., & Douglas, K. M. (2017) · Social Psychology · doi:10.1027/1864-9335/a000306 · Cited by 176 · see citing works
Scientific Utopia III: Crowdsourcing Science
Uhlmann, E. L., Ebersole, C. R., Chartier, C. R., Errington, T. M., Kidwell, M. C., Lai, C. K., McCarthy, R. J., Riegelman, A., Silberzahn, R., & Nosek, B. A. (2019) · Perspectives on Psychological Science · doi:10.1177/1745691619850561 · Cited by 182 · see citing works
Cochrane’s risk of bias tool in the context of psychotherapy outcome research
Munder, T., & Barth, J. (2018) · Psychotherapy Research · doi:10.1080/10503307.2017.1411628 · Cited by 174 · see citing works
A purely confirmatory replication study of structural brain-behavior correlations
Boekel, W., Wagenmakers, E., Belay, L., Verhagen, J., Brown, S. D., & Forstmann, B. U. (2015) · Cortex · doi:10.1016/j.cortex.2014.11.019 · Cited by 167 · see citing works
Self-citation
Contrast coding choices in a decade of mixed models
Brehm, L. E., & Alday, P. M. (2022) · Journal of Memory and Language · doi:10.1016/j.jml.2022.104334 · Cited by 171 · see citing works
Crowdsourcing hypothesis tests: Making transparent how design choices shape research results.
Landy, J. F., Jia, M. (., Ding, I. L., Viganola, D., Tierney, W., Dreber, A., Johannesson, M., Pfeiffer, T., Ebersole, C. R., Gronau, Q. F., et al. (2020) · Psychological Bulletin · doi:10.1037/bul0000220 · Cited by 168 · see citing works
Self-citation
The Preregistration RevolutionPreprint
Nosek, B. A., Ebersole, C. R., DeHaven, A. C., & Mellor, D. T. (2017) · doi:10.31219/osf.io/2dxu5 · Cited by 220 · see citing works
There’s more than one way to conduct a replication study: Beyond statistical significance.
Anderson, S. F., & Maxwell, S. E. (2016) · Psychological Methods · doi:10.1037/met0000051 · Cited by 167 · see citing works
Questions About Questionable Research Practices in the Field of Management: A Guest Commentary
Banks, G. C., O’Boyle, E. H., Pollack, J. M., White, C. D., Batchelor, J. H., Whelpley, C. E., Abston, K. A., Bennett, A. A., & Adkins, C. L. (2016) · Journal of Management · doi:10.1177/0149206315619011 · Cited by 163 · see citing works
Questionable research practices among italian research psychologists
Agnoli, F., Wicherts, J. M., Veldkamp, C. L. S., Albiero, P., & Cubelli, R. (2017) · PLoS ONE · doi:10.1371/journal.pone.0172792 · Cited by 160 · see citing works
Preliminary evidence of linguistic bias in academic reviewing
Politzer‐Ahles, S., Girolamo, T. M., & Ghali, S. (2020) · Journal of English for Academic Purposes · doi:10.1016/j.jeap.2020.100895 · Cited by 157 · see citing works
Replicability and Reproducibility in Comparative Psychology
Stevens, J. R. (2017) · Frontiers in Psychology · doi:10.3389/fpsyg.2017.00862 · Cited by 155 · see citing works
Machine Learning and Psychological Research: The Unexplored Effect of Measurement
Jacobucci, R., & Grimm, K. J. (2020) · Perspectives on Psychological Science · doi:10.1177/1745691620902467 · Cited by 160 · see citing works
Weight stigma and health behaviors: evidence from the Eating in America Study
Lee, K. M., Hunger, J. M., & Tomiyama, A. J. (2021) · International Journal of Obesity · doi:10.1038/s41366-021-00814-5 · Cited by 159 · see citing works
Seven Easy Steps to Open Science
Crüwell, S., van Doorn, J. B., Etz, A. J., Makel, M. C., Moshontz, H., Niebaum, J. C., Orben, A. C., Parsons, S. D., & Schulte‐Mecklenbeck, M. (2019) · Zeitschrift für Psychologie · doi:10.1027/2151-2604/a000387 · Cited by 153 · see citing works

Preregistration of Analyses of Preexisting Data
Mertens, G., & Krypotos, A. (2019) · Psychologica Belgica · doi:10.5334/pb.493 · Cited by 49 · see citing works
Flexible yet fair: blinding analyses in experimental psychology
Dutilh, G., Sarafoglou, A., & Wagenmakers, E. (2019) · Synthese · doi:10.1007/s11229-019-02456-7 · Cited by 45 · see citing works
Self-citation
Comparing dream to reality: an assessment of adherence of the first generation of preregistered studies
Claesen, A., Gomes, S., Tuerlinckx, F., & Vanpaemel, W. (2021) · Royal Society Open Science · doi:10.1098/rsos.211037 · Cited by 197 · see citing works
The meaning of “significance” for different types of research [translated and annotated by Eric-Jan Wagenmakers, Denny Borsboom, Josine Verhagen, Rogier Kievit, Marjan Bakker, Angelique Cramer, Dora Matzke, Don Mellenbergh, and Han L. J. van der Maas]
Groot, A. D. D. (2014) · Acta Psychologica · doi:10.1016/j.actpsy.2014.02.001 · Cited by 190 · see citing works
Preregister if you want to.
Moore, D. A. (2016) · American Psychologist · doi:10.1037/a0040195 · Cited by 42 · see citing works

Authors: Wagenmakers, E.-J., Wetzels, R., Borsboom, D., van der Maas, H. L. J., & Kievit, R. A.
Affiliations: University of Amsterdam (NL)
University of Amsterdam, The Netherlands
ORCID: Wagenmakers, Eric-Jan
Borsboom, Denny
Kievit, Rogier A.
Venue: Perspectives on Psychological Science
Publisher: SAGE Publications
Published: 2012-11
Volume: 7
Issue: 6
Pages: 632-638
ISSN: 1745-6916 (ISSN-L)
1745-6924
DOI: 10.1177/1745691612463078
OpenAlex: W2123263696
Cited by: 1,042
References: 48
Citations by year: 2026: 57 · 2025: 41 · 2024: 65 · 2023: 85 · 2022: 102 · 2021: 108 · 2020: 114 · 2019: 98 · 2018: 88 · 2017: 86 · 2016: 75 · 2015: 51 · 2014: 43 · 2013: 28 · 2012: 1
FWCI: 44.75
Type: journal article
Language: en
Open access: closed
Licence: journals.sagepub.com/page/policies/text-and-data-mining-license (tdm)
Locations: Perspectives on Psychological Science (publishedVersion)
PubMed (publishedVersion)
UvA-DARE (University of Amsterdam) (publishedVersion)
UvA-DARE (University of Amsterdam) (submittedVersion)
Data Archiving and Networked Services (DANS) (submittedVersion)
Topics: Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
Social Sciences › Psychology › Experimental and Cognitive Psychology › Psychology Research and Bibliometrics
Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods and Bayesian Inference
Keywords: preregistration, confirmatory analysis, exploratory analysis, data snooping, statistical validity, research integrity, extrasensory perception, replication, hypothesis testing

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

An Agenda for Purely Confirmatory Research

Abstract

Integrity signals

Claims the paper makes

Claims other papers make about this one

Cites

Cited by

Related

Record

About these checks