Cited by 1,042Meta-analysis and systematic reviewsPsychology Research and BibliometricsStatistical Methods and Bayesian Inference

The DOI matches its Crossref record; 49 of the 49 references it cites were found in a registry or an index.

An Agenda for Purely Confirmatory Research

Wagenmakers, Eric-Jan, Wetzels, Ruud, Borsboom, Denny, van der Maas, Han L. J., Kievit, Rogier A.·2012·Perspectives on Psychological Science

DOI 10.1177/1745691612463078

Open this paper in CoChat — free account

Abstract

The veracity of substantive research claims hinges on the way experimental data are collected and analyzed. In this article, we discuss an uncomfortable fact that threatens the core of psychology's academic enterprise: almost without exception, psychologists do not commit themselves to a method of data analysis before they see the actual data. It then becomes tempting to fine tune the analysis to the data in order to obtain a desired result-a procedure that invalidates the interpretation of the common statistical tests. The extent of the fine tuning varies widely across experiments and experimenters but is almost impossible for reviewers and readers to gauge. To remedy the situation, we propose that researchers preregister their studies and indicate in advance the analyses they intend to conduct. Only these analyses deserve the label "confirmatory," and only for these analyses are the common statistical tests valid. Other analyses can be carried out but these should be labeled "exploratory." We illustrate our proposal with a confirmatory replication attempt of a study on extrasensory perception.

Integrity signals

Registry

Crossref · 10.1177/1745691612463078

Cites

49 of 49 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Claims the paper makes

This paper’s full text isn’t openly available, so its claims and self-check weren’t run. The registry and citation checks still apply.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Bayes factors
Kass, R. E., & Raftery, A. E. (1995) · Journal of the American Statistical Association · doi:10.1080/01621459.1995.10476572 · Cited by 12,558
Journal of personality and social psychology
Armstrong, J. (1988) · International Journal of Forecasting · doi:10.1016/0169-2070(88)90126-4 · Cited by 10,544
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · PLoS Medicine · doi:10.1371/journal.pmed.0020124 · Cited by 10,505
Confirmation Bias: A Ubiquitous Phenomenon in Many Guises
Nickerson, R. S. (1998) · Review of General Psychology · doi:10.1037/1089-2680.2.2.175 · Cited by 7,267
False-Positive Psychology: Undisclosed Flexibility in Data Collection and Analysis Allows Presenting Anything as Significant
Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011) · Psychological Science · doi:10.1177/0956797611417632 · Cited by 6,567
Wildlife Society Bulletin
Wildlife Society (2011) · Wildlife Society Bulletin · doi:10.1002/(issn)1938-5463a · Cited by 5,571
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · CHANCE · doi:10.1080/09332480.2005.10722754 · Cited by 4,702 · cited 1×
“confirmatory experiments, wonky statistics, ESP, Bayesian hypothesis test Several years ago, Ioannidis (2005) famously argued that “most published research findings are false.””
Journal of the American Statistical Association
Fildes, R. A. (1989) · International Journal of Forecasting · doi:10.1016/0169-2070(89)90106-4 · Cited by 3,956
Bayesian t tests for accepting and rejecting the null hypothesis
Rouder, J. N., Speckman, P. L., Sun, D., Morey, R. D., & Iverson, G. (2009) · Psychonomic Bulletin & Review · doi:10.3758/pbr.16.2.225 · Cited by 3,688
Raise standards for preclinical cancer research
Begley, C. G., & Ellis, L. M. (2012) · Nature · doi:10.1038/483531a · Cited by 2,979 · cited 1×
“And indeed, recent results from biomedical and cancer research suggest that replication rates are lower than 50%, with some as low as 11% (Begley & Ellis, 2012; Osherovich, 2011; Prinz, Schlange, & Asadullah, 2011).”
Result

Show all 48 references

Research works printed but not linked: 4

Feynman R. P., 1974, Engineering & Science, 37, 10
Goldacre B., 2009, Bad science
Jeffreys H., 1961, Theory of probability
Roediger H. L., 2012, APS Observer, 25, 9

Sorted by citations

Random effects structure for confirmatory hypothesis testing: Keep it maximal
Barr, D. J., Levy, R., Scheepers, C., & Tily, H. J. (2013) · Journal of Memory and Language · doi:10.1016/j.jml.2012.11.001 · Cited by 10,328
The New Statistics: Why and How
Cumming, G. (2014) · Psychological Science · doi:10.1177/0956797613504966 · Cited by 3,200
Promoting an open research culture
Nosek, B. A., Alter, G., Banks, G. C., Borsboom, D., Bowman, S. D., Breckler, S. J., Buck, S., Chambers, C. D., Chin, G., Christensen, G., et al. (2015) · Science · doi:10.1126/science.aab2374 · Cited by 2,708
Self-citation
Using Bayes to get the most out of non-significant results
Dienes, Z. (2014) · Frontiers in Psychology · doi:10.3389/fpsyg.2014.00781 · Cited by 1,998
The preregistration revolution
Nosek, B. A., Ebersole, C. R., DeHaven, A. C., & Mellor, D. T. (2018) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1708274114 · Cited by 1,998
Choosing Prediction Over Explanation in Psychology: Lessons From Machine Learning
Yarkoni, T., & Westfall, J. (2017) · Perspectives on Psychological Science · doi:10.1177/1745691617693393 · Cited by 2,088
Bayesian inference for psychology. Part I: Theoretical advantages and practical ramifications
Wagenmakers, E., Marsman, M., Jamil, T., Ly, A., Verhagen, J., Love, J., Selker, R., Gronau, Q. F., Šmíra, M., Epskamp, S., et al. (2017) · Psychonomic Bulletin & Review · doi:10.3758/s13423-017-1343-3 · Cited by 1,767
Self-citation
Increasing Transparency Through a Multiverse Analysis
Steegen, S., Tuerlinckx, F., Gelman, A., & Vanpaemel, W. (2016) · Perspectives on Psychological Science · doi:10.1177/1745691616658637 · Cited by 1,512
Untitled work
(2013) · Cited by 1,351
Self-citation
Many Labs 2: Investigating variation in replicability across samples and settings
Klein, R. A., Vianello, M., Hasselman, F., Adams, B. G., Adams, R. B., Alper, S., Aveyard, M., Axt, J. R., Babalola, M. T., Bahník, Š., et al. (2018) · Advances in Methods and Practices in Psychological Science · doi:10.1177/2515245918810225 · Cited by 944
Recommendations for Increasing Replicability in Psychology
Asendorpf, J. B., Conner, M. T., Fruyt, F. D., Houwer, J. D., Denissen, J. J. A., Fiedler, K., Fiedler, S., Funder, D. C., Kliegl, R. M., Nosek, B. A., et al. (2013) · European Journal of Personality · doi:10.1002/per.1919 · Cited by 876
Many Analysts, One Data Set: Making Transparent How Variations in Analytic Choices Affect Results
Silberzahn, R., Uhlmann, E. L., Martin, D. P., Anselmi, P., Aust, F., Awtrey, E. C., Bahník, Š., Bai, F., Bannard, C., Bonnier, E., et al. (2018) · Advances in Methods and Practices in Psychological Science · doi:10.1177/2515245917747646 · Cited by 773
Self-citation
The meaningfulness of effect sizes in psychological research: differences between sub-disciplines and the impact of potential biases
Schäfer, T., & Schwarz, M. A. (2019) · Frontiers in Psychology · doi:10.3389/fpsyg.2019.00813 · Cited by 774
Replicability, Robustness, and Reproducibility in Psychological Science
Nosek, B. A., Hardwicke, T. E., Moshontz, H., Allard, A., Corker, K. S., Dreber, A., Fidler, F., Hilgard, J., Kline Struhl, M., Nuijten, M. B., et al. (2022) · Annual Review of Psychology · doi:10.1146/annurev-psych-020821-114157 · Cited by 770
Registered Reports: A Method to Increase the Credibility of Published Results
Nosek, B. A., & Lakens, D. (2014) · Social Psychology · doi:10.1027/1864-9335/a000192 · Cited by 716
“Confirmatory results follow a preregistered analysis plan and thereby ensure interpretability of the reported p -values (Wagenmakers et al., 2012).”
Background
Degrees of Freedom in Planning, Running, Analyzing, and Reporting Psychological Studies: A Checklist to Avoid p-Hacking
Wicherts, J. M., Veldkamp, C. L. S., Augusteijn, H. E. M., Bakker, M., van Aert, R. C. M., & van Assen, M. A. L. M. (2016) · Frontiers in Psychology · doi:10.3389/fpsyg.2016.01832 · Cited by 754
Psychology, Science, and Knowledge Construction: Broadening Perspectives from the Replication Crisis
Shrout, P. E., & Rodgers, J. L. (2018) · Annual Review of Psychology · doi:10.1146/annurev-psych-122216-011845 · Cited by 830
The Replication Recipe: What makes for a convincing replication?
Brandt, M. J., IJzerman, H., Dijksterhuis, A., Farach, F. J., Geller, J., Giner‐Sorolla, R., Grange, J. A., Perugini, M., Spies, J. R., & Veer, A. V. '. (2013) · Journal of Experimental Social Psychology · doi:10.1016/j.jesp.2013.10.005 · Cited by 758
Teenagers, screens and social media: a narrative review of reviews and key studies
Orben, A. (2020) · Social Psychiatry and Psychiatric Epidemiology · doi:10.1007/s00127-019-01825-4 · Cited by 706
A simple introduction to Markov Chain Monte–Carlo sampling
van Ravenzwaaij, D., Cassey, P., & Brown, S. D. (2016) · Psychonomic Bulletin & Review · doi:10.3758/s13423-016-1015-8 · Cited by 622
Facts Are More Important Than Novelty: Replication in the Education Sciences
Makel, M. C., & Plucker, J. A. (2014) · Educational Researcher · doi:10.3102/0013189x14545513 · Cited by 609
Crowdsourcing Consumer Research
Goodman, J. K., & Paolacci, G. (2017) · Journal of Consumer Research · doi:10.1093/jcr/ucx047 · Cited by 597
Sequential hypothesis testing with Bayes factors: Efficiently testing mean differences.
Schönbrodt, F. D., Wagenmakers, E., Zehetleitner, M., & Perugini, M. (2015) · Psychological Methods · doi:10.1037/met0000061 · Cited by 599
Self-citation
Why Science Is Not Necessarily Self-Correcting
Ioannidis, J. P. A. (2012) · Perspectives on Psychological Science · doi:10.1177/1745691612464056 · Cited by 563
The Academic, Economic and Societal Impacts of Open Access: An Evidence-Based Review
Tennant, J. P., Waldner, F., Jacques, D. C., Masuzzo, P., Collister, L. B., & Hartgerink, C. H. J. (2016) · F1000Research · doi:10.12688/f1000research.8460.3 · Cited by 560
Psychology's Renaissance
Nelson, L. D., Simmons, J., & Simonsohn, U. (2018) · Annual Review of Psychology · doi:10.1146/annurev-psych-122216-011836 · Cited by 548
Small Effects: The Indispensable Foundation for a Cumulative Psychological Science
Götz, F. M., Gosling, S. D., & Rentfrow, P. J. (2021) · Perspectives on Psychological Science · doi:10.1177/1745691620984483 · Cited by 610
Optional stopping: No problem for Bayesians
Rouder, J. N. (2014) · Psychonomic Bulletin & Review · doi:10.3758/s13423-014-0595-4 · Cited by 492
Publication and other reporting biases in cognitive sciences: detection, prevalence, and prevention
Ioannidis, J. P. A., Munafò, M. R., Fusar-Poli, P., Nosek, B. A., & David, S. P. (2014) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2014.02.010 · Cited by 471
Addressing the theory crisis in psychology
Oberauer, K., & Lewandowsky, S. (2019) · Psychonomic Bulletin & Review · doi:10.3758/s13423-019-01645-2 · Cited by 462
Good Practices for Real‐World Data Studies of Treatment and/or Comparative Effectiveness: Recommendations from the Joint ISPOR‐ISPE Special Task Force on Real‐World Evidence in Health Care Decision Making
Berger, M. L., Sox, H. C., Willke, R. J., Brixner, D. L., Eichler, H., Goettsch, W., Madigan, D., Makady, A., Schneeweiß, S., Tarricone, R., et al. (2017) · Value in Health · doi:10.1016/j.jval.2017.08.3019 · Cited by 464
Pre-registration in social psychology—A discussion and suggested template
Veer, A. V. '., & Giner‐Sorolla, R. (2016) · Journal of Experimental Social Psychology · doi:10.1016/j.jesp.2016.03.004 · Cited by 464
Detecting and avoiding likely false‐positive findings – a practical guide
Forstmeier, W., Wagenmakers, E., & Parker, T. H. (2017) · Biological Reviews · doi:10.1111/brv.12315 · Cited by 436
Self-citation
Registered Replication Report: Strack, Martin, & Stepper (1988)
Wagenmakers, E.-J., Beek, T., Dijkhoff, L., Gronau, Q. F., Acosta, A., Adams, R. B., Albohn, D. N., Allard, E. S., Benning, S. D., Blouin-Hudon, E.-M., et al. (2016) · Perspectives on Psychological Science · doi:10.1177/1745691616674458 · Cited by 430
Self-citation
Hidden multiplicity in exploratory multiway ANOVA: Prevalence and remedies
Cramer, A. O. J., van Ravenzwaaij, D., Matzke, D., Steingroever, H., Wetzels, R., Grasman, R. P. P. P., Waldorp, L., & Wagenmakers, E. (2015) · Psychonomic Bulletin & Review · doi:10.3758/s13423-015-0913-5 · Cited by 440
Self-citation
Diffusion Models in Experimental Psychology
Voß, A., Nagler, M., & Lerche, V. (2013) · Experimental Psychology (formerly Zeitschrift für Experimentelle Psychologie) · doi:10.1027/1618-3169/a000218 · Cited by 447
Does stereotype threat influence performance of girls in stereotyped domains? A meta-analysis
Flore, P. C., & Wicherts, J. M. (2015) · Journal of School Psychology · doi:10.1016/j.jsp.2014.10.002 · Cited by 410
The Nature–Nurture Debates
Eagly, A. H., & Wood, W. (2013) · Perspectives on Psychological Science · doi:10.1177/1745691613484767 · Cited by 471
How Bayes factors change scientific practice
Dienes, Z. (2016) · Journal of Mathematical Psychology · doi:10.1016/j.jmp.2015.10.003 · Cited by 365
Exploratory Research
Swedberg, R. (2020) · Cambridge University Press eBooks · doi:10.1017/9781108762519.002 · Cited by 471
Why Hypothesis Testers Should Spend Less Time Testing Hypotheses
Scheel, A. M., Tiokhin, L., Isager, P. M., & Lakens, D. (2021) · Perspectives on Psychological Science · doi:10.1177/1745691620966795 · Cited by 382
“One influential proposal was to separate confirmatory (hypothesis-testing) and exploratory (hypothesis-generating) research using preregistration (Wagenmakers et al., 2012).”
Background
Ten simple rules for predictive modeling of individual differences in neuroimaging
Scheinost, D., Noble, S., Horien, C., Greene, A. S., Lake, E. M., Salehi, M., Gao, S., Shen, X., O'Connor, D., Barron, D. S., et al. (2019) · NeuroImage · doi:10.1016/j.neuroimage.2019.02.057 · Cited by 381
A systematic review of Bayesian articles in psychology: The last 25 years.
van de Schoot, R., Winter, S. D., Ryan, O., Zondervan-Zwijnenburg, M. A. J., & Depaoli, S. A. (2017) · Psychological Methods · doi:10.1037/met0000100 · Cited by 401
Questionable research practices in ecology and evolution
Fraser, H., Parker, T., Nakagawa, S., Barnett, A., & Fidler, F. (2018) · PLOS ONE · doi:10.1371/journal.pone.0200303 · Cited by 346
Good practices for real‐world data studies of treatment and/or comparative effectiveness: Recommendations from the joint ISPOR‐ISPE Special Task Force on real‐world evidence in health care decision making
Berger, M. L., Sox, H. C., Willke, R. J., Brixner, D. L., Eichler, H., Goettsch, W., Madigan, D., Makady, A., Schneeweiß, S., Tarricone, R., et al. (2017) · Pharmacoepidemiology and Drug Safety · doi:10.1002/pds.4297 · Cited by 355
A weak scientific basis for gaming disorder: Let us err on the side of caution
van Rooij, A. J., Ferguson, C. J., Colder Carras, M., Kardefelt-Winther, D., Shi, J., Aarseth, E., Bean, A. M., Bergmark, K. H., Brus, A., Coulson, M., et al. (2018) · Journal of Behavioral Addictions · doi:10.1556/2006.7.2018.19 · Cited by 346
A Bayesian Perspective on the Reproducibility Project: Psychology
Etz, A., & Vandekerckhove, J. (2016) · PLOS ONE · doi:10.1371/journal.pone.0149794 · Cited by 295
“[34] for more detail and see Matzke et al.”
Preregistration Is Hard, And Worthwhile
Nosek, B. A., Beck, E. D., Campbell, L., Flake, J. K., Hardwicke, T. E., Mellor, D. T., Veer, A. V. '., & Vazire, S. (2019) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2019.07.009 · Cited by 326
The Sisyphean Cycle of Technology Panics
Orben, A. (2020) · Perspectives on Psychological Science · doi:10.1177/1745691620919372 · Cited by 316
Replication in Social Science
Jeremy Freese, A., & Peterson, D. (2017) · Annual Review of Sociology · doi:10.1146/annurev-soc-060116-053450 · Cited by 341
Challenges for assessing replicability in preclinical cancer biology
Errington, T. M., Denis, A., Perfito, N., Iorns, E., & Nosek, B. A. (2021) · eLife · doi:10.7554/elife.67995 · Cited by 345
Linear mixed-effects models for within-participant psychology experiments: an introductory tutorial and free, graphical user interface (LMMgui)
Magezi, D. A. (2015) · Frontiers in Psychology · doi:10.3389/fpsyg.2015.00002 · Cited by 314
Bayesian tests to quantify the result of a replication attempt.
Verhagen, J., & Wagenmakers, E. (2014) · Journal of Experimental Psychology General · doi:10.1037/a0036731 · Cited by 287
Self-citation
Evaluating Recommender Systems: Survey and Framework
Zangerle, E., & Bauer, C. (2022) · ACM Computing Surveys · doi:10.1145/3556536 · Cited by 362
Qualitative Research Methods
Kazdin, A. E. (2021) · Cambridge University Press eBooks · doi:10.1017/9781108993647.010 · Cited by 279
Bayesian benefits with JASP
Marsman, M., & Wagenmakers, E.-J. (2017) · European Journal of Developmental Psychology · doi:10.1080/17405629.2016.1259614 · Cited by 267
Self-citation
Sailing From the Seas of Chaos Into the Corridor of Stability: Practical Recommendations to Increase the Informational Value of Studies
Lakens, D., & Evers, E. R. K. (2014) · Perspectives on Psychological Science · doi:10.1177/1745691614528520 · Cited by 266
Statistical Rituals: The Replication Delusion and How We Got There
Gigerenzer, G. (2018) · Advances in Methods and Practices in Psychological Science · doi:10.1177/2515245918771329 · Cited by 263
An open investigation of the reproducibility of cancer biology research
Errington, T. M., Iorns, E., Gunn, W., Tan, F. E., Lomax, J., & Nosek, B. A. (2014) · eLife · doi:10.7554/elife.04333 · Cited by 312
Priming Intelligent Behavior: An Elusive Phenomenon
Shanks, D. R., Newell, B. R., Lee, E. H., Balakrishnan, D., Ekelund, L., Cenac, Z., Kavvadia, F., & Moore, C. (2013) · PLoS ONE · doi:10.1371/journal.pone.0056515 · Cited by 263
Research Design in Clinical Psychology
Kazdin, A. E. (2021) · doi:10.1017/9781108993647 · Cited by 240
Exploratory data analysis as a foundation of inductive research
Jebb, A. T., Parrigon, S., & Woo, S. E. (2016) · Human Resource Management Review · doi:10.1016/j.hrmr.2016.08.003 · Cited by 300
A tutorial on testing hypotheses using the Bayes factor.
Hoijtink, H., Mulder, J., Lissa, C. J. V., & Gu, X. (2019) · Psychological Methods · doi:10.1037/met0000201 · Cited by 256
How Many Participants Do I Need to Test an Interaction? Conducting an Appropriate Power Analysis and Achieving Sufficient Power to Detect an Interaction
Sommet, N., Weissman, D. L., Cheutin, N., & Elliot, A. J. (2023) · Advances in Methods and Practices in Psychological Science · doi:10.1177/25152459231178728 · Cited by 313
Distinguishing between exploratory and confirmatory preclinical research will improve translation
Kimmelman, J., Mogil, J. S., & Dirnagl, U. (2014) · PLoS Biology · doi:10.1371/journal.pbio.1001863 · Cited by 236
A Practical Guide for Transparency in Psychological Science
Klein, O., Hardwicke, T. E., Aust, F., Breuer, J., Danielsson, H., Mohr, A. H., IJzerman, H., Nilsonne, G., Vanpaemel, W., & Frank, M. C. (2018) · Collabra Psychology · doi:10.1525/collabra.158 · Cited by 251
The P600-as-P3 hypothesis revisited: Single-trial analyses reveal that the late EEG positivity following linguistically deviant material is reaction time aligned
Sassenhagen, J., Schlesewsky, M., & Bornkessel‐Schlesewsky, I. (2014) · Brain and Language · doi:10.1016/j.bandl.2014.07.010 · Cited by 233
The three ghosts of medical AI: Can the black-box present deliver?
Quinn, T. P., Jacobs, S., Senadeera, M., Le, V., & Coghlan, S. (2021) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2021.102158 · Cited by 246
Beyond differences in means: robust graphical methods to compare two groups in neuroscience
Rousselet, G. A., Pernet, C. R., & Wilcox, R. R. (2017) · European Journal of Neuroscience · doi:10.1111/ejn.13610 · Cited by 225
A Short (Personal) Future History of Revolution 2.0
Spellman, B. A. (2015) · Perspectives on Psychological Science · doi:10.1177/1745691615609918 · Cited by 213
Modeling psychopathology: From data models to formal theories.
Haslbeck, J. M. B., Ryan, O., Robinaugh, D. J., Waldorp, L., & Borsboom, D. (2021) · Psychological Methods · doi:10.1037/met0000303 · Cited by 253
Self-citation
Do p Values Lose Their Meaning in Exploratory Analyses? It Depends How You Define the Familywise Error Rate
Rubin, M. (2017) · Review of General Psychology · doi:10.1037/gpr0000123 · Cited by 226
Recommendations for Increasing the Transparency of Analysis of Preexisting Data Sets
Weston, S. J., Ritchie, S. J., Rohrer, J. M., & Przybylski, A. K. (2019) · Advances in Methods and Practices in Psychological Science · doi:10.1177/2515245919848684 · Cited by 233
Best research practices in psychology: Illustrating epistemological and pragmatic considerations with the case of relationship science.
Finkel, E. J., Eastwick, P. W., & Reis, H. T. (2015) · Journal of Personality and Social Psychology · doi:10.1037/pspi0000007 · Cited by 208
Transparency in Ecology and Evolution: Real Problems, Real Solutions
Parker, T. H., Forstmeier, W., Koricheva, J., Fidler, F., Hadfield, J. D., Chee, Y. E., Kelly, C. D., Gurevitch, J., & Nakagawa, S. (2016) · Trends in Ecology & Evolution · doi:10.1016/j.tree.2016.07.002 · Cited by 201
Safeguard Power as a Protection Against Imprecise Power Estimates
Perugini, M., Gallucci, M., & Costantini, G. (2014) · Perspectives on Psychological Science · doi:10.1177/1745691614528519 · Cited by 203
The case for formal methodology in scientific reform
Devezer, B., Navarro, D., Vandekerckhove, J., & Buzbas, E. O. (2021) · Royal Society Open Science · doi:10.1098/rsos.200805 · Cited by 204
An empirical assessment of transparency and reproducibility-related research practices in the social sciences (2014–2017)
Hardwicke, T. E., Wallach, J. D., Kidwell, M. C., Bendixen, T., Crüwell, S., & Ioannidis, J. P. A. (2020) · Royal Society Open Science · doi:10.1098/rsos.190806 · Cited by 217
Meta-Analysis of Menstrual Cycle Effects on Women’s Mate Preferences
Wood, W., Kressel, L. M., Joshi, P. D., & Louie, B. E. (2014) · Emotion Review · doi:10.1177/1754073914523073 · Cited by 199
The meaning of “significance” for different types of research [translated and annotated by Eric-Jan Wagenmakers, Denny Borsboom, Josine Verhagen, Rogier Kievit, Marjan Bakker, Angelique Cramer, Dora Matzke, Don Mellenbergh, and Han L. J. van der Maas]
de Groot, A. D. (2014) · Acta Psychologica · doi:10.1016/j.actpsy.2014.02.001 · Cited by 165
Estimating the Prevalence of Transparency and Reproducibility-Related Research Practices in Psychology (2014–2017)
Hardwicke, T. E., Thibault, R. T., Kosie, J. E., Wallach, J. D., Kidwell, M. C., & Ioannidis, J. P. A. (2022) · Perspectives on Psychological Science · doi:10.1177/1745691620979806 · Cited by 187
The academic, economic and societal impacts of Open Access: an evidence-based review
Tennant, J., Waldner, F., Jacques, D., Masuzzo, P., Collister, L., & Hartgerink, C. (2016) · F1000Research · doi:10.12688/f1000research.8460.1 · Cited by 592
When Does HARKing Hurt? Identifying When Different Types of Undisclosed Post Hoc Hypothesizing Harm Scientific Progress
Rubin, M. (2017) · Review of General Psychology · doi:10.1037/gpr0000128 · Cited by 209
Reducing bias, increasing transparency and calibrating confidence with preregistration
Hardwicke, T. E., & Wagenmakers, E. (2023) · Nature Human Behaviour · doi:10.1038/s41562-022-01497-2 · Cited by 205
Self-citation
Protecting against researcher bias in secondary data analysis: challenges and potential solutions
Baldwin, J. R., Pingault, J.-B., Schoeler, T., Sallis, H. M., & Munafò, M. R. (2022) · European Journal of Epidemiology · doi:10.1007/s10654-021-00839-0 · Cited by 178
“I Know Things They Don’t Know!”: The Role of Need for Uniqueness in Belief in Conspiracy Theories
Lantian, A., Muller, D., Nurra, C., & Douglas, K. M. (2017) · Social Psychology · doi:10.1027/1864-9335/a000306 · Cited by 176
Scientific Utopia III: Crowdsourcing Science
Uhlmann, E. L., Ebersole, C. R., Chartier, C. R., Errington, T. M., Kidwell, M. C., Lai, C. K., McCarthy, R. J., Riegelman, A., Silberzahn, R., & Nosek, B. A. (2019) · Perspectives on Psychological Science · doi:10.1177/1745691619850561 · Cited by 191
Cochrane’s risk of bias tool in the context of psychotherapy outcome research
Munder, T., & Barth, J. (2018) · Psychotherapy Research · doi:10.1080/10503307.2017.1411628 · Cited by 174
A purely confirmatory replication study of structural brain-behavior correlations
Boekel, W., Wagenmakers, E., Belay, L., Verhagen, J., Brown, S. D., & Forstmann, B. U. (2015) · Cortex · doi:10.1016/j.cortex.2014.11.019 · Cited by 175
Self-citation
Contrast coding choices in a decade of mixed models
Brehm, L. E., & Alday, P. M. (2022) · Journal of Memory and Language · doi:10.1016/j.jml.2022.104334 · Cited by 171
Crowdsourcing hypothesis tests: Making transparent how design choices shape research results.
Landy, J. F., Jia, M. (., Ding, I. L., Viganola, D., Tierney, W., Dreber, A., Johannesson, M., Pfeiffer, T., Ebersole, C. R., Gronau, Q. F., et al. (2020) · Psychological Bulletin · doi:10.1037/bul0000220 · Cited by 168
Self-citation
The Preregistration RevolutionPreprint
Nosek, B. A., Ebersole, C. R., DeHaven, A. C., & Mellor, D. T. (2017) · OSF Preprints (OSF Preprints) · doi:10.31219/osf.io/2dxu5 · Cited by 220
There’s more than one way to conduct a replication study: Beyond statistical significance.
Anderson, S. F., & Maxwell, S. E. (2016) · Psychological Methods · doi:10.1037/met0000051 · Cited by 167
Questions About Questionable Research Practices in the Field of Management: A Guest Commentary
Banks, G. C., O’Boyle, E. H., Pollack, J. M., White, C. D., Batchelor, J. H., Whelpley, C. E., Abston, K. A., Bennett, A. A., & Adkins, C. L. (2016) · Journal of Management · doi:10.1177/0149206315619011 · Cited by 163
Questionable research practices among italian research psychologists
Agnoli, F., Wicherts, J. M., Veldkamp, C. L. S., Albiero, P., & Cubelli, R. (2017) · PLoS ONE · doi:10.1371/journal.pone.0172792 · Cited by 173
Preliminary evidence of linguistic bias in academic reviewing
Politzer‐Ahles, S., Girolamo, T. M., & Ghali, S. (2020) · Journal of English for Academic Purposes · doi:10.1016/j.jeap.2020.100895 · Cited by 183
Replicability and Reproducibility in Comparative Psychology
Stevens, J. R. (2017) · Frontiers in Psychology · doi:10.3389/fpsyg.2017.00862 · Cited by 169
Machine Learning and Psychological Research: The Unexplored Effect of Measurement
Jacobucci, R., & Grimm, K. J. (2020) · Perspectives on Psychological Science · doi:10.1177/1745691620902467 · Cited by 175
Weight stigma and health behaviors: evidence from the Eating in America Study
Lee, K. M., Hunger, J. M., & Tomiyama, A. J. (2021) · International Journal of Obesity · doi:10.1038/s41366-021-00814-5 · Cited by 178
Seven Easy Steps to Open Science
Crüwell, S., van Doorn, J. B., Etz, A. J., Makel, M. C., Moshontz, H., Niebaum, J. C., Orben, A. C., Parsons, S. D., & Schulte‐Mecklenbeck, M. (2019) · Zeitschrift für Psychologie · doi:10.1027/2151-2604/a000387 · Cited by 186

Authors: Wagenmakers, E.-J., Wetzels, R., Borsboom, D., van der Maas, H. L. J., & Kievit, R. A.
Affiliations: University of Amsterdam (NL)
ORCID: Wagenmakers, Eric-Jan
Borsboom, Denny
van der Maas, Han L. J.
Kievit, Rogier A.
Venue: Perspectives on Psychological Science
Publisher: SAGE Publications
Published: 2012-11
Volume: 7
Issue: 6
Pages: 632-638
ISSN: 1745-6916 (ISSN-L)
1745-6924
DOI: 10.1177/1745691612463078
OpenAlex: W2123263696
Cited by: 1,042
References: 48
Citations by year: 2026: 57 · 2025: 41 · 2024: 65 · 2023: 85 · 2022: 102 · 2021: 108 · 2020: 114 · 2019: 98 · 2018: 88 · 2017: 86 · 2016: 75 · 2015: 51 · 2014: 43 · 2013: 28 · 2012: 1
FWCI: 44.75
Type: journal article
Language: en
Open access: closed
Licence: journals.sagepub.com/page/policies/text-and-data-mining-license (tdm)
Locations: Perspectives on Psychological Science (publishedVersion)
PubMed (publishedVersion)
UvA-DARE (University of Amsterdam) (publishedVersion)
UvA-DARE (University of Amsterdam) (submittedVersion)
Data Archiving and Networked Services (DANS) (submittedVersion)
Topics: Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
Social Sciences › Psychology › Experimental and Cognitive Psychology › Psychology Research and Bibliometrics
Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods and Bayesian Inference
Keywords: preregistration, confirmatory analysis, exploratory analysis, data snooping, statistical validity, research integrity, extrasensory perception, replication, hypothesis testing

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

An Agenda for Purely Confirmatory Research

Abstract

Overview

Integrity signals

Verification

Claims the paper makes

Claims other papers make about this one

Citations

Cites

Cited by

Related

Metadata

Record

About these checks