Cited by 800Meta-analysis and systematic reviewsPsychology Research and BibliometricsAcademic integrity and plagiarism

The DOI matches its Crossref record; 75 of the 75 references it cites were found in a registry or an index.

The Rules of the Game Called Psychological Science

Open access

Bakker, Marjan, van Dijk, Annette, Wicherts, Jelte M.·2012·Perspectives on Psychological Science

Open this paper in CoChat — free account Open PDF

Abstract

If science were a game, a dominant rule would probably be to collect results that are statistically significant. Several reviews of the psychological literature have shown that around 96% of papers involving the use of null hypothesis significance testing report significant outcomes for their main results but that the typical studies are insufficiently powerful for such a track record. We explain this paradox by showing that the use of several small underpowered samples often represents a more efficient research strategy (in terms of finding p < .05) than does the use of one larger (more powerful) sample. Publication bias and the most efficient strategy lead to inflated effects and high rates of false positives, especially when researchers also resorted to questionable research practices, such as adding participants after intermediate testing. We provide simulations that highlight the severity of such biases in meta-analyses. We consider 13 meta-analyses covering 281 primary studies in various fields of psychology and find indications of biases and/or an excess of significant results in seven. These results highlight the need for sufficiently powerful replications and changes in journal policies.

Integrity signals

Registry

Crossref · 10.1177/1745691612459060

Cites

75 of 75 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Claims the paper makes

We couldn’t retrieve this paper’s full text, so its claims and self-check weren’t run. The registry and citation checks still apply.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Two new Later Stone Age sites from the Final Pleistocene in the Falémé Valley, eastern Senegal
Ndiaye, M., Lespez, L., Tribolo, C., Rassé, M., Hadjas, I., Davidoux, S., Huysecom, É., & Douze, K. (2024) · Monash University Research Portal (Monash University) · doi:10.3929/ethz-b-000667478 · Cited by 24,214
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · PLoS Medicine · doi:10.1371/journal.pmed.0020124 · Cited by 10,505
The file drawer problem and tolerance for null results.
Rosenthal, R. (1979) · Psychological Bulletin · doi:10.1037/0033-2909.86.3.638 · Cited by 9,068
False-Positive Psychology: Undisclosed Flexibility in Data Collection and Analysis Allows Presenting Anything as Significant
Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011) · Psychological Science · doi:10.1177/0956797611417632 · Cited by 6,567
Journal of The American Statistical Association
Fienberg, S. E. (2004) · Encyclopedia of Statistical Sciences · doi:10.1002/0471667196.ess1317 · Cited by 6,289
Understanding and using the Implicit Association Test: I. An improved scoring algorithm.
Greenwald, A. G., Nosek, B. A., & Banaji, M. R. (2003) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.85.2.197 · Cited by 5,815
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · CHANCE · doi:10.1080/09332480.2005.10722754 · Cited by 4,702 · cited 3×
“Such practices lead to inflated ESs and increased false positive rates (Ioannidis, 2005, 2008b; Simmons et al., 2011).”
BackgroundResult
The earth is round (p < .05).
Cohen, J. (1994) · American Psychologist · doi:10.1037/0003-066x.49.12.997 · Cited by 4,003
Understanding and using the Implicit Association Test: III. Meta-analysis of predictive validity.
Greenwald, A. G., Poehlman, T. A., Uhlmann, E. L., & Banaji, M. R. (2009) · Journal of Personality and Social Psychology · doi:10.1037/a0015575 · Cited by 3,257
Belief in the law of small numbers.
Tversky, A., & Kahneman, D. (1971) · Psychological Bulletin · doi:10.1037/h0031322 · Cited by 3,246

Flagged first, then by citations

Show all 75 references

Research works printed but not linked: 1

10.1037/0003-066X.56.2.128 · DOI · Paper page

Sorted by citations

Items 101–200 of 799

Addressing the “Replication Crisis”: Using Original Studies to Design Replication Studies with Appropriate Statistical Power
Anderson, S. F., & Maxwell, S. (2017) · Multivariate Behavioral Research · doi:10.1080/00273171.2017.1289361 · Cited by 116
Research Practices That Can Prevent an Inflation of False-Positive Rates
Murayama, K., Pekrun, R., & Fiedler, K. (2014) · Personality and Social Psychology Review · doi:10.1177/1088868313496330 · Cited by 124
The frequency of excess success for articles in Psychological Science
Francis, G. (2014) · Psychonomic Bulletin & Review · doi:10.3758/s13423-014-0601-x · Cited by 130
Quality Uncertainty Erodes Trust in Science
Vazire, S. (2017) · Collabra: Psychology · doi:10.1525/collabra.74 · Cited by 106
P-hacking and publication bias interact to distort meta-analytic effect size estimates
Friese, M., & Frankenbach, J. (2020) · Psychological Methods · doi:10.1037/met0000246 · Cited by 122
Do ‘early’ brain responses reveal word form prediction during language comprehension? A critical review
Nieuwland, M. S. (2019) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2018.11.019 · Cited by 123
Replicating Studies in Which Samples of Participants Respond to Samples of Stimuli
Westfall, J., Judd, C. M., & Kenny, D. A. (2014) · Perspectives on Psychological Science · doi:10.1177/1745691614564879 · Cited by 133
Big little lies: a compendium and simulation of p -hacking strategies
Stefan, A. M., & Schönbrodt, F. D. (2023) · Royal Society Open Science · doi:10.1098/rsos.220346 · Cited by 177
Common methodological mistakes
Wulff, J. N., Sajons, G. B., Pogrebna, G., Lonati, S., Bastardoz, N., Banks, G. C., & Antonakis, J. (2023) · The Leadership Quarterly · doi:10.1016/j.leaqua.2023.101677 · Cited by 136
Replicability Crisis in Social Psychology: Looking at the Past to Find New Pathways for the Future
Świątkowski, W., & Dompnier, B. (2017) · International Review of Social Psychology · doi:10.5334/irsp.66 · Cited by 132
The effect of horizontal eye movements on free recall: a preregistered adversarial collaboration.
Matzke, D., Nieuwenhuis, S., van Rijn, H., Slagter, H., van der Molen, M. W., & Wagenmakers, E. (2015) · Journal of experimental psychology. General · doi:10.1037/xge0000038 · Cited by 148
Quantifying Support for the Null Hypothesis in Psychology: An Empirical Investigation
Aczél, B., Pálfi, B., Szollosi, A., Kovács, M., Szászi, B., Szécsi, P., Zrubka, M., Gronau, Q. F., van den Bergh, D., & Wagenmakers, E. (2018) · Advances in Methods and Practices in Psychological Science · doi:10.1177/2515245918773742 · Cited by 122
Reputation management: Why and how gossip enhances generosity
Wu, J., Balliet, D. P., & Lange, P. A. M. V. (2015) · Evolution and Human Behavior · doi:10.1016/j.evolhumbehav.2015.11.001 · Cited by 115
Reinforcement sensitivity, depression and anxiety: A meta-analysis and meta-analytic structural equation model
Katz, B. A., Matanky, K., Aviram, G., & Yovel, I. (2020) · Clinical Psychology Review · doi:10.1016/j.cpr.2020.101842 · Cited by 109
Why Publishing Everything Is More Effective than Selective Publishing of Statistically Significant Results
van Assen, M. A. L. M., van Aert, R. C. M., Nuijten, M. B., & Wicherts, J. M. (2014) · PLoS ONE · doi:10.1371/journal.pone.0084896 · Cited by 113
The Bitter Truth About Sugar and Willpower
Vadillo, M. A., Gold, N., & Osman, M. (2016) · Psychological Science · doi:10.1177/0956797616654911 · Cited by 107
Correcting for Publication Bias in a Meta-Analysis with the P-uniform* MethodPreprint
van Aert, R. C. M., & van Assen, M. A. L. M. (2018) · doi:10.31222/osf.io/zqjr9 · Cited by 108
Is the SNARC Effect Related to the Level of Mathematics? No Systematic Relationship Observed despite More Power, More Repetitions, and More Direct Assessment of Arithmetic Skill
Cipora, K., & Nuerk, H. (2013) · Quarterly Journal of Experimental Psychology · doi:10.1080/17470218.2013.772215 · Cited by 106
A tragedy of the (academic) commons: interpreting the replication crisis in psychology as a social dilemma for early-career researchers
Everett, J. A. C., & Earp, B. D. (2015) · Frontiers in Psychology · doi:10.3389/fpsyg.2015.01152 · Cited by 105
Calibrating the Scientific Ecosystem Through Meta-Research
Hardwicke, T. E., Serghiou, S., Janiaud, P., Danchev, V., Crüwell, S., Goodman, S. N., & Ioannidis, J. P. A. (2020) · Annual Review of Statistics and Its Application · doi:10.1146/annurev-statistics-031219-041104 · Cited by 98
ON NORMATIVITY AND EPISTEMIC INTUITIONS: FAILURE OF REPLICATION
Seyedsayamdost, H. (2014) · Episteme · doi:10.1017/epi.2014.27 · Cited by 102
Outlier Removal and the Relation with Reporting Errors and Quality of Psychological Research
Bakker, M., & Wicherts, J. M. (2014) · PLoS ONE · doi:10.1371/journal.pone.0103360 · Cited by 101
Self-citation
The crisis of confidence in research findings in psychology: Is lack of replication the real problem? Or is it something else?
Schmidt, F. L., & Oh, I.-S. (2016) · Archives of Scientific Psychology · doi:10.1037/arc0000029 · Cited by 145
A Call for Examining Replication and Bias in Special Education Research
Cook, B. G. (2014) · Remedial and Special Education · doi:10.1177/0741932514528995 · Cited by 100
On making the right choice: A meta-analysis and large-scale replication attempt of the unconscious thought advantage
Nieuwenstein, M. R., Wierenga, T., Morey, R. D., Wicherts, J. M., Blom, T. N., Wagenmakers, E., & van Rijn, H. (2015) · Judgment and Decision Making · doi:10.1017/s1930297500003144 · Cited by 105
The propagation of self-control: Self-control in one domain simultaneously improves self-control in other domains.
Tuk, M. A., Zhang, K., & Sweldens, S. (2015) · Journal of Experimental Psychology General · doi:10.1037/xge0000065 · Cited by 96
An Aberrant Abundance of Cronbach’s Alpha Values at .70
Hussey, I., Alsalti, T., Bosco, F. A., Elson, M., & Arslan, R. C. (2025) · Advances in Methods and Practices in Psychological Science · doi:10.1177/25152459241287123 · Cited by 197
To Remove or not to Remove: the Impact of Outlier Handling on Significance Testing in Testosterone Data
Pollet, T., & van der Meij, L. (2016) · Adaptive Human Behavior and Physiology · doi:10.1007/s40750-016-0050-z · Cited by 94
The influence of gender stereotype threat on mathematics test scores of Dutch high school students: a registered report
Flore, P. C., Mulder, J., & Wicherts, J. M. (2018) · Comprehensive Results in Social Psychology · doi:10.1080/23743603.2018.1559647 · Cited by 105
Replication concerns in sports and exercise science: a narrative review of selected methodological issues in the field
Mesquida, C., Murphy, J., Lakens, D., & Warne, J. P. (2022) · Royal Society Open Science · doi:10.1098/rsos.220946 · Cited by 110
The application of Goal Setting Theory to goal setting interventions in sport: a systematic review
Jeong, Y. H., Healy, L. C., & McEwan, D. (2023) · International Review of Sport and Exercise Psychology · doi:10.1080/1750984x.2021.1901298 · Cited by 113
Improving transparency and reproducibility through registration: The status of intervention trials published in clinical psychology journals.
Cybulski, Ł., Mayo‐Wilson, E., & Grant, S. (2016) · Journal of Consulting and Clinical Psychology · doi:10.1037/ccp0000115 · Cited by 94
Addressing Methodological Challenges in Culture-Comparative Research
Fischer, R. D., & Poortinga, Y. H. (2018) · Journal of Cross-Cultural Psychology · doi:10.1177/0022022117738086 · Cited by 92
Sex differences in trust and trustworthiness: A meta-analysis of the trust game and the gift-exchange game
van den Akker, O. R., van Assen, M. A. L. M., van Vugt, M., & Wicherts, J. M. (2020) · Journal of Economic Psychology · doi:10.1016/j.joep.2020.102329 · Cited by 90
Two sides of the same coin: The differentiating role of need satisfaction and frustration in passion for screen‐based activities
Tóth‐Király, I., Bőthe, B., Márki, A. N., Rigó, A., & Orosz, G. (2019) · European Journal of Social Psychology · doi:10.1002/ejsp.2588 · Cited by 84
On gender and philosophical intuition: Failure of replication and other negative results
Seyedsayamdost, H. (2015) · Philosophical Psychology · doi:10.1080/09515089.2014.893288 · Cited by 52
To Replicate or Not to Replicate? Exploring Reproducibility in Economics through the Lens of a Model and a Pilot Study
Maniadis, Z., Tufano, F., & List, J. A. (2017) · The Economic Journal · doi:10.1111/ecoj.12527 · Cited by 87
The role of componential analysis, categorical hypothesising, replicability and confirmation bias in testing for bilingual advantages in executive functioning
Paap, K. R. (2014) · Journal of Cognitive Psychology · doi:10.1080/20445911.2014.891597 · Cited by 85
The Voluntary Coffee Standard Index (VOCSI). Developing a Composite Index to Assess and Compare the Strength of Mainstream Voluntary Sustainability Standards in the Global Coffee Industry
Dietz, T., Auffenberg, J., Chong, A. E., Grabs, J., & Kilian, B. (2018) · Ecological Economics · doi:10.1016/j.ecolecon.2018.03.026 · Cited by 90
Reproducibility of individual effect sizes in meta-analyses in psychology
Maassen, E., van Assen, M. A. L. M., Nuijten, M. B., Olsson-Collentine, A., & Wicherts, J. M. (2020) · PLoS ONE · doi:10.1371/journal.pone.0233107 · Cited by 90
Research Design in Clinical Psychology
Kazdin, A. E. (2021) · doi:10.1017/9781108993647 · Cited by 240
The impact of sample size on the reproducibility of voxel-based lesion-deficit mappings
Lorca‐Puls, D. L., Gajardo‐Vidal, A., White, J., Seghier, M. L., Leff, A. P., Green, D. W., Crinion, J., Ludersdorfer, P., Hope, T. M. H., Bowman, H., et al. (2018) · Neuropsychologia · doi:10.1016/j.neuropsychologia.2018.03.014 · Cited by 86
A comprehensive meta-analysis of money priming.
Lodder, P., Ong, H. H., Grasman, R. P. P. P., & Wicherts, J. M. (2019) · Journal of Experimental Psychology General · doi:10.1037/xge0000570 · Cited by 86
The effect of working memory maintenance on long-term memory
Hartshorne, J. K., & Makovski, T. (2019) · Memory & Cognition · doi:10.3758/s13421-019-00908-6 · Cited by 81
A power fallacy
Wagenmakers, E., Verhagen, J., Ly, A., Bakker, M., Lee, M. D., Matzke, D., Rouder, J. N., & Morey, R. D. (2014) · Behavior Research Methods · doi:10.3758/s13428-014-0517-4 · Cited by 83
Is ego depletion associated with increased distractibility? Results from a basketball free throw task
Englert, C., Bertrams, A., Furley, P., & Oudejans, R. R. D. (2014) · Psychology of sport and exercise · doi:10.1016/j.psychsport.2014.12.001 · Cited by 89
Interpretations and methods: Towards a more effectively self-correcting social psychology
Jussim, L. J., Crawford, J. T., Anglin, S. M., Stevens, S. T., & Duarte, J. L. (2016) · Journal of Experimental Social Psychology · doi:10.1016/j.jesp.2015.10.003 · Cited by 84
Playing with Data—Or How to Discourage Questionable Research Practices and Stimulate Researchers to Do Things Right
Sijtsma, K. (2015) · Psychometrika · doi:10.1007/s11336-015-9446-0 · Cited by 88
What Makes for a Good Theory? How to Evaluate a Theory Using the Strength Model of Self-Control as an Example
Gieseler, K., Loschelder, D. D., & Friese, M. (2019) · Springer eBooks · doi:10.1007/978-3-030-13788-5_1 · Cited by 80
Development of the first-mention bias
Hartshorne, J. K., Nappa, R., & Snedeker, J. (2014) · Journal of Child Language · doi:10.1017/s0305000914000075 · Cited by 78
A Practical Guide to Variable Selection in Structural Equation Modeling by Using Regularized Multiple-Indicators, Multiple-Causes Models
Jacobucci, R., Brandmaier, A. M., & Kievit, R. (2019) · Advances in Methods and Practices in Psychological Science · doi:10.1177/2515245919826527 · Cited by 78
Exposure to thin-ideal media affect most, but not all, women: Results from the Perceived Effects of Media Exposure Scale and open-ended responses
Frederick, D. A., Daniels, E. A., Bates, M. E., & Tylka, T. L. (2017) · Body Image · doi:10.1016/j.bodyim.2017.10.006 · Cited by 87
A systematic review and meta-analysis of the evidence for unaware fear conditioning
Mertens, G., & Engelhard, I. M. (2019) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2019.11.012 · Cited by 81
Calibrate your confidence in research findings: A tutorial on improving research methods and practices
Frost, A. D. S., & Ledgerwood, A. (2020) · Journal of Pacific Rim Psychology · doi:10.1017/prp.2020.7 · Cited by 75
Comparing supervised and unsupervised approaches to emotion categorization in the human brain, body, and subjective experience
Azari, B., Westlin, C., Satpute, A. B., Hutchinson, J. B., Kragel, P. A., Hoemann, K., Khan, Z., Wormwood, J. B., Quigley, K. S., Deniz Erdogmus, B., et al. (2020) · Scientific Reports · doi:10.1038/s41598-020-77117-8 · Cited by 83
Many analysts, one dataset: Making transparent how variations in analytical choices affect resultsPreprint
Silberzahn, R., Uhlmann, E. L., Martin, D. P., Anselmi, P., Aust, F., Awtrey, E. C., Bahník, Š., Bai, F., Bannard, C., Bonnier, E., et al. (2017) · doi:10.31234/osf.io/qkwst · Cited by 78
Heterogeneity in effect size estimates
Holzmeister, F., Johannesson, M., BOHM, R. M., Dreber, A., Huber, J., & Kirchler, M. (2024) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2403490121 · Cited by 89
Peer Influences on Internalizing and Externalizing Problems among Adolescents: A Longitudinal Social Network Analysis
Fortuin, J., van Geel, M., & Vedder, P. H. (2014) · Journal of Youth and Adolescence · doi:10.1007/s10964-014-0168-x · Cited by 81
Memory Detection 2.0: The First Web-Based Memory Detection Test
Kleinberg, B., & Verschuère, B. (2015) · PLoS ONE · doi:10.1371/journal.pone.0118715 · Cited by 75
Questionable, Objectionable or Criminal? Public Opinion on Data Fraud and Selective Reporting in Science
Pickett, J. T., & Roche, S. P. (2016) · Science and Engineering Ethics · doi:10.1007/s11948-017-9886-2 · Cited by 77
Assessing co-creation in strategic planning for urban energy transitions
Sillak, S., Borch, K., & Sperling, K. (2021) · Energy Research & Social Science · doi:10.1016/j.erss.2021.101952 · Cited by 71
Open Science Is Liberating and Can Foster Creativity
Frankenhuis, W. E., & Nettle, D. (2018) · Perspectives on Psychological Science · doi:10.1177/1745691618767878 · Cited by 74
The public's perception of run-of-the-river hydropower across Europe
Venus, T. E., Hinzmann, M., Bakken, T. H., Gerdes, H., Godinho, F. N., Hansen, B., Pinheiro, A. N., & Sauer, J. (2020) · Energy Policy · doi:10.1016/j.enpol.2020.111422 · Cited by 71
Connecting the dots: Mobilizing theory to reveal the big picture in social psychology (and why we should do this)
Ellemers, N. (2013) · European Journal of Social Psychology · doi:10.1002/ejsp.1932 · Cited by 69
Optimizing sampling approaches along ecological gradients
Schweiger, A., Irl, S. D. H., Steinbauer, M. J., Dengler, J., & Beierkuhnlein, C. (2015) · Methods in Ecology and Evolution · doi:10.1111/2041-210x.12495 · Cited by 78
The Replication Paradox: Combining Studies can Decrease Accuracy of Effect Size Estimates
Nuijten, M. B., van Assen, M. A. L. M., Veldkamp, C. L. S., & Wicherts, J. M. (2015) · Review of General Psychology · doi:10.1037/gpr0000034 · Cited by 73
The quest for an optimal alpha
Miller, J. O., & Ulrich, R. (2019) · PLoS ONE · doi:10.1371/journal.pone.0208631 · Cited by 84
The replication crisis: How might philosophy and theory of psychology be of use?
Morawski, J. G. (2019) · Journal of Theoretical and Philosophical Psychology · doi:10.1037/teo0000129 · Cited by 78
Resting frontal EEG asymmetry in children: Meta‐analyses of the effects of psychosocial risk factors and associations with internalizing and externalizing behavior
Peltola, M. J., Bakermans‐Kranenburg, M. J., Alink, L. R. A., Huffmeijer, R., Bı́ró, S., & van IJzendoorn, M. H. (2014) · Developmental Psychobiology · doi:10.1002/dev.21223 · Cited by 68
Replication and Reproducibility in Cross-Cultural Psychology
Milfont, T. L., & Klein, R. A. (2018) · Journal of Cross-Cultural Psychology · doi:10.1177/0022022117744892 · Cited by 69
Intergenerational conflicts of interest and prosocial behavior during the COVID-19 pandemic
Jin, S., Balliet, D. P., Romano, A., Spadaro, G., Lissa, C. J. V., Agostini, M., Bélanger, J. J., Gützkow, B., Kreienkamp, J., Abakoumkin, G., et al. (2020) · Personality and Individual Differences · doi:10.1016/j.paid.2020.110535 · Cited by 66
The significance fallacy in inferential statistics
Kühberger, A., Fritz, A., Lermer, E., & Scherndl, T. (2015) · BMC Research Notes · doi:10.1186/s13104-015-1020-4 · Cited by 67
Questionable Research Practices in Experimental Communication Research: A Systematic Analysis From 1980 to 2013
Matthes, J., Marquart, F., Naderer, B., Arendt, F., Schmuck, D., & Adam, K. (2015) · Communication Methods and Measures · doi:10.1080/19312458.2015.1096334 · Cited by 74
Romance, risk, and replication: Can consumer choices and risk-taking be primed by mating motives?
Shanks, D. R., Vadillo, M. A., Riedel, B., Clymo, A., Govind, S., Hickin, N., Tamman, A. J. F., & Puhlmann, L. M. C. (2015) · Journal of Experimental Psychology General · doi:10.1037/xge0000116 · Cited by 67
The role and value of replication in empirical software engineering results
Shepperd, M., Ajienka, N., & Counsell, S. (2018) · Information and Software Technology · doi:10.1016/j.infsof.2018.01.006 · Cited by 67
Perceived crisis and reforms: Issues, explanations, and remedies.
de Boeck, P. A. L., & Jeon, M. (2018) · Psychological Bulletin · doi:10.1037/bul0000154 · Cited by 71
Redefining Science: The Impact of Complexity on Theory Development in Social and Behavioral Research
Sanbonmatsu, D. M., & Johnston, W. (2019) · Perspectives on Psychological Science · doi:10.1177/1745691619848688 · Cited by 68
Project INTEGRATE: An integrative study of brief alcohol interventions for college students.
Mun, E., The Project INTEGRATE Team, TORRE, J. D. L., Atkins, D. C., WHITE, H. R., Ray, A. E., Kim, S. Y., Jiao, Y., Clarke, N., Huo, Y., et al. (2014) · Psychology of Addictive Behaviors · doi:10.1037/adb0000047 · Cited by 62
Additionality and Implementation Gaps in Voluntary Sustainability Standards
Dietz, T., & Grabs, J. (2021) · New Political Economy · doi:10.1080/13563467.2021.1881473 · Cited by 62
Preregistration in practice: A comparison of preregistered and non-preregistered studies in psychology
van den Akker, O. R., van Assen, M. A. L. M., Bakker, M., Elsherif, M. M., Wong, T. K., & Wicherts, J. M. (2023) · Behavior Research Methods · doi:10.3758/s13428-023-02277-0 · Cited by 84
Benefits of Open and High-Powered Research Outweigh Costs
LeBel, E. P., Campbell, L., & Loving, T. J. (2016) · Journal of Personality and Social Psychology · doi:10.1037/pspi0000049 · Cited by 67
The 40-year debate: a meta-review on what works for juvenile offenders
Pappas, L. N., & Dent, A. L. (2023) · Journal of Experimental Criminology · doi:10.1007/s11292-021-09472-z · Cited by 72
Statistical heartburn: an attempt to digest four pizza publications from the Cornell Food and Brand Lab
van der Zee, T., Anaya, J., & Brown, N. J. L. (2017) · BMC Nutrition · doi:10.1186/s40795-017-0167-x · Cited by 61
The Use and Abuse of Transcranial Magnetic Stimulation to Modulate Corticospinal Excitability in Humans
Héroux, M. E., Taylor, J. L., & Gandevia, S. C. (2015) · PLoS ONE · doi:10.1371/journal.pone.0144151 · Cited by 62
A Whole Other Story
Adelman, L., Leidner, B., Ünal, H., Nahhas, E. A. H., & Shnabel, N. (2016) · Personality and Social Psychology Bulletin · doi:10.1177/0146167216662868 · Cited by 60
How Powerful is the Evidence in Criminology? On Whether We Should Fear a Coming Crisis of Confidence
Barnes, J. C., TenEyck, M. F., Pratt, T. C., & Cullen, F. T. (2019) · Justice Quarterly · doi:10.1080/07418825.2018.1495252 · Cited by 62
Bayesian evaluation of effect size after replicating an original study
van Aert, R. C. M., & van Assen, M. A. L. M. (2017) · PLoS ONE · doi:10.1371/journal.pone.0175302 · Cited by 57
A meta-analysis of the emotional victim effect for female adult rape complainants: Does complainant distress influence credibility?
Nitschke, F., McKimmie, B. M., & Vanman, E. J. (2019) · Psychological Bulletin · doi:10.1037/bul0000206 · Cited by 71
Science as collaborative knowledge generation
Ellemers, N. (2020) · British Journal of Social Psychology · doi:10.1111/bjso.12430 · Cited by 65
Retraction statements and research malpractice in economics
Cox, A., Craig, R., & Tourish, D. (2018) · Research Policy · doi:10.1016/j.respol.2018.02.016 · Cited by 61
A Powerful Nudge? Presenting Calculable Consequences of Underpowered Research Shifts Incentives Toward Adequately Powered Designs
Gervais, W. M., Jewell, J. A., Najle, M. B., & Ng, B. K. L. (2015) · Social Psychological and Personality Science · doi:10.1177/1948550615584199 · Cited by 58
An a priori solution to the replication crisis
Trafimow, D. (2018) · Philosophical Psychology · doi:10.1080/09515089.2018.1490707 · Cited by 56
An overview of issues in infant and developmental research for the creation of robust and replicable science
Davis-Kean, P. E., & Ellis, A. (2019) · Infant Behavior and Development · doi:10.1016/j.infbeh.2019.101339 · Cited by 57
From intrapsychic to ecological theories in social psychology: Outlines of a functional theory approach
Fiedler, K. (2014) · European Journal of Social Psychology · doi:10.1002/ejsp.2069 · Cited by 55
Conflict resolution in sentence processing is the same for bilinguals and monolinguals: The role of confirmation bias in testing for bilingual advantages
Paap, K. R., & Liu, Y. (2013) · Journal of Neurolinguistics · doi:10.1016/j.jneuroling.2013.09.002 · Cited by 55
Social Integration, Social Support and Mortality in the US National Health Interview Survey
Barger, S. D. (2013) · Psychosomatic Medicine · doi:10.1097/psy.0b013e318292ad99 · Cited by 54
Context counts: The different implications of weekday and weekend video gaming for academic performance in mathematics, reading, and science
Hartanto, A., Toh, W. X., & Yang, H. (2018) · Computers & Education · doi:10.1016/j.compedu.2017.12.007 · Cited by 55
A Reappraisal of the Threshold Hypothesis of Creativity and Intelligence
Weiss, S., Steger, D., Schroeders, U., & Wilhelm, O. (2020) · Journal of Intelligence · doi:10.3390/jintelligence8040038 · Cited by 55
Clinical Psychological Science
Lilienfeld, S. O. (2016) · Clinical Psychological Science · doi:10.1177/2167702616673363 · Cited by 55
Implicit identification with death predicts change in suicide ideation during psychiatric treatment in adolescents
Glenn, C. R., Kleiman, E. M., Coppersmith, D. D. L., Santee, A. C., Esposito, E. C., Cha, C. B., Nock, M. K., & Auerbach, R. P. (2017) · Journal of Child Psychology and Psychiatry · doi:10.1111/jcpp.12769 · Cited by 54

Authors: Bakker, M., van Dijk, A., & Wicherts, J. M.
Affiliations: University of Amsterdam (NL)
University of Amsterdam, The Netherlands
Tilburg University (NL)
ORCID: Bakker, Marjan
Wicherts, Jelte M.
Venue: Perspectives on Psychological Science
Publisher: SAGE Publications
Published: 2012-11
Volume: 7
Issue: 6
Pages: 543-554
ISSN: 1745-6916 (ISSN-L)
1745-6924
DOI: 10.1177/1745691612459060
OpenAlex: W2157668596
Cited by: 800
References: 75
Citations by year: 2026: 14 · 2025: 32 · 2024: 39 · 2023: 50 · 2022: 52 · 2021: 100 · 2020: 63 · 2019: 82 · 2018: 87 · 2017: 88 · 2016: 56 · 2015: 59 · 2014: 52 · 2013: 19 · 2012: 5
FWCI: 49.25
Type: journal article
Language: en
Open access: bronze
doi.org
Licence: journals.sagepub.com/page/policies/text-and-data-mining-license (tdm)
Locations: Perspectives on Psychological Science (publishedVersion)
Research portal (Tilburg University) (publishedVersion)
PubMed (publishedVersion)
Data Archiving and Networked Services (DANS) (submittedVersion)
UvA-DARE (University of Amsterdam) (submittedVersion)
Funders: Nederlandse Organisatie voor Wetenschappelijk Onderzoek (016-125-385)
Topics: Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
Social Sciences › Psychology › Experimental and Cognitive Psychology › Psychology Research and Bibliometrics
Social Sciences › Education › Academic integrity and plagiarism
Keywords: null hypothesis significance testing, statistical power, publication bias, questionable research practices, false positives, meta-analysis, replication, sample size, p-values, research integrity, editorial policy

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

The Rules of the Game Called Psychological Science

Abstract

Overview

Integrity signals

Verification

Claims the paper makes

Claims other papers make about this one

Citations

Cites

Cited by

Related

Metadata

Record

About these checks