Cited by 800Meta-analysis and systematic reviewsPsychology Research and BibliometricsAcademic integrity and plagiarism

The DOI matches its Crossref record; 75 of the 75 references it cites were found in a registry or an index.

The Rules of the Game Called Psychological Science

Open access

Bakker, Marjan, van Dijk, Annette, Wicherts, Jelte M.·2012·Perspectives on Psychological Science

Open this paper in CoChat — free account Open PDF

Abstract

If science were a game, a dominant rule would probably be to collect results that are statistically significant. Several reviews of the psychological literature have shown that around 96% of papers involving the use of null hypothesis significance testing report significant outcomes for their main results but that the typical studies are insufficiently powerful for such a track record. We explain this paradox by showing that the use of several small underpowered samples often represents a more efficient research strategy (in terms of finding p < .05) than does the use of one larger (more powerful) sample. Publication bias and the most efficient strategy lead to inflated effects and high rates of false positives, especially when researchers also resorted to questionable research practices, such as adding participants after intermediate testing. We provide simulations that highlight the severity of such biases in meta-analyses. We consider 13 meta-analyses covering 281 primary studies in various fields of psychology and find indications of biases and/or an excess of significant results in seven. These results highlight the need for sufficiently powerful replications and changes in journal policies.

Integrity signals

Registry

Crossref · 10.1177/1745691612459060

Cites

75 of 75 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Claims the paper makes

We couldn’t retrieve this paper’s full text, so its claims and self-check weren’t run. The registry and citation checks still apply.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Two new Later Stone Age sites from the Final Pleistocene in the Falémé Valley, eastern Senegal
Ndiaye, M., Lespez, L., Tribolo, C., Rassé, M., Hadjas, I., Davidoux, S., Huysecom, É., & Douze, K. (2024) · Monash University Research Portal (Monash University) · doi:10.3929/ethz-b-000667478 · Cited by 24,214
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · PLoS Medicine · doi:10.1371/journal.pmed.0020124 · Cited by 10,505
The file drawer problem and tolerance for null results.
Rosenthal, R. (1979) · Psychological Bulletin · doi:10.1037/0033-2909.86.3.638 · Cited by 9,068
False-Positive Psychology: Undisclosed Flexibility in Data Collection and Analysis Allows Presenting Anything as Significant
Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011) · Psychological Science · doi:10.1177/0956797611417632 · Cited by 6,567
Journal of The American Statistical Association
Fienberg, S. E. (2004) · Encyclopedia of Statistical Sciences · doi:10.1002/0471667196.ess1317 · Cited by 6,289
Understanding and using the Implicit Association Test: I. An improved scoring algorithm.
Greenwald, A. G., Nosek, B. A., & Banaji, M. R. (2003) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.85.2.197 · Cited by 5,815
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · CHANCE · doi:10.1080/09332480.2005.10722754 · Cited by 4,702 · cited 3×
“Such practices lead to inflated ESs and increased false positive rates (Ioannidis, 2005, 2008b; Simmons et al., 2011).”
BackgroundResult
The earth is round (p < .05).
Cohen, J. (1994) · American Psychologist · doi:10.1037/0003-066x.49.12.997 · Cited by 4,003
Understanding and using the Implicit Association Test: III. Meta-analysis of predictive validity.
Greenwald, A. G., Poehlman, T. A., Uhlmann, E. L., & Banaji, M. R. (2009) · Journal of Personality and Social Psychology · doi:10.1037/a0015575 · Cited by 3,257
Belief in the law of small numbers.
Tversky, A., & Kahneman, D. (1971) · Psychological Bulletin · doi:10.1037/h0031322 · Cited by 3,246

Show all 75 references

Research works printed but not linked: 1

10.1037/0003-066X.56.2.128 · DOI · Paper page

Sorted by citations

Calculating and reporting effect sizes to facilitate cumulative science: A practical primer for t-tests and ANOVAs
Lakens, D. (2013) · Frontiers in Psychology · doi:10.3389/fpsyg.2013.00863 · Cited by 10,303
The preregistration revolution
Nosek, B. A., Ebersole, C. R., DeHaven, A. C., & Mellor, D. T. (2018) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1708274114 · Cited by 1,998
Choosing Prediction Over Explanation in Psychology: Lessons From Machine Learning
Yarkoni, T., & Westfall, J. (2017) · Perspectives on Psychological Science · doi:10.1177/1745691617693393 · Cited by 2,088
Increasing Transparency Through a Multiverse Analysis
Steegen, S., Tuerlinckx, F., Gelman, A., & Vanpaemel, W. (2016) · Perspectives on Psychological Science · doi:10.1177/1745691616658637 · Cited by 1,512
The Triple P-Positive Parenting Program: a systematic review and meta-analysis of a multi-level system of parenting support
Sanders, M. R., Kirby, J. N., Tellegen, C. L., & Day, J. J. (2014) · Clinical Psychology Review · doi:10.1016/j.cpr.2014.04.003 · Cited by 1,077
Reproducibility in Science: Improving the Standard for Basic and Preclinical Research
Begley, C. G., & Ioannidis, J. P. A. (2015) · Circulation Research · doi:10.1161/circresaha.114.303819 · Cited by 1,035
An Agenda for Purely Confirmatory Research
Wagenmakers, E.-J., Wetzels, R., Borsboom, D., van der Maas, H. L. J., & Kievit, R. A. (2012) · Perspectives on Psychological Science · doi:10.1177/1745691612463078 · Cited by 1,042
The oxytocin receptor: from intracellular signaling to behavior
Jurek, B., & Neumann, I. D. (2018) · Physiological Reviews · doi:10.1152/physrev.00031.2017 · Cited by 999
Statistical power and optimal design in experiments in which samples of participants respond to samples of stimuli.
Westfall, J., Kenny, D. A., & Judd, C. M. (2014) · Journal of Experimental Psychology: General · doi:10.1037/xge0000014 · Cited by 928
Degrees of Freedom in Planning, Running, Analyzing, and Reporting Psychological Studies: A Checklist to Avoid p-Hacking
Wicherts, J. M., Veldkamp, C. L. S., Augusteijn, H. E. M., Bakker, M., van Aert, R. C. M., & van Assen, M. A. L. M. (2016) · Frontiers in Psychology · doi:10.3389/fpsyg.2016.01832 · Cited by 754
Recommendations for Increasing Replicability in Psychology
Asendorpf, J. B., Conner, M. T., Fruyt, F. D., Houwer, J. D., Denissen, J. J. A., Fiedler, K., Fiedler, S., Funder, D. C., Kliegl, R. M., Nosek, B. A., et al. (2013) · European Journal of Personality · doi:10.1002/per.1919 · Cited by 876
Effect Size Guidelines, Sample Size Calculations, and Statistical Power in Gerontology
Brydges, C. R. (2019) · Innovation in Aging · doi:10.1093/geroni/igz036 · Cited by 826
Bilingual advantages in executive functioning either do not exist or are restricted to very specific and undetermined circumstances
Paap, K. R., Johnson, H. A., & Sawi, O. M. (2015) · Cortex · doi:10.1016/j.cortex.2015.04.014 · Cited by 905
The meaningfulness of effect sizes in psychological research: differences between sub-disciplines and the impact of potential biases
Schäfer, T., & Schwarz, M. A. (2019) · Frontiers in Psychology · doi:10.3389/fpsyg.2019.00813 · Cited by 774
Many Analysts, One Data Set: Making Transparent How Variations in Analytic Choices Affect Results
Silberzahn, R., Uhlmann, E. L., Martin, D. P., Anselmi, P., Aust, F., Awtrey, E. C., Bahník, Š., Bai, F., Bannard, C., Bonnier, E., et al. (2018) · Advances in Methods and Practices in Psychological Science · doi:10.1177/2515245917747646 · Cited by 773
Replicability, Robustness, and Reproducibility in Psychological Science
Nosek, B. A., Hardwicke, T. E., Moshontz, H., Allard, A., Corker, K. S., Dreber, A., Fidler, F., Hilgard, J., Kline Struhl, M., Nuijten, M. B., et al. (2022) · Annual Review of Psychology · doi:10.1146/annurev-psych-020821-114157 · Cited by 770
Empirical assessment of published effect sizes and power in the recent cognitive neuroscience and psychology literature
Szucs, D., & Ioannidis, J. P. A. (2017) · PLOS Biology · doi:10.1371/journal.pbio.2000797 · Cited by 761
Is the Replicability Crisis Overblown? Three Arguments Examined
Pashler, H., & Harris, C. R. (2012) · Perspectives on Psychological Science · doi:10.1177/1745691612463401 · Cited by 697
The Power of Bias in Economics Research
Ioannidis, J. P. A., Stanley, T. D., & Doucouliagos, H. (2017) · The Economic Journal · doi:10.1111/ecoj.12461 · Cited by 634
Empirically derived guidelines for effect size interpretation in social psychology
Lovakov, A. V., & Рафиковна, А. Е. (2021) · European Journal of Social Psychology · doi:10.1002/ejsp.2752 · Cited by 755
Facts Are More Important Than Novelty: Replication in the Education Sciences
Makel, M. C., & Plucker, J. A. (2014) · Educational Researcher · doi:10.3102/0013189x14545513 · Cited by 609
Psychology's Renaissance
Nelson, L. D., Simmons, J., & Simonsohn, U. (2018) · Annual Review of Psychology · doi:10.1146/annurev-psych-122216-011836 · Cited by 548
Why Science Is Not Necessarily Self-Correcting
Ioannidis, J. P. A. (2012) · Perspectives on Psychological Science · doi:10.1177/1745691612464056 · Cited by 563
Correcting for Bias in Psychology: A Comparison of Meta-Analytic Methods
Carter, E. C., Schönbrodt, F. D., Gervais, W. M., & Hilgard, J. (2019) · Advances in Methods and Practices in Psychological Science · doi:10.1177/2515245919847196 · Cited by 588
Registered Reports: A new publishing initiative at Cortex
Chambers, C. (2012) · Cortex · doi:10.1016/j.cortex.2012.12.016 · Cited by 608
Sequential hypothesis testing with Bayes factors: Efficiently testing mean differences.
Schönbrodt, F. D., Wagenmakers, E., Zehetleitner, M., & Perugini, M. (2015) · Psychological Methods · doi:10.1037/met0000061 · Cited by 599
The Ordinal Effects of Ostracism: A Meta-Analysis of 120 Cyberball Studies
Hartgerink, C. H. J., van Beest, I., Wicherts, J. M., & Williams, K. D. (2015) · PLOS ONE · doi:10.1371/journal.pone.0127002 · Cited by 558
Top 10 Replicated Findings From Behavioral Genetics
Plomin, R., DeFries, J. C., Knopik, V. S., & Neiderhiser, J. M. (2016) · Perspectives on Psychological Science · doi:10.1177/1745691615617439 · Cited by 517
The past, present and future of Registered Reports
Chambers, C. D., & Tzavella, L. (2021) · Nature Human Behaviour · doi:10.1038/s41562-021-01193-7 · Cited by 499
On doing better science: From thrill of discovery to policy implications
Antonakis, J. (2017) · The Leadership Quarterly · doi:10.1016/j.leaqua.2017.01.006 · Cited by 509
What You See Is What You Get? Enhancing Methodological Transparency in Management Research
Aguinis, H., Ramani, R. S., & Alabduljader, N. (2017) · Academy of Management Annals · doi:10.5465/annals.2016.0011 · Cited by 517
The alleged crisis and the illusion of exact replication
Stroebe, W., & Strack, F. (2014) · Perspectives on Psychological Science · doi:10.1177/1745691613514450 · Cited by 597
Publication Bias in Psychology: A Diagnosis Based on the Correlation between Effect Size and Sample Size
Kühberger, A., Fritz, A., & Scherndl, T. (2014) · PLoS ONE · doi:10.1371/journal.pone.0105825 · Cited by 399
A Practical Primer To Power Analysis for Simple Experimental Designs
Perugini, M., Gallucci, M., & Costantini, G. (2018) · International Review of Social Psychology · doi:10.5334/irsp.181 · Cited by 482
Does stereotype threat influence performance of girls in stereotyped domains? A meta-analysis
Flore, P. C., & Wicherts, J. M. (2015) · Journal of School Psychology · doi:10.1016/j.jsp.2014.10.002 · Cited by 410
Self-citation
Sample-Size Planning for More Accurate Statistical Power: A Method Adjusting Sample Effect Sizes for Publication Bias and Uncertainty
Anderson, S. F., Kelley, K., & Maxwell, S. E. (2017) · Psychological Science · doi:10.1177/0956797617723724 · Cited by 406
Is Ego Depletion Real? An Analysis of Arguments
Friese, M., Loschelder, D. D., Gieseler, K., Frankenbach, J., & Inzlicht, M. (2018) · Personality and Social Psychology Review · doi:10.1177/1088868318762183 · Cited by 438
Meta-analysis of psychological treatments for posttraumatic stress disorder in adult survivors of childhood abuse
Ehring, T., Welboren, R., Morina, N., Wicherts, J. M., Freitag, J., & Emmelkamp, P. M. G. (2014) · Clinical Psychology Review · doi:10.1016/j.cpr.2014.10.004 · Cited by 373
Implications of the Credibility Revolution for Productivity, Creativity, and Progress
Vazire, S. (2018) · Perspectives on Psychological Science · doi:10.1177/1745691617751884 · Cited by 398
A systematic review of teacher guidance during collaborative learning in primary and secondary education
van Leeuwen, A., & Janssen, J. (2019) · Educational Research Review · doi:10.1016/j.edurev.2019.02.001 · Cited by 355
The relation between statistical power and inference in fMRI
Cremers, H. R., Wager, T. D., & Yarkoni, T. (2017) · PLoS ONE · doi:10.1371/journal.pone.0184923 · Cited by 356
Statistical and Methodological Considerations for the Interpretation of Intranasal Oxytocin Studies
Walum, H., Waldman, I. D., & Young, L. J. (2016) · Biological Psychiatry · doi:10.1016/j.biopsych.2015.06.016 · Cited by 347
“From a statistical point of view this problem might not be any more serious for this field than other domains of psychology (20, 21, 31) or neuroscience in general (14).”
Background
Publication bias examined in meta-analyses from psychology and medicine: A meta-meta-analysis
van Aert, R. C. M., Wicherts, J. M., & van Assen, M. A. L. M. (2019) · PLoS ONE · doi:10.1371/journal.pone.0215052 · Cited by 365
Does Self-Control Training Improve Self-Control? A Meta-Analysis
Friese, M., Frankenbach, J., Job, V., & Loschelder, D. D. (2017) · Perspectives on Psychological Science · doi:10.1177/1745691617697076 · Cited by 360
Approach, avoidance, and affect: a meta-analysis of approach-avoidance tendencies in manual reaction time tasks
Phaf, R. H., Mohr, S. E., Rotteveel, M., & Wicherts, J. M. (2014) · Frontiers in Psychology · doi:10.3389/fpsyg.2014.00378 · Cited by 326
Reporting effect sizes in original psychological research: A discussion and tutorial.
Pek, J. C. X., & Flora, D. B. (2017) · Psychological Methods · doi:10.1037/met0000126 · Cited by 356
A Systematic Scoping Review of the Choice Architecture Movement: Toward Understanding When and Why Nudges Work
Szászi, B., Pálinkás, A., Pálfi, B., Szollosi, A., & Aczél, B. (2017) · Journal of Behavioral Decision Making · doi:10.1002/bdm.2035 · Cited by 336
The effects of psychological distance on abstraction: Two meta-analyses.
Soderberg, C. K., Callahan, S. P., Kochersberger, A. O., Amit, E., & Ledgerwood, A. (2014) · Psychological Bulletin · doi:10.1037/bul0000005 · Cited by 350
Continuously Cumulating Meta-Analysis and Replicability
Braver, S. L., Thoemmes, F. J., & Rosenthal, R. (2014) · Perspectives on Psychological Science · doi:10.1177/1745691614529796 · Cited by 308
Sailing From the Seas of Chaos Into the Corridor of Stability: Practical Recommendations to Increase the Informational Value of Studies
Lakens, D., & Evers, E. R. K. (2014) · Perspectives on Psychological Science · doi:10.1177/1745691614528520 · Cited by 266
Science or Art? How Aesthetic Standards Grease the Way Through the Publication Bottleneck but Undermine Science
Giner-Sorolla, R. (2012) · Perspectives on Psychological Science · doi:10.1177/1745691612457576 · Cited by 274
It’s Time to Broaden the Replicability Conversation: Thoughts for and From Clinical Psychological Science
Tackett, J. L., Lilienfeld, S. O., Patrick, C. J., Johnson, S. L., Krueger, R. F., Miller, J. D., Oltmanns, T. F., & Shrout, P. E. (2017) · Perspectives on Psychological Science · doi:10.1177/1745691617690042 · Cited by 251
Qualitative Research Methods
Kazdin, A. E. (2021) · Cambridge University Press eBooks · doi:10.1017/9781108993647.010 · Cited by 279
Replication in Second Language Research: Narrative and Systematic Reviews and Recommendations for the Field
Marsden, E. J., Morgan‐Short, K., Thompson‐Lee, S., & Abugaber, D. (2018) · Language Learning · doi:10.1111/lang.12286 · Cited by 286
Meta-analysis of parent-mediated interventions for young children with autism spectrum disorder
Nevill, R. E., Lecavalier, L., & Stratis, E. A. (2018) · Autism · doi:10.1177/1362361316677838 · Cited by 265
Meta-analysis using effect size distributions of only statistically significant studies.
van Assen, M. A. L. M., van Aert, R. C. M., & Wicherts, J. M. (2015) · Psychological Methods · doi:10.1037/met0000025 · Cited by 265
Self-citation
How WEIRD is CHI?
Linxen, S., Sturm, C., Brühlmann, F., Cassau, V., Opwis, K., & Reinecke, K. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445488 · Cited by 254
Are bilingual advantages dependent upon specific tasks or specific bilingual experiences?
Paap, K. R., Johnson, H. A., & Sawi, O. M. (2014) · Journal of Cognitive Psychology · doi:10.1080/20445911.2014.944914 · Cited by 254
Bilingual advantages in executive functioning: problems in convergent validity, discriminant validity, and the identification of the theoretical constructs
Paap, K. R. (2014) · Frontiers in Psychology · doi:10.3389/fpsyg.2014.00962 · Cited by 275
A Short (Personal) Future History of Revolution 2.0
Spellman, B. A. (2015) · Perspectives on Psychological Science · doi:10.1177/1745691615609918 · Cited by 213
The Chrysalis Effect: How Ugly Initial Results Metamorphosize Into Beautiful Articles
O’Boyle, E. H., Banks, G. C., & Gonzalez-Mulé, E. (2017) · Journal of Management · doi:10.1177/0149206314527133 · Cited by 246
Men, women…who cares? A population-based study on sex differences and gender roles in empathy and moral cognition
Báez, S., Flichtentrei, D., Prats, M., Mastandueno, R., García, A. M., Cetkovich, M., & Ibáñez, A. M. (2017) · PLoS ONE · doi:10.1371/journal.pone.0179336 · Cited by 276
Publication bias and the limited strength model of self-control: has the evidence for ego depletion been overestimated?
Carter, E. C., & McCullough, M. E. (2014) · Frontiers in Psychology · doi:10.3389/fpsyg.2014.00823 · Cited by 381
Meaningful associations in the adolescent brain cognitive development study
Dick, A. S., Lopez, D. A., Watts, A. L., Heeringa, S. G., Reuter, C., Bartsch, H., Fan, C. C., Kennedy, D. N., Palmer, C. E., Marshall, A. T., et al. (2021) · NeuroImage · doi:10.1016/j.neuroimage.2021.118262 · Cited by 240
Exploratory data analysis as a foundation of inductive research
Jebb, A. T., Parrigon, S., & Woo, S. E. (2016) · Human Resource Management Review · doi:10.1016/j.hrmr.2016.08.003 · Cited by 300
What meta-analyses reveal about the replicability of psychological research.
Stanley, T. D., Carter, E. C., & Doucouliagos, H. (2018) · Psychological Bulletin · doi:10.1037/bul0000169 · Cited by 307
Do the disadvantaged legitimize the social system? A large-scale test of the status–legitimacy hypothesis.
Brandt, M. J. (2013) · Journal of Personality and Social Psychology · doi:10.1037/a0031751 · Cited by 248
“An A Is An A”: The New Bottom Line For Valuing Academic Research
Aguinis, H., Cummings, C., Ramani, R. S., & Cummings, T. G. (2019) · Academy of Management Perspectives · doi:10.5465/amp.2017.0193 · Cited by 242
Conducting Meta-Analyses Based on p Values: Reservations and Recommendations for Applying p -Uniform and p -Curve
van Aert, R. C. M., Wicherts, J. M., & van Assen, M. A. L. M. (2016) · Perspectives on Psychological Science · doi:10.1177/1745691616650874 · Cited by 219
Self-citation
A tutorial on assessing statistical power and determining sample size for structural equation models.
Jobst, L. J., Bader, M., & Moshagen, M. (2021) · Psychological Methods · doi:10.1037/met0000423 · Cited by 236
Current Incentives for Scientists Lead to Underpowered Studies with Erroneous Conclusions
Higginson, A. D., & Munafò, M. R. (2016) · PLOS Biology · doi:10.1371/journal.pbio.2000995 · Cited by 192
Outlier removal, sum scores, and the inflation of the Type I error rate in independent samples t tests: the power of alternatives and recommendations.
Bakker, M., & Wicherts, J. (2014) · Psychological methods · doi:10.1037/met0000014 · Cited by 173
Best research practices in psychology: Illustrating epistemological and pragmatic considerations with the case of relationship science.
Finkel, E. J., Eastwick, P. W., & Reis, H. T. (2015) · Journal of Personality and Social Psychology · doi:10.1037/pspi0000007 · Cited by 208
Safeguard Power as a Protection Against Imprecise Power Estimates
Perugini, M., Gallucci, M., & Costantini, G. (2014) · Perspectives on Psychological Science · doi:10.1177/1745691614528519 · Cited by 203
Psychology's Replication Crisis and Clinical Psychological Science
Tackett, J. L., Brandes, C. M., King, K. M., & Markon, K. E. (2019) · Annual Review of Clinical Psychology · doi:10.1146/annurev-clinpsy-050718-095710 · Cited by 221
An Assessment of the Magnitude of Effect Sizes
Paterson, T. A., Harms, P. D., Steel, P., & Credé, M. (2015) · Journal of Leadership & Organizational Studies · doi:10.1177/1548051815614321 · Cited by 211
Mapping brain asymmetry in health and disease through the ENIGMA consortium
Kong, X., Postema, M. C., Guadalupe, T. M., de Kovel, C. G. F., Boedhoe, P. S. W., Hoogman, M., Mathias, S. R., van Rooij, D., Schijven, D., Glahn, D. C., et al. (2020) · Human Brain Mapping · doi:10.1002/hbm.25033 · Cited by 196
Reproducible research in sport and exercise psychology: The role of sample sizes
Schweizer, G., & Furley, P. (2016) · Psychology of sport and exercise · doi:10.1016/j.psychsport.2015.11.005 · Cited by 168
Slice-to-volume medical image registration: A survey
Ferrante, E., & Nikos Paragios, A. (2017) · Medical Image Analysis · doi:10.1016/j.media.2017.04.010 · Cited by 190
Statistical Rituals: The Replication Delusion and How We Got There
Gigerenzer, G. (2018) · Advances in Methods and Practices in Psychological Science · doi:10.1177/2515245918771329 · Cited by 263
Reducing bias, increasing transparency and calibrating confidence with preregistration
Hardwicke, T. E., & Wagenmakers, E. (2023) · Nature Human Behaviour · doi:10.1038/s41562-022-01497-2 · Cited by 205
Scientific Utopia III: Crowdsourcing Science
Uhlmann, E. L., Ebersole, C. R., Chartier, C. R., Errington, T. M., Kidwell, M. C., Lai, C. K., McCarthy, R. J., Riegelman, A., Silberzahn, R., & Nosek, B. A. (2019) · Perspectives on Psychological Science · doi:10.1177/1745691619850561 · Cited by 191
“Rarely safe to assume”: Evaluating the use and interpretation of manipulation checks in experimental social psychology
Ejelöv, E., & Luke, T. J. (2019) · Journal of Experimental Social Psychology · doi:10.1016/j.jesp.2019.103937 · Cited by 175
The Preregistration RevolutionPreprint
Nosek, B. A., Ebersole, C. R., DeHaven, A. C., & Mellor, D. T. (2017) · OSF Preprints (OSF Preprints) · doi:10.31219/osf.io/2dxu5 · Cited by 220
Reconceptualizing replication as a sequence of different studies: A replication typology
Hüffmeier, J., Mazei, J., & Schultze, T. (2016) · Journal of Experimental Social Psychology · doi:10.1016/j.jesp.2015.09.009 · Cited by 177
Questions About Questionable Research Practices in the Field of Management: A Guest Commentary
Banks, G. C., O’Boyle, E. H., Pollack, J. M., White, C. D., Batchelor, J. H., Whelpley, C. E., Abston, K. A., Bennett, A. A., & Adkins, C. L. (2016) · Journal of Management · doi:10.1177/0149206315619011 · Cited by 163
Science’s reproducibility and replicability crisis: International business is not immune
Aguinis, H., Cascio, W., & Ramani, R. S. (2017) · Research Methods in International Business · doi:10.1057/s41267-017-0081-0 · Cited by 163
Questionable research practices among italian research psychologists
Agnoli, F., Wicherts, J. M., Veldkamp, C. L. S., Albiero, P., & Cubelli, R. (2017) · PLoS ONE · doi:10.1371/journal.pone.0172792 · Cited by 173
Problems with Traditional Science Publishing and Finding a Wider Niche for Post-Publication Peer Review
Teixeira da Silva, J. A., & Dobránszki, J. (2015) · Accountability in Research · doi:10.1080/08989621.2014.899909 · Cited by 158
Lessons From Pinocchio: Cues to Deception May Be Highly Exaggerated
Luke, T. J. (2019) · Perspectives on Psychological Science · doi:10.1177/1745691619838258 · Cited by 174
How (not) to measure infant Theory of Mind: Testing the replicability and validity of four non-verbal measures
Dörrenberg, S., Hannes Rakoczy, B., & Liszkowski, U. (2018) · Cognitive Development · doi:10.1016/j.cogdev.2018.01.001 · Cited by 158
Seven Easy Steps to Open Science
Crüwell, S., van Doorn, J. B., Etz, A. J., Makel, M. C., Moshontz, H., Niebaum, J. C., Orben, A. C., Parsons, S. D., & Schulte‐Mecklenbeck, M. (2019) · Zeitschrift für Psychologie · doi:10.1027/2151-2604/a000387 · Cited by 186
Clarifications on the application and interpretation of the test for excess significance and its extensions
Ioannidis, J. P. A. (2013) · Journal of Mathematical Psychology · doi:10.1016/j.jmp.2013.03.002 · Cited by 149
Answers to 18 Questions About Open Science Practices
Banks, G. C., Field, J. G., Oswald, F. L., O’Boyle, E. H., Landis, R. S., Rupp, D. E., & Rogelberg, S. G. (2018) · Journal of Business and Psychology · doi:10.1007/s10869-018-9547-8 · Cited by 167
Novel Effect Size Interpretation Guidelines and an Evaluation of Statistical Power in Rehabilitation Research
Kinney, A. R., Eakman, A. M., & Graham, J. E. (2020) · Archives of Physical Medicine and Rehabilitation · doi:10.1016/j.apmr.2020.02.017 · Cited by 148
Researchers’ Intuitions About Power in Psychological Research
Bakker, M., Hartgerink, C. H. J., Wicherts, J. M., & van der Maas, H. L. J. (2016) · Psychological Science · doi:10.1177/0956797616647519 · Cited by 158
Having your cake and eating it too: Flexibility and power with mass univariate statistics for ERP data
Fields, E. C., & Kuperberg, G. R. (2019) · Psychophysiology · doi:10.1111/psyp.13468 · Cited by 153
The role of test-retest reliability in measuring individual and group differences in executive functioning
Paap, K. R., & Sawi, O. M. (2016) · Journal of Neuroscience Methods · doi:10.1016/j.jneumeth.2016.10.002 · Cited by 150
A Primer for Conducting Experiments in Human–Robot Interaction
Hoffman, G., & Zhao, X. (2020) · ACM Transactions on Human-Robot Interaction · doi:10.1145/3412374 · Cited by 150
How robust are anticipatory looking measures of Theory of Mind? Replication attempts across the life span
Kulke, L., Reiß, M., Krist, H., & Hannes Rakoczy, B. (2017) · Cognitive Development · doi:10.1016/j.cogdev.2017.09.001 · Cited by 134

Authors: Bakker, M., van Dijk, A., & Wicherts, J. M.
Affiliations: University of Amsterdam (NL)
University of Amsterdam, The Netherlands
Tilburg University (NL)
ORCID: Bakker, Marjan
Wicherts, Jelte M.
Venue: Perspectives on Psychological Science
Publisher: SAGE Publications
Published: 2012-11
Volume: 7
Issue: 6
Pages: 543-554
ISSN: 1745-6916 (ISSN-L)
1745-6924
DOI: 10.1177/1745691612459060
OpenAlex: W2157668596
Cited by: 800
References: 75
Citations by year: 2026: 14 · 2025: 32 · 2024: 39 · 2023: 50 · 2022: 52 · 2021: 100 · 2020: 63 · 2019: 82 · 2018: 87 · 2017: 88 · 2016: 56 · 2015: 59 · 2014: 52 · 2013: 19 · 2012: 5
FWCI: 49.25
Type: journal article
Language: en
Open access: bronze
doi.org
Licence: journals.sagepub.com/page/policies/text-and-data-mining-license (tdm)
Locations: Perspectives on Psychological Science (publishedVersion)
Research portal (Tilburg University) (publishedVersion)
PubMed (publishedVersion)
Data Archiving and Networked Services (DANS) (submittedVersion)
UvA-DARE (University of Amsterdam) (submittedVersion)
Funders: Nederlandse Organisatie voor Wetenschappelijk Onderzoek (016-125-385)
Topics: Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
Social Sciences › Psychology › Experimental and Cognitive Psychology › Psychology Research and Bibliometrics
Social Sciences › Education › Academic integrity and plagiarism
Keywords: null hypothesis significance testing, statistical power, publication bias, questionable research practices, false positives, meta-analysis, replication, sample size, p-values, research integrity, editorial policy

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

The Rules of the Game Called Psychological Science

Abstract

Overview

Integrity signals

Verification

Claims the paper makes

Claims other papers make about this one

Citations

Cites

Cited by

Related

Metadata

Record

About these checks