Cited by 800Meta-analysis and systematic reviewsPsychology Research and BibliometricsAcademic integrity and plagiarism

The DOI matches its Crossref record; 75 of the 75 references it cites were found in a registry or an index.

The Rules of the Game Called Psychological Science

Open access

Bakker, Marjan, van Dijk, Annette, Wicherts, Jelte M.·2012·Perspectives on Psychological Science

Open this paper in CoChat — free account Open PDF

Abstract

If science were a game, a dominant rule would probably be to collect results that are statistically significant. Several reviews of the psychological literature have shown that around 96% of papers involving the use of null hypothesis significance testing report significant outcomes for their main results but that the typical studies are insufficiently powerful for such a track record. We explain this paradox by showing that the use of several small underpowered samples often represents a more efficient research strategy (in terms of finding p < .05) than does the use of one larger (more powerful) sample. Publication bias and the most efficient strategy lead to inflated effects and high rates of false positives, especially when researchers also resorted to questionable research practices, such as adding participants after intermediate testing. We provide simulations that highlight the severity of such biases in meta-analyses. We consider 13 meta-analyses covering 281 primary studies in various fields of psychology and find indications of biases and/or an excess of significant results in seven. These results highlight the need for sufficiently powerful replications and changes in journal policies.

Integrity signals

Registry

Crossref · 10.1177/1745691612459060

Cites

75 of 75 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Claims the paper makes

We couldn’t retrieve this paper’s full text, so its claims and self-check weren’t run. The registry and citation checks still apply.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Two new Later Stone Age sites from the Final Pleistocene in the Falémé Valley, eastern Senegal
Ndiaye, M., Lespez, L., Tribolo, C., Rassé, M., Hadjas, I., Davidoux, S., Huysecom, É., & Douze, K. (2024) · Monash University Research Portal (Monash University) · doi:10.3929/ethz-b-000667478 · Cited by 24,214
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · PLoS Medicine · doi:10.1371/journal.pmed.0020124 · Cited by 10,505
The file drawer problem and tolerance for null results.
Rosenthal, R. (1979) · Psychological Bulletin · doi:10.1037/0033-2909.86.3.638 · Cited by 9,068
False-Positive Psychology: Undisclosed Flexibility in Data Collection and Analysis Allows Presenting Anything as Significant
Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011) · Psychological Science · doi:10.1177/0956797611417632 · Cited by 6,567
Journal of The American Statistical Association
Fienberg, S. E. (2004) · Encyclopedia of Statistical Sciences · doi:10.1002/0471667196.ess1317 · Cited by 6,289
Understanding and using the Implicit Association Test: I. An improved scoring algorithm.
Greenwald, A. G., Nosek, B. A., & Banaji, M. R. (2003) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.85.2.197 · Cited by 5,815
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · CHANCE · doi:10.1080/09332480.2005.10722754 · Cited by 4,702 · cited 3×
“Such practices lead to inflated ESs and increased false positive rates (Ioannidis, 2005, 2008b; Simmons et al., 2011).”
BackgroundResult
The earth is round (p < .05).
Cohen, J. (1994) · American Psychologist · doi:10.1037/0003-066x.49.12.997 · Cited by 4,003
Understanding and using the Implicit Association Test: III. Meta-analysis of predictive validity.
Greenwald, A. G., Poehlman, T. A., Uhlmann, E. L., & Banaji, M. R. (2009) · Journal of Personality and Social Psychology · doi:10.1037/a0015575 · Cited by 3,257
Belief in the law of small numbers.
Tversky, A., & Kahneman, D. (1971) · Psychological Bulletin · doi:10.1037/h0031322 · Cited by 3,246

Show all 75 references

Research works printed but not linked: 1

10.1037/0003-066X.56.2.128 · DOI · Paper page

Sorted by citations

Items 201–300 of 799

The Weak Spots in Contemporary Science (and How to Fix Them)
Wicherts, J. M. (2017) · Animals · doi:10.3390/ani7120090 · Cited by 56
Realizing the full potential of psychometric meta-analysis for a cumulative science and practice of human resource management
Öneş, D. S., Viswesvaran, C., & Schmidt, F. L. (2016) · Human Resource Management Review · doi:10.1016/j.hrmr.2016.09.011 · Cited by 56
Null hypothesis significance testing and Type I error: The domain problem
Trafimow, D., & Earp, B. (2017) · New Ideas in Psychology · doi:10.1016/j.newideapsych.2017.01.002 · Cited by 57
Raising Awareness for the Replication Crisis in Clinical Psychology by Focusing on Inconsistencies in Psychotherapy Research: How Much Can We Rely on Published Findings from Efficacy Trials?
Hengartner, M. P. (2018) · Frontiers in Psychology · doi:10.3389/fpsyg.2018.00256 · Cited by 57
Detecting when timeseries differ: Using the Bootstrapped Differences of Timeseries (BDOTS) to analyze Visual World Paradigm data (and more)
Seedorff, M., Oleson, J., & McMurray, B. (2018) · Journal of Memory and Language · doi:10.1016/j.jml.2018.05.004 · Cited by 52
A reproducible systematic map of research on the illusory truth effect
Henderson, E. L., Westwood, S. J., & Simons, D. J. (2021) · Psychonomic Bulletin & Review · doi:10.3758/s13423-021-01995-w · Cited by 68
Motivating the academic mind: High-level construal of academic goals enhances goal meaningfulness, motivation, and self-concordance
Davis, W. E., Kelley, N. J., Kim, J., Tang, D., & Hicks, J. A. (2015) · Motivation and Emotion · doi:10.1007/s11031-015-9522-x · Cited by 63
Incorrect results in software engineering experiments: How to improve research practices
Jørgensen, M., Dybå, T., Liestøl, K., & Sjøberg, D. I. K. (2015) · Journal of Systems and Software · doi:10.1016/j.jss.2015.03.065 · Cited by 63
Eight‐week antidepressant treatment reduces functional connectivity in first‐episode drug‐naïve patients with major depressive disorder
Li, L., Su, Y., Wu, Y., Castellanos, F. X., Li, K., Li, J., Si, T., & Yan, C. (2021) · Human Brain Mapping · doi:10.1002/hbm.25391 · Cited by 53
Can the goal of honesty be primed?
Pashler, H., Rohrer, D., & Harris, C. R. (2013) · Journal of Experimental Social Psychology · doi:10.1016/j.jesp.2013.05.011 · Cited by 50
Is research in social psychology politically biased? Systematic empirical tests and a forecasting survey to address the controversy
Eitan, O., Viganola, D., Inbar, Y., Dreber, A., Johannesson, M., Pfeiffer, T., Thau, S., & Uhlmann, E. L. (2018) · Journal of Experimental Social Psychology · doi:10.1016/j.jesp.2018.06.004 · Cited by 55
Preregistration of Analyses of Preexisting Data
Mertens, G., & Krypotos, A. (2019) · Psychologica Belgica · doi:10.5334/pb.493 · Cited by 49
A Concise Set of Core Recommendations to Improve the Dependability of Psychological Research
Lishner, D. A. (2015) · Review of General Psychology · doi:10.1037/gpr0000028 · Cited by 49
Emotion Regulation in Borderline Personality Disorder: An Experimental Investigation of the Effects of Instructed Acceptance and Suppression
Dixon–Gordon, K. L., Turner, B. J., Rosenthal, M. Z., & Chapman, A. L. (2017) · Behavior Therapy · doi:10.1016/j.beth.2017.03.001 · Cited by 49
Dark Pathways to Achievement in Science: Researchers’ Achievement Goals Predict Engagement in Questionable Research Practices
Janke, S., Daumiller, M. H., & Rudert, S. C. (2018) · Social Psychological and Personality Science · doi:10.1177/1948550618790227 · Cited by 46
Validity of content‐based techniques for credibility assessment—How telling is an extended meta‐analysis taking research bias into account?
Oberlader, V. A., Quinten, L. K. J. L., Banse, R., Volbert, R., Schmidt, A. F., & Schönbrodt, F. D. (2020) · Applied Cognitive Psychology · doi:10.1002/acp.3776 · Cited by 53
Peer Tutoring and Academic Achievement in Mathematics: A Meta-Analysis
Alegre, F., Miravet, L. M., Valentín, G. L., & Maroto, A. (2017) · Eurasia Journal of Mathematics Science and Technology Education · doi:10.12973/ejmste/79805 · Cited by 50
Questionable research practices in student final theses – Prevalence, attitudes, and the role of the supervisor’s perceived attitudes
Krishna, A., & Peter, S. M. (2018) · PLoS ONE · doi:10.1371/journal.pone.0203470 · Cited by 46
Robust data and power in infant research: A case study of the effect of number of infants and number of trials in visual preference procedures
DeBolt, M. C., Rhemtulla, M., & Oakes, L. M. (2020) · Infancy · doi:10.1111/infa.12337 · Cited by 44
Benefits and Pitfalls of Debunking Interventions to Counter mRNA Vaccination Misinformation During the COVID-19 Pandemic
Schmid, P., & Betsch, C. (2022) · Science Communication · doi:10.1177/10755470221129608 · Cited by 44
Publication bias, statistical power and reporting practices in the Journal of Sports Sciences: potential barriers to replicability
Mesquida, C., Murphy, J., Lakens, D., & Warne, J. P. (2023) · Journal of Sports Sciences · doi:10.1080/02640414.2023.2269357 · Cited by 57
A Tutorial on Hunting Statistical Significance by Chasing N
Szűcs, D. (2016) · Frontiers in Psychology · doi:10.3389/fpsyg.2016.01444 · Cited by 47
Strategies to improve the evidence base of animal-assisted interventions
Kazdin, A. E. (2017) · Applied Developmental Science · doi:10.1080/10888691.2016.1191952 · Cited by 56
Too Good to be False: Nonsignificant Results Revisited
Hartgerink, C. H. J., Wicherts, J. M., & van Assen, M. A. L. M. (2017) · Collabra Psychology · doi:10.1525/collabra.71 · Cited by 45
Flexible yet fair: blinding analyses in experimental psychology
Dutilh, G., Sarafoglou, A., & Wagenmakers, E. (2019) · Synthese · doi:10.1007/s11229-019-02456-7 · Cited by 52
Intelligence and Adaptive Behavior: A Meta-Analysis
Alexander, R. M., & Reynolds, M. R. (2020) · School Psychology Review · doi:10.1080/2372966x.2020.1717374 · Cited by 52
Practical Methodological Reform Needs Good Theory
Gervais, W. M. (2021) · Perspectives on Psychological Science · doi:10.1177/1745691620977471 · Cited by 50
Ethnic preferences in friendships and casual contacts between majority and minority children in the Netherlands
Fortuin, J., van Geel, M., Žiberna, A., & Vedder, P. H. (2014) · International Journal of Intercultural Relations · doi:10.1016/j.ijintrel.2014.05.005 · Cited by 44
Effect Declines Are Systematic, Strong, and Ubiquitous: A Meta-Meta-Analysis of the Decline Effect in Intelligence Research
Pietschnig, J., Siegel, M., Eder, J. S. N., & Gittler, G. (2019) · Frontiers in Psychology · doi:10.3389/fpsyg.2019.02874 · Cited by 52
Culture and vocational interests: The moderating role of collectivism and gender egalitarianism.
Ott-Holland, C. J., Huang, J. L. W., Ryan, A. M., Elizondo, F., & Wadlington, P. L. (2013) · Journal of Counseling Psychology · doi:10.1037/a0033587 · Cited by 44
I’m merciful, am I not? Facets of narcissism and forgiveness revisited
Fatfouta, R., Zeigler‐Hill, V., & Schröder–Abé, M. (2017) · Journal of Research in Personality · doi:10.1016/j.jrp.2017.07.007 · Cited by 47
Approaching a fair deal for significance and other concerns
Giner‐Sorolla, R. (2016) · Journal of Experimental Social Psychology · doi:10.1016/j.jesp.2016.01.010 · Cited by 44
Effect Sizes, Power, and Biases in Intelligence Research: A Meta-Meta-Analysis
Nuijten, M. B., van Assen, M. A. L. M., Augusteijn, H. E. M., Crompvoets, E. A. V., & Wicherts, J. M. (2020) · Journal of Intelligence · doi:10.3390/jintelligence8040036 · Cited by 42
Controlling decision errors with minimal costs: The sequential probability ratio t test.
Schnuerch, M., & Erdfelder, E. (2019) · Psychological Methods · doi:10.1037/met0000234 · Cited by 41
Psychology's WEIRD Problems
de Oliveira, G. S., & Baggs, E. (2023) · Cambridge University Press eBooks · doi:10.1017/9781009303538 · Cited by 57
Enhancing transparency of the research process to increase accuracy of findings: A guide for relationship researchers
Campbell, L., Loving, T. J., & LeBel, E. P. (2014) · Personal Relationships · doi:10.1111/pere.12053 · Cited by 40
Questionable research practices may have little effect on replicability
Ulrich, R., & Miller, J. O. (2020) · eLife · doi:10.7554/elife.58237 · Cited by 50
America and the Age of Genocide
Leidner, B. (2015) · Personality and Social Psychology Bulletin · doi:10.1177/0146167215607195 · Cited by 42
How to build models for government: criteria driving model acceptance in policymaking
Kolkman, D. A., Campo, P., Balke-Visser, T., & Gilbert, N. (2016) · Policy Sciences · doi:10.1007/s11077-016-9250-4 · Cited by 39
Responsible Research Assessment I: Implementing DORA and CoARA for hiring and promotion in psychologyPreprint
Schönbrodt, F. D., Gärtner, A., Frank, M., Gollwitzer, M., Ihle, M., Mischkowski, D., Phan, L. V., Schmitt, M., Scheel, A. M., Schubert, A., et al. (2022) · doi:10.31234/osf.io/rgh5b · Cited by 40
Single-case experimental research designs.
Kazdin, A. E. (2015) · Methodological issues and strategies in clinical research (4th ed.). · doi:10.1037/14805-029 · Cited by 39
Applying Bayesian statistics to the study of psychological trauma: A suggestion for future research.
Yalch, M. (2016) · Psychological Trauma · doi:10.1037/tra0000096 · Cited by 38
Optimizing Research Payoff
Miller, J. O., & Ulrich, R. (2016) · Perspectives on Psychological Science · doi:10.1177/1745691616649170 · Cited by 41
Conditional equivalence testing: An alternative remedy for publication bias
Campbell, H., & Gustafson, P. (2018) · PLoS ONE · doi:10.1371/journal.pone.0195145 · Cited by 42
Degrees of Freedom in Multigroup Confirmatory Factor Analyses
Schroeders, U., & Gnambs, T. (2018) · European Journal of Psychological Assessment · doi:10.1027/1015-5759/a000500 · Cited by 40
Is Healthy Neuroticism Associated with Chronic Conditions? A Coordinated Integrative Data Analysis
Weston, S. J., Graham, E. K., Turiano, N. A., Aschwanden, D., Booth, T., Harrison, F., James, B. D., Lewis, N. A., Makkar, S. R., Mueller, S., et al. (2020) · Collabra Psychology · doi:10.1525/collabra.267 · Cited by 41
In the Best Interest of the Child: the Norwegian Approach to Child Protection
Melinder, A. M. D., van der Hagen, M. A., & Sandberg, K. (2021) · International Journal on Child Maltreatment Research Policy and Practice · doi:10.1007/s42448-021-00078-6 · Cited by 39
Domestic Homicide: A Synthesis of Systematic Review Evidence
Kim, B., & Merlo, A. V. (2021) · Trauma Violence & Abuse · doi:10.1177/15248380211043812 · Cited by 39
Introduction to the Special Section on Advancing Our Methods and Practices
Ledgerwood, A. (2014) · Perspectives on Psychological Science · doi:10.1177/1745691614529448 · Cited by 39
More than noise?—Field investigations of intraspecific acoustic communication in dogs (Canis familiaris)
Pongrácz, P., Szabó, É., Kis, A., Péter, A., & Ádám, M. (2014) · Applied Animal Behaviour Science · doi:10.1016/j.applanim.2014.08.003 · Cited by 37
Competition for priority harms the reliability of science, but reforms can help
Tiokhin, L., Yan, M., & Morgan, T. J. H. (2021) · Nature Human Behaviour · doi:10.1038/s41562-020-01040-1 · Cited by 48
The credibility crisis in research: Can economics tools help?
Gall, T., Ioannidis, J. P. A., & Maniadis, Z. (2017) · PLoS Biology · doi:10.1371/journal.pbio.2001846 · Cited by 35
Reproducibility in the absence of selective reporting: An illustration from large‐scale brain asymmetry research
Kong, X., ENIGMA Laterality Working Group, & Francks, C. (2020) · Human Brain Mapping · doi:10.1002/hbm.25154 · Cited by 34
(When and how) does basic research in clinical psychology lead to more effective psychological treatment for mental disorders?
Ehring, T. W., Limburg, K., Kunze, A. E., Wittekind, C. E., Werner, G. G., Wolkenstein, L., Guzey, M., & Cludius, B. (2022) · Clinical Psychology Review · doi:10.1016/j.cpr.2022.102163 · Cited by 43
How can preregistration contribute to research in our field?
Jonas, K. J., & Cesario, J. (2015) · Comprehensive Results in Social Psychology · doi:10.1080/23743603.2015.1070611 · Cited by 38
Internal conceptual replications do not increase independent replication success
Kunert, R. (2016) · Psychonomic Bulletin & Review · doi:10.3758/s13423-016-1030-9 · Cited by 35
Psychological science’s replicability crisis and what it means for science in the courtroom.
Chin, J. M. (2014) · Psychology Public Policy and Law · doi:10.1037/law0000012 · Cited by 39
Practical tools and strategies for researchers to increase replicability
Nuijten, M. B. (2018) · Developmental Medicine & Child Neurology · doi:10.1111/dmcn.14054 · Cited by 39
Methods of Evolutionary Sciences
Simpson, J. A., & Campbell, L. (2015) · Wiley eBooks · doi:10.1002/9780470939376.ch4 · Cited by 36
A Bayesian Approach to the Correction for Multiplicity
de Jong, T. (2019) · doi:10.31234/osf.io/s56mk · Cited by 35
Do Salient Social Norms Moderate Mortality Salience Effects? A (Challenging) Meta-Analysis of Terror Management Studies
Schindler, S., Hilgard, J., Fritsche, I., Burke, B., & Pfattheicher, S. (2023) · Personality and Social Psychology Review · doi:10.1177/10888683221107267 · Cited by 43
“Given that publication bias and researcher degrees of freedom are understood to be pervasive problems in psychological science (Bakker et al., 2012; Franco et al., 2014, 2016; Hesse, 2018; John et al., 2012), unadjusted effect sizes are likely to be overestimated (Kvarven et al., 2020).”
BackgroundResultInfluential
Recommendations in pre-registrations and internal review board proposals promote formal power analyses but do not increase sample size
Bakker, M., Veldkamp, C. L. S., van den Akker, O. R., van Assen, M. A. L. M., Crompvoets, E. A. V., Ong, H. H., & Wicherts, J. M. (2020) · PLoS ONE · doi:10.1371/journal.pone.0236079 · Cited by 39
Bringing an Intersectional Lens to “Open” Science: An Analysis of Representation in the Reproducibility Project
Sabik, N. J., Matsick, J. L., McCormick‐Huhn, K., & Cole, E. R. (2021) · Psychology of Women Quarterly · doi:10.1177/03616843211035678 · Cited by 35
I’m not sure that curve means what you think it means: Toward a [more] realistic understanding of the role of eye-movement generation in the Visual World Paradigm
McMurray, B. (2022) · Psychonomic Bulletin & Review · doi:10.3758/s13423-022-02143-8 · Cited by 37
Big Little Lies: A Compendium and Simulation of p-Hacking StrategiesPreprint
Stefan, A. M., & Schönbrodt, F. D. (2022) · doi:10.31234/osf.io/xy2dk · Cited by 35
Are they for us or against us? How intergroup metaperceptions shape foreign policy attitudes
O’Brien, T. C., Leidner, B., & Tropp, L. R. (2017) · Group Processes & Intergroup Relations · doi:10.1177/1368430216684645 · Cited by 33
Negative affect is unrelated to fluctuations in hormone levels across the menstrual cycle: Evidence from a multisite observational study across two successive cycles
Hengartner, M. P., Krüger, T. H. C., Geraedts, K., Tronci, E., Mancini, T., Ille, F., Egli, M., Röblitz, S., Ehrig, R. M., Saleh, L., et al. (2017) · Journal of Psychosomatic Research · doi:10.1016/j.jpsychores.2017.05.018 · Cited by 36
Predicting attention shifting abilities from self-reported media multitasking
Elbe, P., Sörman, D. E., Mellqvist, E., Brändström, J., & Ljungberg, J. K. (2019) · Psychonomic Bulletin & Review · doi:10.3758/s13423-018-01566-6 · Cited by 33
Decline Effects
Protzko, J., & Schooler, J. W. (2017) · Wiley eBooks · doi:10.1002/9781119095910.ch6 · Cited by 36
Context Moderates Affirmation Effects on the Ethnic Achievement Gap
Protzko, J., & Aronson, J. (2016) · Social Psychological and Personality Science · doi:10.1177/1948550616646426 · Cited by 39
The Brain Mechanisms Underlying the Cognitive Benefits of Bilingualism may be Extraordinarily Difficult to Discover
Paap, K. R., Sawi, O. M., Dalibar, C., Darrow, J., & Johnson, H. A. (2014) · AIMS neuroscience · doi:10.3934/neuroscience.2014.3.245 · Cited by 32
An Introduction to Replication Research in Gifted Education
Makel, M. C., & Plucker, J. A. (2015) · Gifted Child Quarterly · doi:10.1177/0016986215578747 · Cited by 31
Early-Career Researchers’ Perceptions of the Prevalence of Questionable Research Practices, Potential Causes, and Open Science
Stürmer, S., Oeberst, A., Trötschel, R., & Decker, O. (2017) · Social Psychology · doi:10.1027/1864-9335/a000324 · Cited by 32
Best (but oft forgotten) practices: sample size planning for powerful studies
Anderson, S. F. (2019) · American Journal of Clinical Nutrition · doi:10.1093/ajcn/nqz058 · Cited by 34
Decisive Evidence on a Smaller-Than-You-Think Phenomenon
Sirota, M., Juanchich, M., Kostopoulou, O., & Hanák, R. (2013) · Medical Decision Making · doi:10.1177/0272989x13514776 · Cited by 32
Targeting Next Generations to Change the Common Practice of Underpowered Research
Crutzen, R., & Peters, G.-J. (2017) · Frontiers in Psychology · doi:10.3389/fpsyg.2017.01184 · Cited by 39
Inflation von falsch-positiven Befunden in der psychologischen Forschung
Ulrich, R., Erdfelder, E., Deutsch, R., Strauss, B. M., Brüggemann, A., Hannover, B., Tuschen‐Caffier, B., Kirschbaum, C., Blickle, G., Møller, J., et al. (2016) · Psychologische Rundschau · doi:10.1026/0033-3042/a000296 · Cited by 35
Verify original results through reanalysis before replicating
Nuijten, M. B., Bakker, M., Maassen, E., & Wicherts, J. M. (2018) · Behavioral and Brain Sciences · doi:10.1017/s0140525x18000791 · Cited by 33
Inflectional morphology: evidence for an advantage of bilingualism in dyslexia
Vender, M., Hu, S., Mantione, F., Savazzi, S., Delfitto, D., & Melloni, C. (2018) · International Journal of Bilingual Education and Bilingualism · doi:10.1080/13670050.2018.1450355 · Cited by 31
Science revolves around the data
Wicherts, J. M. (2013) · Journal of Open Psychology Data · doi:10.5334/jopd.e1 · Cited by 31
Is something rotten in the state of Denmark? Cross-national evidence for widespread involvement but not systematic use of questionable research practices across all fields of research
Schneider, J. W., Allum, N., Andersen, J. P., Petersen, M. B., Madsen, E. B., Mejlgaard, N., & Zachariae, R. H. (2024) · PLoS ONE · doi:10.1371/journal.pone.0304342 · Cited by 34
Do Organizational Personification and Personality Matter? The Effect of Interaction and Conversational Tone on Relationship Quality in Social Media
Sung, K. H., & Kim, S. (2018) · International Journal of Business Communication · doi:10.1177/2329488418796631 · Cited by 30
Characterization of fracture behavior of human atherosclerotic fibrous caps using a miniature single edge notched tensile test
Davis, L. A., Stewart, S. E., Carsten, C. G., Snyder, B. A., Sutton, M. A., & Lessner, S. M. (2016) · Acta Biomaterialia · doi:10.1016/j.actbio.2016.07.027 · Cited by 30
The Affect Misattribution Procedure: In Search of Prejudice Effects
Teige-Mocigemba, S., Becker, M., Sherman, J., Reichardt, R., & Klauer, K. C. (2017) · Experimental Psychology · doi:10.1027/1618-3169/a000364 · Cited by 19
Moderation of classic social psychological effects by demographics in the U.S. adult population: New opportunities for theoretical advancement.
Yeager, D. S., Krosnick, J. A., Visser, P. S., Holbrook, A. L., & Tahk, A. M. (2019) · Journal of Personality and Social Psychology · doi:10.1037/pspa0000171 · Cited by 31
The Bilingual Advantage Debate
de Bruin, A., & Sala, S. D. (2019) · Wiley eBooks · doi:10.1002/9781119387725.ch35 · Cited by 32
A Bayesian Analysis of Evidence in Support of the Null Hypothesis in Gerontological Psychology (or Lack Thereof)
Brydges, C. R., & Bielak, A. A. M. (2019) · The Journals of Gerontology Series B · doi:10.1093/geronb/gbz033 · Cited by 32
Bridging Feminist Psychology and Open Science: Feminist Tools and Shared Values Inform Best Practices for Science Reform
Matsick, J. L., Kruk, M., Oswald, F., & Palmer, L. (2021) · Psychology of Women Quarterly · doi:10.1177/03616843211026564 · Cited by 31
Is “Culture” a Workable Concept for (Cross-)Cultural Psychology?
Poortinga, Y. H. (2015) · Online Readings in Psychology and Culture · doi:10.9707/2307-0919.1139 · Cited by 29
From Discovery to Justification: Outline of an Ideal Research Program in Empirical Psychology
Witte, E. H., & Zenker, F. (2017) · Frontiers in Psychology · doi:10.3389/fpsyg.2017.01847 · Cited by 31
Knowing how effective an intervention, treatment, or manipulation is and increasing replication rates: accuracy in parameter estimation as a partial solution to the replication crisis
Peters, G.-J., & Crutzen, R. (2020) · Psychology and Health · doi:10.1080/08870446.2020.1757098 · Cited by 32
Transplant Renal Vein Thrombosis.
Elzorkany, K. M. A., Bridson, J.-M., Sharma, A. K., & Halawa, A. M. (2017) · Experimental and Clinical Transplantation · doi:10.6002/ect.2016.0060 · Cited by 29
An aberrant abundance of Cronbach’s alpha values at .70Preprint
Hussey, I., Alsalti, T., Bosco, F. A., Elson, M., & Arslan, R. C. (2023) · PsyArXiv (OSF Preprints) · doi:10.31234/osf.io/dm8xn · Cited by 39
Meta-analyzing the multiverse: A peek under the hood of selective reporting.
Olsson-Collentine, A., van Aert, R. C. M., Bakker, M., & Wicherts, J. M. (2023) · Psychological Methods · doi:10.1037/met0000559 · Cited by 40
No Effect of Weight on Judgments of Importance in the Moral Domain and Evidence of Publication Bias from a Meta-Analysis
Rabelo, A. L. A., Keller, V. N., Pilati, R., & Wicherts, J. M. (2015) · PLoS ONE · doi:10.1371/journal.pone.0134808 · Cited by 32
Who Should Do Replication Labor?
Romero, F. (2018) · Advances in Methods and Practices in Psychological Science · doi:10.1177/2515245918803619 · Cited by 28
Effect size: a statistical basis for clinical practice
Valladares‐Neto, J. (2018) · Revista Odonto Ciência · doi:10.15448/1980-6523.2018.1.29437 · Cited by 29
Adapting open science and pre‐registration to longitudinal research
Petersen, I. T., Apfelbaum, K. S., & McMurray, B. (2022) · Infant and Child Development · doi:10.1002/icd.2315 · Cited by 28
Tighter nets for smaller fishes? Mapping the development of statistical practices in consumer research between 2008 and 2020
Krefeld-Schwalb, A., & Scheibehenne, B. (2022) · Marketing Letters · doi:10.1007/s11002-022-09662-3 · Cited by 28
Toward open science in PLS-SEM: Assessing the state of the art and future perspectives
Adler, S. J., Sharma, P. N., & Radomir, L. (2023) · Journal of Business Research · doi:10.1016/j.jbusres.2023.114291 · Cited by 40

Authors: Bakker, M., van Dijk, A., & Wicherts, J. M.
Affiliations: University of Amsterdam (NL)
University of Amsterdam, The Netherlands
Tilburg University (NL)
ORCID: Bakker, Marjan
Wicherts, Jelte M.
Venue: Perspectives on Psychological Science
Publisher: SAGE Publications
Published: 2012-11
Volume: 7
Issue: 6
Pages: 543-554
ISSN: 1745-6916 (ISSN-L)
1745-6924
DOI: 10.1177/1745691612459060
OpenAlex: W2157668596
Cited by: 800
References: 75
Citations by year: 2026: 14 · 2025: 32 · 2024: 39 · 2023: 50 · 2022: 52 · 2021: 100 · 2020: 63 · 2019: 82 · 2018: 87 · 2017: 88 · 2016: 56 · 2015: 59 · 2014: 52 · 2013: 19 · 2012: 5
FWCI: 49.25
Type: journal article
Language: en
Open access: bronze
doi.org
Licence: journals.sagepub.com/page/policies/text-and-data-mining-license (tdm)
Locations: Perspectives on Psychological Science (publishedVersion)
Research portal (Tilburg University) (publishedVersion)
PubMed (publishedVersion)
Data Archiving and Networked Services (DANS) (submittedVersion)
UvA-DARE (University of Amsterdam) (submittedVersion)
Funders: Nederlandse Organisatie voor Wetenschappelijk Onderzoek (016-125-385)
Topics: Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
Social Sciences › Psychology › Experimental and Cognitive Psychology › Psychology Research and Bibliometrics
Social Sciences › Education › Academic integrity and plagiarism
Keywords: null hypothesis significance testing, statistical power, publication bias, questionable research practices, false positives, meta-analysis, replication, sample size, p-values, research integrity, editorial policy

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

The Rules of the Game Called Psychological Science

Abstract

Overview

Integrity signals

Verification

Claims the paper makes

Claims other papers make about this one

Citations

Cites

Cited by

Related

Metadata

Record

About these checks