The DOI matches its Crossref record; 75 of the 75 references it cites were found in a registry or an index.
The Rules of the Game Called Psychological Science
Open accessAbstract
If science were a game, a dominant rule would probably be to collect results that are statistically significant. Several reviews of the psychological literature have shown that around 96% of papers involving the use of null hypothesis significance testing report significant outcomes for their main results but that the typical studies are insufficiently powerful for such a track record. We explain this paradox by showing that the use of several small underpowered samples often represents a more efficient research strategy (in terms of finding p < .05) than does the use of one larger (more powerful) sample. Publication bias and the most efficient strategy lead to inflated effects and high rates of false positives, especially when researchers also resorted to questionable research practices, such as adding participants after intermediate testing. We provide simulations that highlight the severity of such biases in meta-analyses. We consider 13 meta-analyses covering 281 primary studies in various fields of psychology and find indications of biases and/or an excess of significant results in seven. These results highlight the need for sufficiently powerful replications and changes in journal policies.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Also, the publication culture promotes novel, and often surprising, findings and this will motivate researchers to postulate improbable hypotheses (20).”
From Statistical and Methodological Considerations for the Interpretation of Intranasal Oxytocin Studies (2016) · cites this paper as [20]
In the citing paper
“From a statistical point of view this problem might not be any more serious for this field than other domains of psychology (20, 21, 31) or neuroscience in general (14).”
From Statistical and Methodological Considerations for the Interpretation of Intranasal Oxytocin Studies (2016) · cites this paper as [20]
In the citing paper
“However, one pervasive problem in the social sciences, including psychology, that threatens the accuracy of meta-analytic estimates is publication bias (Bakker et al., 2012; Franco et al., 2014, 2016)”
From Do Salient Social Norms Moderate Mortality Salience Effects? A (Challenging) Meta-Analysis of Terror Management Studies (2023) · cites this paper as Bakker et al., 2012
In the citing paper
“Given that publication bias and researcher degrees of freedom are understood to be pervasive problems in psychological science (Bakker et al., 2012; Franco et al., 2014, 2016; Hesse, 2018; John et al., 2012), unadjusted effect sizes are likely to be overestimated (Kvarven et al., 2020).”
From Do Salient Social Norms Moderate Mortality Salience Effects? A (Challenging) Meta-Analysis of Terror Management Studies (2023) · cites this paper as Bakker et al., 2012
Citations
Cites
Two new Later Stone Age sites from the Final Pleistocene in the Falémé Valley, eastern Senegal
Ndiaye, M., Lespez, L., Tribolo, C., Rassé, M., Hadjas, I., Davidoux, S., Huysecom, É., & Douze, K. (2024) · Monash University Research Portal (Monash University) · doi:10.3929/ethz-b-000667478 · Cited by 24,214
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · PLoS Medicine · doi:10.1371/journal.pmed.0020124 · Cited by 10,505
The file drawer problem and tolerance for null results.
Rosenthal, R. (1979) · Psychological Bulletin · doi:10.1037/0033-2909.86.3.638 · Cited by 9,068
Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011) · Psychological Science · doi:10.1177/0956797611417632 · Cited by 6,567
Journal of The American Statistical Association
Fienberg, S. E. (2004) · Encyclopedia of Statistical Sciences · doi:10.1002/0471667196.ess1317 · Cited by 6,289
Understanding and using the Implicit Association Test: I. An improved scoring algorithm.
Greenwald, A. G., Nosek, B. A., & Banaji, M. R. (2003) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.85.2.197 · Cited by 5,815
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · CHANCE · doi:10.1080/09332480.2005.10722754 · Cited by 4,702 · cited 3×
“Such practices lead to inflated ESs and increased false positive rates (Ioannidis, 2005, 2008b; Simmons et al., 2011).”
BackgroundResultCohen, J. (1994) · American Psychologist · doi:10.1037/0003-066x.49.12.997 · Cited by 4,003
Understanding and using the Implicit Association Test: III. Meta-analysis of predictive validity.
Greenwald, A. G., Poehlman, T. A., Uhlmann, E. L., & Banaji, M. R. (2009) · Journal of Personality and Social Psychology · doi:10.1037/a0015575 · Cited by 3,257
Belief in the law of small numbers.
Tversky, A., & Kahneman, D. (1971) · Psychological Bulletin · doi:10.1037/h0031322 · Cited by 3,246
Raise standards for preclinical cancer research
Begley, C. G., & Ellis, L. M. (2012) · Nature · doi:10.1038/483531a · Cited by 2,979 · cited 1×
“Publication bias can have dire consequences, as illustrated recently by clear failures to replicate medical findings (Begley & Ellis, 2012; Prinz, Schlange & Asadullah, 2011).”
BackgroundA practical solution to the pervasive problems ofp values
Wagenmakers, E.-J. (2007) · Psychonomic Bulletin & Review · doi:10.3758/bf03194105 · Cited by 2,490
Publication Bias in Meta‐Analysis: Prevention, Assessment and Adjustments
(2005) · doi:10.1002/0470870168 · Cited by 2,368
Measuring the Prevalence of Questionable Research Practices With Incentives for Truth Telling
John, L. K., Loewenstein, G., & Prelec, D. (2012) · Psychological Science · doi:10.1177/0956797611430953 · Cited by 2,239
Meehl, P. E. (1978) · Journal of Consulting and Clinical Psychology · doi:10.1037/0022-006x.46.4.806 · Cited by 2,061
Lipsey, M. W., & Wilson, D. B. (1993) · American Psychologist · doi:10.1037/0003-066x.48.12.1181 · Cited by 2,033
Believe it or not: how much can we rely on published data on potential drug targets?
Prinz, F., Schlange, T., & Asadullah, K. (2011) · Nature Reviews Drug Discovery · doi:10.1038/nrd3439-c1 · Cited by 1,888
Things I have learned (so far).
Cohen, J. (1990) · American Psychologist · doi:10.1037/0003-066x.45.12.1304 · Cited by 1,781
One Hundred Years of Social Psychology Quantitatively Described
Richard, F. D., Bond, C. F., & Stokes-Zoota, J. J. (2003) · Review of General Psychology · doi:10.1037/1089-2680.7.4.331 · Cited by 1,558
Why Most Discovered True Associations Are Inflated
Ioannidis, J. P. A. (2008) · Epidemiology · doi:10.1097/ede.0b013e31818131e7 · Cited by 1,495
Summing up: The Science of Reviewing Research
Wilkinson, I., Light, R. J., & Pillemer, D. B. (1987) · Journal of Educational Statistics · doi:10.2307/1164691 · Cited by 1,486
Puzzlingly High Correlations in fMRI Studies of Emotion, Personality, and Social Cognition
Vul, E., Harris, C., Winkielman, P., & Pashler, H. (2009) · Perspectives on Psychological Science · doi:10.1111/j.1745-6924.2009.01125.x · Cited by 1,475
Psychological testing and psychological assessment: A review of evidence and issues.
Meyer, G. J., Finn, S. E., Eyde, L. D., Kay, G. G., Moreland, K. L., Dies, R. R., Eisman, E. J., Kubiszyn, T. W., & Reed, G. M. (2001) · American Psychologist · doi:10.1037//0003-066x.56.2.128 · Cited by 1,469
Light, R. J., & Pillemer, D. B. (1984) · Harvard University Press eBooks · doi:10.4159/9780674040243 · Cited by 1,422
Does discovery-based instruction enhance learning?
Alfieri, L., Brooks, P. J., Aldrich, N. J., & Tenenbaum, H. R. (2011) · Journal of Educational Psychology · doi:10.1037/a0021017 · Cited by 1,338
To read or not to read: a meta-analysis of print exposure from infancy to early adulthood
Mol, S. E., & Bus, A. G. (2011) · Psychological Bulletin · doi:10.1037/a0021890 · Cited by 1,299
Null hypothesis significance testing: a review of an old and continuing controversy
Nickerson, R. S. (2000) · Psychological Methods · doi:10.1037/1082-989x.5.2.241 · Cited by 1,090
Beyond significance testing: Reforming data analysis methods in behavioral research.
Kline, R. B. (2004) · doi:10.1037/10693-000 · Cited by 1,034
Publication prejudices: An experimental study of confirmatory bias in the peer review system
Mahoney, M. J. (1977) · Cognitive Therapy and Research · doi:10.1007/bf01173636 · Cited by 1,013
Wagenmakers, E.-J., Wetzels, R., Borsboom, D., & van der Maas, H. L. J. (2011) · Journal of Personality and Social Psychology · doi:10.1037/a0022790 · Cited by 977
Sterling, T. D. (1959) · Journal of the American Statistical Association · doi:10.1080/01621459.1959.10501497 · Cited by 964 · cited 3×
“It has long been argued that the combined outcomes of NHST in the scientific literature are too good to be true (Fanelli, 2010; Fiedler, 2011; Ioannidis, 2008a; Sterling, 1959; Vul, Harris, Winkielman, & Pashler, 2009).”
BackgroundStatistical Evidence in Experimental Psychology: An Empirical Comparison Using 855 t Tests
Wetzels, R., Matzke, D., Lee, M. D., Rouder, J. N., Iverson, G. J., & Wagenmakers, E.-J. (2011) · Perspectives on Psychological Science · doi:10.1177/1745691611406923 · Cited by 945
A meta-analysis of the effect of cognitive bias modification on anxiety and depression
Hallion, L. S., & Ruscio, A. M. (2011) · Psychological Bulletin · doi:10.1037/a0024355 · Cited by 891
Gigerenzer, G. (2004) · The Journal of Socio-Economics · doi:10.1016/j.socec.2004.09.033 · Cited by 890
Regression Methods to Detect Publication and Other Bias in Meta‐Analysis
Sterne, J. A. C., & Egger, M. (2005) · Publication Bias in Meta‐Analysis · doi:10.1002/0470870168.ch6 · Cited by 826 · cited 2×
“4), although it is important to note that they are sensitive to actual heterogeneity of the underlying ESs (Ioannidis & Trikalinos, 2007; Sterne & Egger, 2005).”
BackgroundMethodInfluentialAn exploratory test for an excess of significant findings
Ioannidis, J. P., & Trikalinos, T. A. (2007) · Clinical Trials · doi:10.1177/1740774507079441 · Cited by 751
The fallacy of the null-hypothesis significance test.
Rozeboom, W. W. (1960) · Psychological Bulletin · doi:10.1037/h0042040 · Cited by 740
“Positive” results increase down the Hierarchy of the Sciences
Fanelli, D. (2010) · PLoS ONE · doi:10.1371/journal.pone.0010068 · Cited by 734
Culturally adapted psychotherapy and the legitimacy of myth: A direct-comparison meta-analysis.
Benish, S. G., Quintana, S., & Wampold, B. E. (2011) · Journal of Counseling Psychology · doi:10.1037/a0023626 · Cited by 661
Maxwell, S. E. (2004) · Psychological Methods · doi:10.1037/1082-989x.9.2.147 · Cited by 655
Berry, C. M., Carpenter, N. C., & Barratt, C. L. (2011) · Journal of Applied Psychology · doi:10.1037/a0026739 · Cited by 623
Ferguson, C. J., & Brannick, M. Τ. (2011) · Psychological Methods · doi:10.1037/a0024445 · Cited by 512
Research in the Psychological Laboratory
Anderson, C. A., Lindsay, J. J., & Bushman, B. J. (1999) · Current Directions in Psychological Science · doi:10.1111/1467-8721.00002 · Cited by 500
Bayesian Assessment of Null Values Via Parameter Estimation and Model Comparison
Kruschke, J. K. (2011) · Perspectives on Psychological Science · doi:10.1177/1745691611406925 · Cited by 488
Sterling, T. D., Rosenbaum, W. L., & Weinkam, J. J. (1995) · The American Statistician · doi:10.1080/00031305.1995.10476125 · Cited by 453
Wicherts, J. M., Bakker, M., & Molenaar, D. (2011) · PLoS ONE · doi:10.1371/journal.pone.0026828 · Cited by 424
Self-citationScientist as Subject: The Psychological Imperative.
Lewis, L. S., & Mahoney, M. J. (1978) · Contemporary Sociology A Journal of Reviews · doi:10.2307/2064480 · Cited by 415
A Meta-Analysis of Infant Habituation and Recognition Memory Performance as Predictors of Later IQ
McCall, R. B., & Carriger, M. S. (1993) · Child Development · doi:10.1111/j.1467-8624.1993.tb02895.x · Cited by 400
The (mis)reporting of statistical results in psychology journals
Bakker, M., & Wicherts, J. M. (2011) · Behavior Research Methods · doi:10.3758/s13428-011-0089-5 · Cited by 356
Self-citationLucassen, N., Tharner, A., Van IJzendoorn, M. H., Bakermans-Kranenburg, M. J., Volling, B. L., Verhulst, F. C., Lambregtse-Van den Berg, M. P., & Tiemeier, H. (2011) · Journal of Family Psychology · doi:10.1037/a0025855 · Cited by 296
A Meta-Analysis of Infant Habituation and Recognition Memory Performance as Predictors of Later IQ
McCall, R. B., & Carriger, M. S. (1993) · Child Development · doi:10.2307/1131437 · Cited by 256
Voodoo Correlations Are Everywhere—Not Only in Neuroscience
Fiedler, K. (2011) · Perspectives on Psychological Science · doi:10.1177/1745691611400237 · Cited by 248
Too good to be true: Publication bias in two prominent studies from experimental psychology
Francis, G. (2012) · Psychonomic Bulletin & Review · doi:10.3758/s13423-012-0227-9 · Cited by 221
Red, rank, and romance in women viewing men.
Elliot, A. J., Kayser, D. N., Greitemeyer, T., Lichtenfeld, S., Gramzow, R. H., Maier, M., & Liu, H. (2010) · Journal of Experimental Psychology General · doi:10.1037/a0019689 · Cited by 199
Levine, T. R., Asada, K. J. K., & Carpenter, C. (2009) · Communication Monographs · doi:10.1080/03637750903074685 · Cited by 170
Sample size in psychological research over the past 30 years
Marszalek, J. M., Barber, C., Kohlhart, J., & Cooper, B. H. (2011) · Perceptual and Motor Skills · doi:10.2466/03.11.pms.112.2.331-348 · Cited by 167
Effect of positive findings on submission and acceptance rates: A note on meta-analysis bias.
Coursol, A., & Wagner, E. E. (1986) · Professional Psychology Research and Practice · doi:10.1037/0735-7028.17.2.136 · Cited by 165
A two-dimensional approach to relationship conflict: meta-analytic findings.
Woodin, E. M. (2011) · Journal of family psychology · doi:10.1037/a0023791 · Cited by 156
Card, N. A., Bosch, L. A., Casper, D. M., Wiggs, C. B., Hawkins, S. A., Schlomer, G. L., & Borden, L. M. (2011) · Journal of Family Psychology · doi:10.1037/a0024395 · Cited by 144
Finding the missing science: The fate of studies submitted for review by a human subjects committee.
Cooper, H., DeNeve, K., & Charlton, K. (1997) · Psychological Methods · doi:10.1037/1082-989x.2.4.447 · Cited by 144
Kraemer, H. C., Gardner, C., Brooks, J. O., & Yesavage, J. A. (1998) · Psychological Methods · doi:10.1037/1082-989x.3.1.23 · Cited by 141
Probability as certainty: Dichotomous thinking and the misuse ofp values
Hoekstra, R., Finch, S., Kiers, H., & Johnson, A. M. (2006) · Psychonomic Bulletin & Review · doi:10.3758/bf03213921 · Cited by 113
Interpretation of significance levels and effect sizes by psychological researchers.
Nelson, N., Rosenthal, R., & Rosnow, R. L. (1986) · American Psychologist · doi:10.1037/0003-066x.41.11.1299 · Cited by 113
Menie, M. A. W. O. (2011) · Review of General Psychology · doi:10.1037/a0024348 · Cited by 92
Green, D., & Rosenfeld, B. D. (2011) · Psychological Assessment · doi:10.1037/a0021149 · Cited by 79
Farber, B. A., & Doolin, E. M. (2011) · Psychotherapy · doi:10.1037/a0022141 · Cited by 77
Short, Sweet, and Problematic? The Rise of the Short Report in Psychological Science
Ledgerwood, A., & Sherman, J. W. (2012) · Perspectives on Psychological Science · doi:10.1177/1745691611427304 · Cited by 72
How Big Are Nonverbal Sex Differences? The Case of Smiling and Sensitivity to Nonverbal Cues
Hall, J. A. (1998) · Psychology Press eBooks · doi:10.4324/9781315805870-7 · Cited by 69
Effect of Formal Statistical Significance on the Credibility of Observational Associations
Ioannidis, J. P. A. (2008) · American Journal of Epidemiology · doi:10.1093/aje/kwn156 · Cited by 66
Bones, A. K. (2012) · Perspectives on Psychological Science · doi:10.1177/1745691612441216 · Cited by 51
Shadish, W. R., Doherty, M., & Montgomery, L. M. (1989) · Clinical Psychology Review · doi:10.1016/0272-7358(89)90013-5 · Cited by 45
The Same Old New Look: Publication Bias in a Study of Wishful Seeing
Francis, G. (2012) · i-Perception · doi:10.1068/i0519ic · Cited by 36
Sterba, S. K. (2006) · Ethics & Behavior · doi:10.1207/s15327019eb1604_3 · Cited by 15
Bloch, H. (1994) · Review of Industrial Organization · doi:10.1007/bf01024220 · Cited by 12
Applications of meta-analysis: 1987-1992
Tett, R. P., Meyer, J., & Roese, N. J. (1994) · Cited by 8
Research works printed but not linked: 1
- 10.1037/0003-066X.56.2.128 · DOI · Paper page
Cited by
Sorted by citations
Lakens, D. (2013) · Frontiers in Psychology · doi:10.3389/fpsyg.2013.00863 · Cited by 10,303
The preregistration revolution
Nosek, B. A., Ebersole, C. R., DeHaven, A. C., & Mellor, D. T. (2018) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1708274114 · Cited by 1,998
Choosing Prediction Over Explanation in Psychology: Lessons From Machine Learning
Yarkoni, T., & Westfall, J. (2017) · Perspectives on Psychological Science · doi:10.1177/1745691617693393 · Cited by 2,088
Increasing Transparency Through a Multiverse Analysis
Steegen, S., Tuerlinckx, F., Gelman, A., & Vanpaemel, W. (2016) · Perspectives on Psychological Science · doi:10.1177/1745691616658637 · Cited by 1,512
Sanders, M. R., Kirby, J. N., Tellegen, C. L., & Day, J. J. (2014) · Clinical Psychology Review · doi:10.1016/j.cpr.2014.04.003 · Cited by 1,077
Metadata
- Authors
- Bakker, M., van Dijk, A., & Wicherts, J. M.
- Affiliations
- University of Amsterdam (NL)
- University of Amsterdam, The Netherlands
- Tilburg University (NL)
- Venue
- Perspectives on Psychological Science
- Publisher
- SAGE Publications
- Published
- 2012-11
- Volume
- 7
- Issue
- 6
- Pages
- 543-554
- ISSN
- 1745-6916 (ISSN-L)
- 1745-6924
- OpenAlex
- W2157668596
- Cited by
- 800
- References
- 75
- Citations by year
- 2026: 14 · 2025: 32 · 2024: 39 · 2023: 50 · 2022: 52 · 2021: 100 · 2020: 63 · 2019: 82 · 2018: 87 · 2017: 88 · 2016: 56 · 2015: 59 · 2014: 52 · 2013: 19 · 2012: 5
- FWCI
- 49.25
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- Perspectives on Psychological Science (publishedVersion)
- Research portal (Tilburg University) (publishedVersion)
- PubMed (publishedVersion)
- Data Archiving and Networked Services (DANS) (submittedVersion)
- UvA-DARE (University of Amsterdam) (submittedVersion)
- Funders
- Nederlandse Organisatie voor Wetenschappelijk Onderzoek (016-125-385)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Psychology Research and Bibliometrics
- Social Sciences › Education › Academic integrity and plagiarism
- Keywords
- null hypothesis significance testing, statistical power, publication bias, questionable research practices, false positives, meta-analysis, replication, sample size, p-values, research integrity, editorial policy
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
