The DOI matches its Crossref record; 75 of the 75 references it cites were found in a registry or an index.
The Rules of the Game Called Psychological Science
Open accessAbstract
If science were a game, a dominant rule would probably be to collect results that are statistically significant. Several reviews of the psychological literature have shown that around 96% of papers involving the use of null hypothesis significance testing report significant outcomes for their main results but that the typical studies are insufficiently powerful for such a track record. We explain this paradox by showing that the use of several small underpowered samples often represents a more efficient research strategy (in terms of finding p < .05) than does the use of one larger (more powerful) sample. Publication bias and the most efficient strategy lead to inflated effects and high rates of false positives, especially when researchers also resorted to questionable research practices, such as adding participants after intermediate testing. We provide simulations that highlight the severity of such biases in meta-analyses. We consider 13 meta-analyses covering 281 primary studies in various fields of psychology and find indications of biases and/or an excess of significant results in seven. These results highlight the need for sufficiently powerful replications and changes in journal policies.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Also, the publication culture promotes novel, and often surprising, findings and this will motivate researchers to postulate improbable hypotheses (20).”
From Statistical and Methodological Considerations for the Interpretation of Intranasal Oxytocin Studies (2016) · cites this paper as [20]
In the citing paper
“From a statistical point of view this problem might not be any more serious for this field than other domains of psychology (20, 21, 31) or neuroscience in general (14).”
From Statistical and Methodological Considerations for the Interpretation of Intranasal Oxytocin Studies (2016) · cites this paper as [20]
In the citing paper
“However, one pervasive problem in the social sciences, including psychology, that threatens the accuracy of meta-analytic estimates is publication bias (Bakker et al., 2012; Franco et al., 2014, 2016)”
From Do Salient Social Norms Moderate Mortality Salience Effects? A (Challenging) Meta-Analysis of Terror Management Studies (2023) · cites this paper as Bakker et al., 2012
In the citing paper
“Given that publication bias and researcher degrees of freedom are understood to be pervasive problems in psychological science (Bakker et al., 2012; Franco et al., 2014, 2016; Hesse, 2018; John et al., 2012), unadjusted effect sizes are likely to be overestimated (Kvarven et al., 2020).”
From Do Salient Social Norms Moderate Mortality Salience Effects? A (Challenging) Meta-Analysis of Terror Management Studies (2023) · cites this paper as Bakker et al., 2012
Citations
Cites
Two new Later Stone Age sites from the Final Pleistocene in the Falémé Valley, eastern Senegal
Ndiaye, M., Lespez, L., Tribolo, C., Rassé, M., Hadjas, I., Davidoux, S., Huysecom, É., & Douze, K. (2024) · Monash University Research Portal (Monash University) · doi:10.3929/ethz-b-000667478 · Cited by 24,214
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · PLoS Medicine · doi:10.1371/journal.pmed.0020124 · Cited by 10,505
The file drawer problem and tolerance for null results.
Rosenthal, R. (1979) · Psychological Bulletin · doi:10.1037/0033-2909.86.3.638 · Cited by 9,068
Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011) · Psychological Science · doi:10.1177/0956797611417632 · Cited by 6,567
Journal of The American Statistical Association
Fienberg, S. E. (2004) · Encyclopedia of Statistical Sciences · doi:10.1002/0471667196.ess1317 · Cited by 6,289
Understanding and using the Implicit Association Test: I. An improved scoring algorithm.
Greenwald, A. G., Nosek, B. A., & Banaji, M. R. (2003) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.85.2.197 · Cited by 5,815
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · CHANCE · doi:10.1080/09332480.2005.10722754 · Cited by 4,702 · cited 3×
“Such practices lead to inflated ESs and increased false positive rates (Ioannidis, 2005, 2008b; Simmons et al., 2011).”
BackgroundResultCohen, J. (1994) · American Psychologist · doi:10.1037/0003-066x.49.12.997 · Cited by 4,003
Understanding and using the Implicit Association Test: III. Meta-analysis of predictive validity.
Greenwald, A. G., Poehlman, T. A., Uhlmann, E. L., & Banaji, M. R. (2009) · Journal of Personality and Social Psychology · doi:10.1037/a0015575 · Cited by 3,257
Belief in the law of small numbers.
Tversky, A., & Kahneman, D. (1971) · Psychological Bulletin · doi:10.1037/h0031322 · Cited by 3,246
Research works printed but not linked: 1
- 10.1037/0003-066X.56.2.128 · DOI · Paper page
Cited by
Sorted by citations
Items 701–799 of 799
Bangma, D. F. (2021) · University of Groningen research database (University of Groningen / Centre for Information Technology) · doi:10.33612/diss.190019115
Kazdin, A. E. (2021) · Cambridge University Press eBooks · doi:10.1017/9781108993647.001
Construct and Data-Evaluation Validity
Kazdin, A. E. (2021) · Cambridge University Press eBooks · doi:10.1017/9781108993647.004
Single-Case Experimental Research Designs
Kazdin, A. E. (2021) · Cambridge University Press eBooks · doi:10.1017/9781108993647.009
Questionable Research Practices and Determinants of Their FrequencyPreprint
Alsalti, T. (2021) · Thesis Commons (OSF Preprints) · doi:10.31237/osf.io/uqz2e
Kazdin, A. E. (2021) · Cambridge University Press eBooks · doi:10.1017/9781108993647.007
Cautions, Negative Effects, and Replication
Kazdin, A. E. (2021) · Cambridge University Press eBooks · doi:10.1017/9781108993647.016
Selecting Measures for Research
Kazdin, A. E. (2021) · Cambridge University Press eBooks · doi:10.1017/9781108993647.011
Kazdin, A. E. (2021) · Cambridge University Press eBooks · doi:10.1017/9781108993647.021
Kazdin, A. E. (2021) · Cambridge University Press eBooks · doi:10.1017/9781108993647.013
Kazdin, A. E. (2021) · Cambridge University Press eBooks · doi:10.1017/9781108993647.022
Ideas that Begin the Research Process
Kazdin, A. E. (2021) · Cambridge University Press eBooks · doi:10.1017/9781108993647.005
Kazdin, A. E. (2021) · Cambridge University Press eBooks · doi:10.1017/9781108993647.020
Kazdin, A. E. (2021) · Cambridge University Press eBooks · doi:10.1017/9781108993647.025
Presenting and Analyzing the Data
Kazdin, A. E. (2021) · Cambridge University Press eBooks · doi:10.1017/9781108993647.015
Kazdin, A. E. (2021) · Cambridge University Press eBooks · doi:10.1017/9781108993647.024
Kazdin, A. E. (2021) · Cambridge University Press eBooks · doi:10.1017/9781108993647.018
Questionable Research Practices and Determinants of Their FrequencyPreprint
Alsalti, T. (2021) · Thesis Commons (OSF Preprints) · doi:10.31237/osf.io/mjvwu
Kazdin, A. E. (2021) · Cambridge University Press eBooks · doi:10.1017/9781108993647.002
Forschungsbetrug und problematische Forschungspraktiken
Burghardt, J. (2021) · Springer eBooks · doi:10.1007/978-3-662-63893-4_3
Cristea, I. A., Ioannidis, J. P. A., & Georgescu, R. (2021) · PsyArXiv (OSF Preprints) · doi:10.31234/osf.io/6t7pw
Assessment: Types of Measures and Their Use
Kazdin, A. E. (2021) · Cambridge University Press eBooks · doi:10.1017/9781108993647.012
Experimental Research Using Group Designs
Kazdin, A. E. (2021) · Cambridge University Press eBooks · doi:10.1017/9781108993647.006
Ethical Issues and Guidelines for Research
Kazdin, A. E. (2021) · Cambridge University Press eBooks · doi:10.1017/9781108993647.017
Communication of Research Findings
Kazdin, A. E. (2021) · Cambridge University Press eBooks · doi:10.1017/9781108993647.019
Do red objects enhance sexual attractiveness? No evidence from two large replicationsPreprint
Pollet, T. V., Costello, J., Groeneboom, L., Peperkoorn, L. S., & Wu, J. (2018) · doi:10.31234/osf.io/3bfwh
Mellenbergh, G. J. (2019) · Springer eBooks · doi:10.1007/978-3-030-12272-0_20 · Cited by 1
Beek, T. F., Matzke, D., Pinto, Y., Rotteveel, M., Gierholz, A., Selker, R., Sasiadek, A., Steingroever, H., Jostmann, N. B., & Wagenmakers, E. (2017) · doi:10.31234/osf.io/2avns
Sullivan, R. M. (2018) · Faculty Opinions – Post-Publication Peer Review of the Biomedical Literature · doi:10.3410/f.727734580.793547482
Online data collection for developmental researchPreprint
Scott, K. M. (2018) · doi:10.31237/osf.io/qjs9y
To Have Practical Relevance, Scientific Claims Need to be AccuratePreprint
Zwaan, R. A., & Eerland, A. (2018) · doi:10.31234/osf.io/kqes9
vanAert_Niemeyer21Preprint
van Aert, R. C. M., & Niemeyer, H. (2020) · MetArXiv (OSF Preprints) · doi:10.31222/osf.io/3rdys
Wege aus der Vertrauenskrise – Individuelle Schritte hin zu verlässlicher und offener ForschungPreprint
Utesch, K., Schönbrodt, F. D., Utesch, T., Geukes, S., & Back, M. D. (2016) · OSF Preprints (OSF Preprints) · doi:10.31219/osf.io/sn5bq
Training Psychologists for the Post-Truth EraPreprint
Chandler, P. (2018) · doi:10.31219/osf.io/ws5vd
Wicherts, J. M. (2017) · Springer eBooks · doi:10.1007/978-3-319-28099-8_1307-1
Mellenbergh, G. J. (2019) · Springer eBooks · doi:10.1007/978-3-030-12272-0_19
JSE 36:1 Spring 2022 Whole Issue PDF
Erickson, K. E. (2022) · Journal of Scientific Exploration · doi:10.31275/20222613
Psychology and the Paranormal by David F. Marks
Kennedy, J. E. (2022) · Journal of Scientific Exploration · doi:10.31275/20222403
Interim Design Analysis Using Bayes Factor ForecastsPreprint
Stefan, A. M., Gronau, Q. F., & Wagenmakers, E. (2022) · doi:10.31234/osf.io/9sazk
Ethical Considerations in Psychology Research
Oates, J. (2019) · Springer eBooks · doi:10.1007/978-3-319-76040-7_35-1
Aarts, A. A. (2022) · SSRN Electronic Journal · doi:10.2139/ssrn.4176916
Cohen’s Convention, the Seriousness of Errors, and the Body of Knowledge in Behavioral SciencePreprint
Arslan, A., & Zenker, F. (2022) · doi:10.31234/osf.io/mjb2x
[PhD Dissertation] Good Research PracticesPreprint
Sarafoglou, A. (2022) · PsyArXiv (OSF Preprints) · doi:10.31234/osf.io/7aj5h
“Voodoo” Science in Neuroimaging: How a Controversy Transformed into a Crisis
Sauvayre, R. (2022) · Social Sciences · doi:10.3390/socsci12010015
Assessing QRPs and RRPs in Psychology Masters’ ThesesPreprint
Augusteijn, H. E. M., van Assen, M. A. L. M., Sijtsma, K., & Wicherts, J. M. (2023) · doi:10.31219/osf.io/cwyk8
Gieseler, K., Loschelder, D. D., & Friese, M. (2023) · Springer eBooks · doi:10.1007/978-3-031-17529-9_1
Is the future on the pinky finger of your right hand?
Ikeda, S. (2023) · The Japanese journal of psychology · doi:10.4992/jjpsy.94.22317 · Cited by 1
Clark, K., Lindsay, A., Gilligan-Lee, K. A., de‐Wit, L., Scott‐Samuel, N. E., Azevedo, F., & Pennington, C. R. (2023) · doi:10.31234/osf.io/2qx6m
Terror Management Theory and the Existential Psychological Function of Heroes
Schindler, S. (2023) · Springer eBooks · doi:10.1007/978-3-031-17125-3_142-1
Toward open science in PLS-SEM: Assessing the state of the art and future perspectivesPreprint
Adler, S. J., Sharma, P. N., & Radomir, L. (2023) · doi:10.31219/osf.io/sbpe9
Public events knowledge in an age-heterogeneous sample: Reminiscence bump or bummer?
Achaa-Amankwaa, P., Steger, D., Wilhelm, O., & Schroeders, U. (2023) · Psychology and Aging · doi:10.1037/pag0000786
On the Credibility of Statistical Inference Within the Neyman-Pearson Hypothesis Testing FrameworkPreprint
Wong, T. K., & Mulder, J. (2023) · doi:10.31234/osf.io/kh3mu · Cited by 1
Two Voices on the Credibility Crisis in Psychology
Białek, A., & Wolski, P. (2023) · Przegląd Psychologiczny · doi:10.31648/przegldpsychologiczny.9679
Integrating Front-of-House with Behind-the-Scenes Practice in Contemporary Art Conservation
Spangler-Bickell, C. (2023) · Studies in art, heritage, law and the market · doi:10.1007/978-3-031-42357-4_17
McLean, K. C. (2024) · Oxford University Press eBooks · doi:10.1093/oso/9780197764640.003.0007
Our Scientific Responsibility for Change
McLean, K. C. (2024) · Oxford University Press eBooks · doi:10.1093/oso/9780197764640.003.0008
Transgressions as an Opportunity for Change?
McLean, K. C. (2024) · Oxford University Press eBooks · doi:10.1093/oso/9780197764640.003.0006
McLean, K. C. (2024) · Oxford University Press eBooks · doi:10.1093/oso/9780197764640.003.0001
Evidence for Change From the Field of Personality Development
McLean, K. C. (2024) · Oxford University Press eBooks · doi:10.1093/oso/9780197764640.003.0004
Evidence for Change in Narrative Identity
McLean, K. C. (2024) · Oxford University Press eBooks · doi:10.1093/oso/9780197764640.003.0005 · Cited by 1
McLean, K. C. (2024) · Oxford University Press eBooks · doi:10.1093/oso/9780197764640.003.0002
McLean, K. C. (2024) · Oxford University Press eBooks · doi:10.1093/oso/9780197764640.002.0003
McLean, K. C. (2024) · Oxford University Press eBooks · doi:10.1093/oso/9780197764640.002.0004
How the Credibility Revolution Created a Paradigm ShiftPreprint
Forozish, A. O. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.4744474 · Cited by 2
Persönlichkeit in Alltag, Wissenschaft und Praxis
Asendorpf, J. B. (2024) · Springer eBooks · doi:10.1007/978-3-662-68451-1_1
A Methodological Evaluation of Meta-Analyses in tDCS - Motor Learning ResearchPreprint
Alsalti, T., Hussey, I., Elson, M., Krause, R. W., & Pohl, S. (2024) · medRxiv · doi:10.1101/2024.07.26.24311068
Terror Management Theory and the Existential Psychological Function of Heroes
Schindler, S. (2024) · Springer eBooks · doi:10.1007/978-3-031-48129-1_142
Collecting Digital Footprints in the Wild
Kosiński, M. (2024) · Cambridge University Press eBooks · doi:10.1017/9781009170123.016
Mitä toistettavuuskriisistä pitäisi ajatella? Kymmenen suomalaistutkijaa sähköpostihaastattelussa
Heino, M., Ilmarinen, V., Leikas, S., Lipsanen, J. O., Lönnqvist, J., Olkkonen, M., Palosaari, E. J., Renvik, T. A., Saurio, K. J., & Silfver-Kuhalampi, M. (2025) · Psykologia · doi:10.62443/psykologia.v54i6.143375
Cognitive bias and human factors in statistics and data in healthcare
Lockett, A. (2025) · Medicine · doi:10.1016/j.mpmed.2025.03.008
Challenges and actions towards a more comprehensive developmental science
Tantaros, S. (2025) · European Journal of Developmental Psychology · doi:10.1080/17405629.2025.2498756
Why psychopathy might be present and prosperous in present-day psychological sciencePreprint
Aarts, A. A. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5166553
Exploratory Simulations for Confirmatory Analysis: English Auditory Word Learning
Neergaard, K. D. (2025) · Institute for the Future of Education Conference (IFE) · doi:10.1109/ife63672.2025.11025014
Tomat, N., Kindermann, N., & Ohlhorst, J. (2025) · Analysis · doi:10.1093/analys/anaf032
Statistics and Persuasion in Medicine
Osimani, B. (2025) · Oxford University Press eBooks · doi:10.1093/oxfordhb/9780197625835.013.0013
Emons, W. H. M., Sijtsma, K., & Bouter, L. M. (2025) · Accountability in Research · doi:10.1080/08989621.2025.2575442
Losee, J. E., Higginbotham, G. D., Pogge, G. C., Kerner, L., & Shepperd, J. A. (2025) · Political Psychology · doi:10.1111/pops.70091
Kovač, N., Simeunović, M., Farahani, H., Bezdan, T., & Watson, P. (2025) · Elsevier eBooks · doi:10.1016/b978-0-443-30248-0.00014-0
No room at the inn? The case for dedicated replication journals
Reed, W. R., Röseler, L., Saam, M., & Wallrich, L. (2025) · Journal of Behavioral and Experimental Economics · doi:10.1016/j.socec.2025.102502
50-Year review of the Door-in-the-Face message strategy
Feeley, T. H. (2025) · Communication Monographs · doi:10.1080/03637751.2025.2594987
Benefits of Open and High-Powered Research Outweigh CostsPreprint
LeBel, E. P., Campbell, L., & Loving, T. J. (2016) · SSRN Electronic Journal · doi:10.2139/ssrn.2616384
Researcher Degrees of Freedom in the Psychology of ReligionPreprint
Bartlett, J. E., Uzdavines, A., Coleman, T. J., III, Messick, K. J., & Charles, S. J. (2018) · OSF Preprints (OSF Preprints)
Mosen, J., Voracek, M., Kirchler, E., & Olsen, J. (2019) · OSF Preprints (OSF Preprints)
Individual and collective epistemic virtue in science
Tunç, D. U., & Pritchard, D. (2026) · Synthese · doi:10.1007/s11229-025-05396-7
Assessing Questionable and Responsible Research Practices in Psychology Master’s Theses
Augusteijn, H. E. M., van Assen, M. A. L. M., Sijtsma, K., & Wicherts, J. M. (2026) · Behavioral Sciences · doi:10.3390/bs16010110
Reproducibility of Individual Effect Sizes in Meta-Analyses in PsychologyPreprint
Wicherts, J. M., Maassen, E., van Assen, M. A. L. M., Nuijten, M. B., & Olsson-Collentine, A. (2016) · doi:10.17605/osf.io/7nsmd
The Replication Crisis in Psychology: Causes, Consequences, and Strategies for Overcoming It
Гуляс, І. А. (2026) · PSIHOLOGÌÂ Ì SUSPÌLʹSTVO · doi:10.35774/pis2026.01.083
Psychology’s RenaissancePreprint
Simonsohn, U., Nelson, L. D., & Simmons, J. P. (2022) · OSF Preprints (OSF Preprints)
Ordinal Effects of Ostracism: A Meta-Analysis of Cyberball Studies.Preprint
Williams, K. D., Hartgerink, C., van Beest, I., & Wicherts, J. M. (2015) · OSF Preprints (OSF Preprints)
Verschuère, B., Kleinberg, B., & Warmelink, L. (2023) · OSF Preprints (OSF Preprints)
Flexible Yet Fair: Blinding Analyses in Experimental PsychologyPreprint
Wagenmakers, E., Sarafoglou, A., & Dutilh, G. (2019) · OSF Preprints (OSF Preprints)
Stout, W. A., & Brandone, A. C. (2022) · OSF Preprints (OSF Preprints)
975 Statistical power and power analysis in quantitative communication science
Sun, Y., & Qian, S. (2026) · De Gruyter eBooks · doi:10.1515/9783111087139-005
Ätiologie und Pathogenese psychischer Störungen
Hengartner, M. P. (2026) · Angewandte Psychologie Kompakt · doi:10.1007/978-3-662-72845-1_4
Freuli, F., Noventa, S., & Lombardi, L. (2026) · Frontiers in Psychology · doi:10.3389/fpsyg.2026.1769309
Augusteijn, H. E. M., Wicherts, J. M., Sijtsma, K., & van Assen, M. A. L. M. (2026) · PLoS ONE · doi:10.1371/journal.pone.0355938 · Cited by 4
Metadata
- Authors
- Bakker, M., van Dijk, A., & Wicherts, J. M.
- Affiliations
- University of Amsterdam (NL)
- University of Amsterdam, The Netherlands
- Tilburg University (NL)
- Venue
- Perspectives on Psychological Science
- Publisher
- SAGE Publications
- Published
- 2012-11
- Volume
- 7
- Issue
- 6
- Pages
- 543-554
- ISSN
- 1745-6916 (ISSN-L)
- 1745-6924
- OpenAlex
- W2157668596
- Cited by
- 800
- References
- 75
- Citations by year
- 2026: 14 · 2025: 32 · 2024: 39 · 2023: 50 · 2022: 52 · 2021: 100 · 2020: 63 · 2019: 82 · 2018: 87 · 2017: 88 · 2016: 56 · 2015: 59 · 2014: 52 · 2013: 19 · 2012: 5
- FWCI
- 49.25
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- Perspectives on Psychological Science (publishedVersion)
- Research portal (Tilburg University) (publishedVersion)
- PubMed (publishedVersion)
- Data Archiving and Networked Services (DANS) (submittedVersion)
- UvA-DARE (University of Amsterdam) (submittedVersion)
- Funders
- Nederlandse Organisatie voor Wetenschappelijk Onderzoek (016-125-385)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Psychology Research and Bibliometrics
- Social Sciences › Education › Academic integrity and plagiarism
- Keywords
- null hypothesis significance testing, statistical power, publication bias, questionable research practices, false positives, meta-analysis, replication, sample size, p-values, research integrity, editorial policy
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
