The rules of the game called psychological science

Bakker, Marjan; van Dijk, Annette; Wicherts, Jelte M.

doi:10.1177/1745691612459060

Cited by 800Meta-analysis and systematic reviewsPsychology Research and BibliometricsAcademic integrity and plagiarism

The rules of the game called psychological science

Open access

Bakker, Marjan, van Dijk, Annette, Wicherts, Jelte M.·2012·Perspectives on Psychological Science

DOI 10.1177/1745691612459060

Open this paper in CoChat — free account Open PDF

Abstract

If science were a game, a dominant rule would probably be to collect results that are statistically significant. Several reviews of the psychological literature have shown that around 96% of papers involving the use of null hypothesis significance testing report significant outcomes for their main results but that the typical studies are insufficiently powerful for such a track record. We explain this paradox by showing that the use of several small underpowered samples often represents a more efficient research strategy (in terms of finding p < .05) than does the use of one larger (more powerful) sample. Publication bias and the most efficient strategy lead to inflated effects and high rates of false positives, especially when researchers also resorted to questionable research practices, such as adding participants after intermediate testing. We provide simulations that highlight the severity of such biases in meta-analyses. We consider 13 meta-analyses covering 281 primary studies in various fields of psychology and find indications of biases and/or an excess of significant results in seven. These results highlight the need for sufficiently powerful replications and changes in journal policies.

Integrity signals

Registry

Crossref · 10.1177/1745691612459060

Cites

75 of 75 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Claims the paper makes

We couldn’t retrieve this paper’s full text, so its claims and self-check weren’t run. The registry and citation checks still apply.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

The publication culture promotes novel, and often surprising, findings and this motivates researchers to postulate improbable hypotheses.
“Also, the publication culture promotes novel, and often surprising, findings and this will motivate researchers to postulate improbable hypotheses (20).”
From Statistical and Methodological Considerations for the Interpretation of Intranasal Oxytocin Studies (2016) · cites this paper as [20]
From a statistical point of view this problem might not be any more serious for this field than other domains of psychology or neuroscience in general.
“From a statistical point of view this problem might not be any more serious for this field than other domains of psychology (20, 21, 31) or neuroscience in general (14).”
From Statistical and Methodological Considerations for the Interpretation of Intranasal Oxytocin Studies (2016) · cites this paper as [20]
Publication bias and researcher degrees of freedom are pervasive problems in psychological science, so unadjusted effect sizes are likely overestimated.
“Given that publication bias and researcher degrees of freedom are understood to be pervasive problems in psychological science (Bakker et al., 2012; Franco et al., 2014, 2016; Hesse, 2018; John et al., 2012), unadjusted effect sizes are likely to be overestimated (Kvarven et al., 2020).”
From Do Salient Social Norms Moderate Mortality Salience Effects? A (Challenging) Meta-Analysis of Terror Management Studies (2023) · cites this paper as Bakker et al., 2012

Two new Later Stone Age sites from the Final Pleistocene in the Falémé Valley, eastern Senegal
Ndiaye, M., Lespez, L., Tribolo, C., Rassé, M., Hadjas, I., Davidoux, S., Huysecom, É., & Douze, K. (2024) · Monash University Research Portal (Monash University) · doi:10.3929/ethz-b-000667478 · Cited by 24,015 · see citing works
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · PLoS Medicine · doi:10.1371/journal.pmed.0020124 · Cited by 10,505 · see citing works
The file drawer problem and tolerance for null results.
Rosenthal, R. (1979) · Psychological Bulletin · doi:10.1037/0033-2909.86.3.638 · Cited by 9,068 · see citing works
False-Positive Psychology: Undisclosed Flexibility in Data Collection and Analysis Allows Presenting Anything as Significant
Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011) · Psychological Science · doi:10.1177/0956797611417632 · Cited by 6,567 · see citing works
Journal of The American Statistical Association
Fienberg, S. E. (2004) · Encyclopedia of Statistical Sciences · doi:10.1002/0471667196.ess1317 · Cited by 6,289 · see citing works
Understanding and using the Implicit Association Test: I. An improved scoring algorithm.
Greenwald, A. G., Nosek, B. A., & Banaji, M. R. (2003) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.85.2.197 · Cited by 5,815 · see citing works
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · CHANCE · doi:10.1080/09332480.2005.10722754 · Cited by 4,702 · see citing works
The earth is round (p < .05).
Cohen, J. (1994) · American Psychologist · doi:10.1037/0003-066x.49.12.997 · Cited by 4,003 · see citing works
Understanding and using the Implicit Association Test: III. Meta-analysis of predictive validity.
Greenwald, A. G., Poehlman, T. A., Uhlmann, E. L., & Banaji, M. R. (2009) · Journal of Personality and Social Psychology · doi:10.1037/a0015575 · Cited by 3,257 · see citing works
Belief in the law of small numbers.
Tversky, A., & Kahneman, D. (1971) · Psychological Bulletin · doi:10.1037/h0031322 · Cited by 3,246 · see citing works
Raise standards for preclinical cancer research
Begley, C. G., & Ellis, L. M. (2012) · Nature · doi:10.1038/483531a · Cited by 2,979 · cited 1× · see citing works
“Publication bias can have dire consequences, as illustrated recently by clear failures to replicate medical findings (Begley & Ellis, 2012; Prinz, Schlange & Asadullah, 2011).”
Background
A practical solution to the pervasive problems ofp values
Wagenmakers, E.-J. (2007) · Psychonomic Bulletin & Review · doi:10.3758/bf03194105 · Cited by 2,490 · see citing works
Publication Bias in Meta‐Analysis: Prevention, Assessment and Adjustments
(2005) · doi:10.1002/0470870168 · Cited by 2,368 · see citing works
Measuring the Prevalence of Questionable Research Practices With Incentives for Truth Telling
John, L. K., Loewenstein, G., & Prelec, D. (2012) · Psychological Science · doi:10.1177/0956797611430953 · Cited by 2,239 · see citing works
The efficacy of psychological, educational, and behavioral treatment: Confirmation from meta-analysis.
Lipsey, M. W., & Wilson, D. B. (1993) · American Psychologist · doi:10.1037/0003-066x.48.12.1181 · Cited by 2,033 · see citing works
Theoretical risks and tabular asterisks: Sir Karl, Sir Ronald, and the slow progress of soft psychology.
Meehl, P. E. (1978) · Journal of Consulting and Clinical Psychology · doi:10.1037/0022-006x.46.4.806 · Cited by 1,952 · see citing works
Believe it or not: how much can we rely on published data on potential drug targets?
Prinz, F., Schlange, T., & Asadullah, K. (2011) · Nature Reviews Drug Discovery · doi:10.1038/nrd3439-c1 · Cited by 1,888 · see citing works
Things I have learned (so far).
Cohen, J. (1990) · American Psychologist · doi:10.1037/0003-066x.45.12.1304 · Cited by 1,781 · see citing works
One Hundred Years of Social Psychology Quantitatively Described
Richard, F. D., Bond, C. F., & Stokes-Zoota, J. J. (2003) · Review of General Psychology · doi:10.1037/1089-2680.7.4.331 · Cited by 1,558 · see citing works
Why Most Discovered True Associations Are Inflated
Ioannidis, J. P. A. (2008) · Epidemiology · doi:10.1097/ede.0b013e31818131e7 · Cited by 1,495 · see citing works
Summing up: The Science of Reviewing Research
Wilkinson, I., Light, R. J., & Pillemer, D. B. (1987) · Journal of Educational Statistics · doi:10.2307/1164691 · Cited by 1,486 · see citing works
Puzzlingly High Correlations in fMRI Studies of Emotion, Personality, and Social Cognition
Vul, E., Harris, C., Winkielman, P., & Pashler, H. (2009) · Perspectives on Psychological Science · doi:10.1111/j.1745-6924.2009.01125.x · Cited by 1,475 · see citing works
Psychological testing and psychological assessment: A review of evidence and issues.
Meyer, G. J., Finn, S. E., Eyde, L. D., Kay, G. G., Moreland, K. L., Dies, R. R., Eisman, E. J., Kubiszyn, T. W., & Reed, G. M. (2001) · American Psychologist · doi:10.1037//0003-066x.56.2.128 · Cited by 1,469 · see citing works
Summing Up
Light, R. J., & Pillemer, D. B. (1984) · Harvard University Press eBooks · doi:10.4159/9780674040243 · Cited by 1,422 · see citing works
Does discovery-based instruction enhance learning?
Alfieri, L., Brooks, P. J., Aldrich, N. J., & Tenenbaum, H. R. (2011) · Journal of Educational Psychology · doi:10.1037/a0021017 · Cited by 1,338 · see citing works
To read or not to read: a meta-analysis of print exposure from infancy to early adulthood
Mol, S. E., & Bus, A. G. (2011) · Psychological Bulletin · doi:10.1037/a0021890 · Cited by 1,299 · see citing works
Null hypothesis significance testing: a review of an old and continuing controversy
Nickerson, R. S. (2000) · Psychological Methods · doi:10.1037/1082-989x.5.2.241 · Cited by 1,090 · see citing works
Beyond significance testing: Reforming data analysis methods in behavioral research.
Kline, R. B. (2004) · doi:10.1037/10693-000 · Cited by 1,034 · see citing works
Publication prejudices: An experimental study of confirmatory bias in the peer review system
Mahoney, M. J. (1977) · Cognitive Therapy and Research · doi:10.1007/bf01173636 · Cited by 1,013 · see citing works
Why psychologists must change the way they analyze their data: The case of psi: Comment on Bem (2011).
Wagenmakers, E.-J., Wetzels, R., Borsboom, D., & van der Maas, H. L. J. (2011) · Journal of Personality and Social Psychology · doi:10.1037/a0022790 · Cited by 977 · see citing works
Publication Decisions and their Possible Effects on Inferences Drawn from Tests of Significance—or Vice Versa
Sterling, T. D. (1959) · Journal of the American Statistical Association · doi:10.1080/01621459.1959.10501497 · Cited by 964 · cited 3× · see citing works
“It has long been argued that the combined outcomes of NHST in the scientific literature are too good to be true (Fanelli, 2010; Fiedler, 2011; Ioannidis, 2008a; Sterling, 1959; Vul, Harris, Winkielman, & Pashler, 2009).”
Background
Statistical Evidence in Experimental Psychology: An Empirical Comparison Using 855 t Tests
Wetzels, R., Matzke, D., Lee, M. D., Rouder, J. N., Iverson, G. J., & Wagenmakers, E.-J. (2011) · Perspectives on Psychological Science · doi:10.1177/1745691611406923 · Cited by 945 · see citing works
A meta-analysis of the effect of cognitive bias modification on anxiety and depression
Hallion, L. S., & Ruscio, A. M. (2011) · Psychological Bulletin · doi:10.1037/a0024355 · Cited by 891 · see citing works
Mindless statistics.
Gigerenzer, G. (2004) · The Journal of Socio-Economics · doi:10.1016/j.socec.2004.09.033 · Cited by 890 · see citing works
Regression Methods to Detect Publication and Other Bias in Meta‐Analysis
Sterne, J. A. C., & Egger, M. (2005) · Publication Bias in Meta‐Analysis · doi:10.1002/0470870168.ch6 · Cited by 826 · cited 2× · see citing works
“4), although it is important to note that they are sensitive to actual heterogeneity of the underlying ESs (Ioannidis & Trikalinos, 2007; Sterne & Egger, 2005).”
BackgroundMethodInfluential
An exploratory test for an excess of significant findings
Ioannidis, J. P., & Trikalinos, T. A. (2007) · Clinical Trials · doi:10.1177/1740774507079441 · Cited by 751 · see citing works
The fallacy of the null-hypothesis significance test.
Rozeboom, W. W. (1960) · Psychological Bulletin · doi:10.1037/h0042040 · Cited by 738 · see citing works
“Positive” results increase down the Hierarchy of the Sciences
Fanelli, D. (2010) · PLoS ONE · doi:10.1371/journal.pone.0010068 · Cited by 734 · see citing works
Culturally adapted psychotherapy and the legitimacy of myth: A direct-comparison meta-analysis.
Benish, S. G., Quintana, S., & Wampold, B. E. (2011) · Journal of Counseling Psychology · doi:10.1037/a0023626 · Cited by 661 · see citing works
The Persistence of Underpowered Studies in Psychological Research: Causes, Consequences, and Remedies.
Maxwell, S. E. (2004) · Psychological Methods · doi:10.1037/1082-989x.9.2.147 · Cited by 655 · see citing works
Do other-reports of counterproductive work behavior provide an incremental contribution over self-reports? A meta-analytic comparison.
Berry, C. M., Carpenter, N. C., & Barratt, C. L. (2011) · Journal of Applied Psychology · doi:10.1037/a0026739 · Cited by 623 · see citing works
Research in the Psychological Laboratory
Anderson, C. A., Lindsay, J. J., & Bushman, B. J. (1999) · Current Directions in Psychological Science · doi:10.1111/1467-8721.00002 · Cited by 500 · see citing works
Bayesian Assessment of Null Values Via Parameter Estimation and Model Comparison
Kruschke, J. (2011) · Perspectives on Psychological Science · doi:10.1177/1745691611406925 · Cited by 498 · see citing works
Publication bias in psychological science: Prevalence, methods for identifying and controlling, and implications for the use of meta-analyses.
Ferguson, C. J., & Brannick, M. Τ. (2011) · Psychological Methods · doi:10.1037/a0024445 · Cited by 493 · see citing works
Publication Decisions Revisited: The Effect of the Outcome of Statistical Tests on the Decision to Publish and Vice Versa
Sterling, T. D., Rosenbaum, W. L., & Weinkam, J. J. (1995) · The American Statistician · doi:10.1080/00031305.1995.10476125 · Cited by 453 · see citing works
Willingness to Share Research Data Is Related to the Strength of the Evidence and the Quality of Reporting of Statistical Results
Wicherts, J. M., Bakker, M., & Molenaar, D. (2011) · PLoS ONE · doi:10.1371/journal.pone.0026828 · Cited by 424 · see citing works
Self-citation
Scientist as Subject: The Psychological Imperative.
Lewis, L. S., & Mahoney, M. J. (1978) · Contemporary Sociology A Journal of Reviews · doi:10.2307/2064480 · Cited by 417 · see citing works
The (mis)reporting of statistical results in psychology journals
Bakker, M., & Wicherts, J. M. (2011) · Behavior Research Methods · doi:10.3758/s13428-011-0089-5 · Cited by 356 · see citing works
Self-citation
A Meta-Analysis of Infant Habituation and Recognition Memory Performance as Predictors of Later IQ
McCall, R. B., & Carriger, M. S. (1993) · Child Development · doi:10.1111/j.1467-8624.1993.tb02895.x · Cited by 337 · see citing works
The association between paternal sensitivity and infant-father attachment security: A meta-analysis of three decades of research
Lucassen, N., Tharner, A., Van IJzendoorn, M. H., Bakermans-Kranenburg, M. J., Volling, B. L., Verhulst, F. C., Lambregtse-Van den Berg, M. P., & Tiemeier, H. (2011) · Journal of Family Psychology · doi:10.1037/a0025855 · Cited by 296 · see citing works
A Meta-Analysis of Infant Habituation and Recognition Memory Performance as Predictors of Later IQ
McCall, R. B., & Carriger, M. S. (1993) · Child Development · doi:10.2307/1131437 · Cited by 256 · see citing works
Voodoo Correlations Are Everywhere—Not Only in Neuroscience
Fiedler, K. (2011) · Perspectives on Psychological Science · doi:10.1177/1745691611400237 · Cited by 232 · see citing works
Too good to be true: Publication bias in two prominent studies from experimental psychology
Francis, G. (2012) · Psychonomic Bulletin & Review · doi:10.3758/s13423-012-0227-9 · Cited by 210 · see citing works
Red, rank, and romance in women viewing men.
Elliot, A. J., Kayser, D. N., Greitemeyer, T., Lichtenfeld, S., Gramzow, R. H., Maier, M., & Liu, H. (2010) · Journal of Experimental Psychology General · doi:10.1037/a0019689 · Cited by 186 · see citing works
Sample size in psychological research over the past 30 years
Marszalek, J. M., Barber, C., Kohlhart, J., & Cooper, B. H. (2011) · Perceptual and Motor Skills · doi:10.2466/03.11.pms.112.2.331-348 · Cited by 167 · see citing works
Effect of positive findings on submission and acceptance rates: A note on meta-analysis bias.
Coursol, A., & Wagner, E. E. (1986) · Professional Psychology Research and Practice · doi:10.1037/0735-7028.17.2.136 · Cited by 165 · see citing works
Sample Sizes and Effect Sizes are Negatively Correlated in Meta-Analyses: Evidence and Implications of a Publication Bias Against NonSignificant Findings
Levine, T. R., Asada, K. J. K., & Carpenter, C. (2009) · Communication Monographs · doi:10.1080/03637750903074685 · Cited by 159 · see citing works
A two-dimensional approach to relationship conflict: meta-analytic findings.
Woodin, E. M. (2011) · Journal of family psychology · doi:10.1037/a0023791 · Cited by 156 · see citing works
A meta-analytic review of internalizing, externalizing, and academic adjustment among children of deployed military service members.
Card, N. A., Bosch, L. A., Casper, D. M., Wiggs, C. B., Hawkins, S. A., Schlomer, G. L., & Borden, L. M. (2011) · Journal of Family Psychology · doi:10.1037/a0024395 · Cited by 144 · see citing works
Finding the missing science: The fate of studies submitted for review by a human subjects committee.
Cooper, H., DeNeve, K., & Charlton, K. (1997) · Psychological Methods · doi:10.1037/1082-989x.2.4.447 · Cited by 144 · see citing works
Advantages of excluding underpowered studies in meta-analysis: Inclusionist versus exclusionist viewpoints.
Kraemer, H. C., Gardner, C., Brooks, J. O., & Yesavage, J. A. (1998) · Psychological Methods · doi:10.1037/1082-989x.3.1.23 · Cited by 141 · see citing works
Interpretation of significance levels and effect sizes by psychological researchers.
Nelson, N., Rosenthal, R., & Rosnow, R. L. (1986) · American Psychologist · doi:10.1037/0003-066x.41.11.1299 · Cited by 118 · see citing works
Probability as certainty: Dichotomous thinking and the misuse ofp values
Hoekstra, R., Finch, S., Kiers, H., & Johnson, A. M. (2006) · Psychonomic Bulletin & Review · doi:10.3758/bf03213921 · Cited by 113 · see citing works
The Cognitive Differentiation-Integration Effort Hypothesis: A Synthesis between the Fitness Indicator and Life History Models of Human Intelligence
Menie, M. A. W. O. (2011) · Review of General Psychology · doi:10.1037/a0024348 · Cited by 91 · see citing works
Evaluating the gold standard: A review and meta-analysis of the Structured Interview of Reported Symptoms.
Green, D., & Rosenfeld, B. D. (2011) · Psychological Assessment · doi:10.1037/a0021149 · Cited by 79 · see citing works
Positive regard.
Farber, B. A., & Doolin, E. M. (2011) · Psychotherapy · doi:10.1037/a0022141 · Cited by 77 · see citing works
Short, Sweet, and Problematic? The Rise of the Short Report in Psychological Science
Ledgerwood, A., & Sherman, J. W. (2012) · Perspectives on Psychological Science · doi:10.1177/1745691611427304 · Cited by 72 · see citing works
How Big Are Nonverbal Sex Differences? The Case of Smiling and Sensitivity to Nonverbal Cues
Hall, J. A. (1998) · Psychology Press eBooks · doi:10.4324/9781315805870-7 · Cited by 69 · see citing works
Effect of Formal Statistical Significance on the Credibility of Observational Associations
Ioannidis, J. P. A. (2008) · American Journal of Epidemiology · doi:10.1093/aje/kwn156 · Cited by 64 · see citing works
We Knew the Future All Along
Bones, A. K. (2012) · Perspectives on Psychological Science · doi:10.1177/1745691612441216 · Cited by 48 · see citing works
How many studies are in the file drawer? An estimate from the family/marital psychotherapy literature
Shadish, W. R., Doherty, M., & Montgomery, L. M. (1989) · Clinical Psychology Review · doi:10.1016/0272-7358(89)90013-5 · Cited by 45 · see citing works
The Same Old New Look: Publication Bias in a Study of Wishful Seeing
Francis, G. (2012) · i-Perception · doi:10.1068/i0519ic · Cited by 37 · see citing works
Misconduct in the Analysis and Reporting of Data: Bridging Methodological and Ethical Agendas for Change
Sterba, S. K. (2006) · Ethics & Behavior · doi:10.1207/s15327019eb1604_3 · Cited by 15 · see citing works
Sample-selection procedures for estimating the relationship between concentration and profitability from cross-industry data
Bloch, H. (1994) · Review of Industrial Organization · doi:10.1007/bf01024220 · Cited by 12 · see citing works
Applications of meta-analysis: 1987-1992
Tett, R. P., Meyer, J., & Roese, N. J. (1994) · Cited by 8 · see citing works

Flagged first, then by citations

Sorted by citations

Calculating and reporting effect sizes to facilitate cumulative science: A practical primer for t-tests and ANOVAs
Lakens, D. (2013) · Frontiers in Psychology · doi:10.3389/fpsyg.2013.00863 · Cited by 10,303 · see citing works
Choosing Prediction Over Explanation in Psychology: Lessons From Machine Learning
Yarkoni, T., & Westfall, J. (2017) · Perspectives on Psychological Science · doi:10.1177/1745691617693393 · Cited by 1,930 · see citing works
The preregistration revolution
Nosek, B. A., Ebersole, C. R., DeHaven, A. C., & Mellor, D. T. (2018) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1708274114 · Cited by 1,998 · see citing works
Increasing Transparency Through a Multiverse Analysis
Steegen, S., Tuerlinckx, F., Gelman, A., & Vanpaemel, W. (2016) · Perspectives on Psychological Science · doi:10.1177/1745691616658637 · Cited by 1,388 · see citing works
The Triple P-Positive Parenting Program: a systematic review and meta-analysis of a multi-level system of parenting support
Sanders, M. R., Kirby, J. N., Tellegen, C. L., & Day, J. J. (2014) · Clinical Psychology Review · doi:10.1016/j.cpr.2014.04.003 · Cited by 1,077 · see citing works
Reproducibility in Science: Improving the Standard for Basic and Preclinical Research
Begley, C. G., & Ioannidis, J. P. A. (2015) · Circulation Research · doi:10.1161/circresaha.114.303819 · Cited by 1,035 · see citing works
An Agenda for Purely Confirmatory Research
Wagenmakers, E.-J., Wetzels, R., Borsboom, D., van der Maas, H. L. J., & Kievit, R. A. (2012) · Perspectives on Psychological Science · doi:10.1177/1745691612463078 · Cited by 1,042 · see citing works
The oxytocin receptor: from intracellular signaling to behavior
Jurek, B., & Neumann, I. D. (2018) · Physiological Reviews · doi:10.1152/physrev.00031.2017 · Cited by 999 · see citing works
Degrees of Freedom in Planning, Running, Analyzing, and Reporting Psychological Studies: A Checklist to Avoid p-Hacking
Wicherts, J. M., Veldkamp, C. L. S., Augusteijn, H. E. M., Bakker, M., van Aert, R. C. M., & van Assen, M. A. L. M. (2016) · Frontiers in Psychology · doi:10.3389/fpsyg.2016.01832 · Cited by 754 · see citing works
Recommendations for Increasing Replicability in Psychology
Asendorpf, J. B., Conner, M. T., Fruyt, F. D., Houwer, J. D., Denissen, J. J. A., Fiedler, K., Fiedler, S., Funder, D. C., Kliegl, R. M., Nosek, B. A., et al. (2013) · European Journal of Personality · doi:10.1002/per.1919 · Cited by 818 · see citing works
Effect Size Guidelines, Sample Size Calculations, and Statistical Power in Gerontology
Brydges, C. R. (2019) · Innovation in Aging · doi:10.1093/geroni/igz036 · Cited by 826 · see citing works
Bilingual advantages in executive functioning either do not exist or are restricted to very specific and undetermined circumstances
Paap, K. R., Johnson, H. A., & Sawi, O. M. (2015) · Cortex · doi:10.1016/j.cortex.2015.04.014 · Cited by 820 · see citing works
The meaningfulness of effect sizes in psychological research: differences between sub-disciplines and the impact of potential biases
Schäfer, T., & Schwarz, M. A. (2019) · Frontiers in Psychology · doi:10.3389/fpsyg.2019.00813 · Cited by 774 · see citing works
Many Analysts, One Data Set: Making Transparent How Variations in Analytic Choices Affect Results
Silberzahn, R., Uhlmann, E. L., Martin, D. P., Anselmi, P., Aust, F., Awtrey, E. C., Bahník, Š., Bai, F., Bannard, C., Bonnier, E., et al. (2018) · Advances in Methods and Practices in Psychological Science · doi:10.1177/2515245917747646 · Cited by 773 · see citing works
Empirical assessment of published effect sizes and power in the recent cognitive neuroscience and psychology literature
Szucs, D., & Ioannidis, J. P. A. (2017) · PLOS Biology · doi:10.1371/journal.pbio.2000797 · Cited by 761 · see citing works
Replicability, Robustness, and Reproducibility in Psychological Science
Nosek, B. A., Hardwicke, T. E., Moshontz, H., Allard, A., Corker, K. S., Dreber, A., Fidler, F., Hilgard, J., Kline Struhl, M., Nuijten, M. B., et al. (2022) · Annual Review of Psychology · doi:10.1146/annurev-psych-020821-114157 · Cited by 770 · see citing works
Statistical power and optimal design in experiments in which samples of participants respond to samples of stimuli.
Westfall, J., Kenny, D. A., & Judd, C. M. (2014) · Journal of Experimental Psychology: General · doi:10.1037/xge0000014 · Cited by 928 · see citing works
Is the Replicability Crisis Overblown? Three Arguments Examined
Pashler, H., & Harris, C. R. (2012) · Perspectives on Psychological Science · doi:10.1177/1745691612463401 · Cited by 697 · see citing works
The Power of Bias in Economics Research
Ioannidis, J. P. A., Stanley, T. D., & Doucouliagos, H. (2017) · The Economic Journal · doi:10.1111/ecoj.12461 · Cited by 634 · see citing works
Empirically derived guidelines for effect size interpretation in social psychology
Lovakov, A. V., & Рафиковна, А. Е. (2021) · European Journal of Social Psychology · doi:10.1002/ejsp.2752 · Cited by 692 · see citing works
Facts Are More Important Than Novelty: Replication in the Education Sciences
Makel, M. C., & Plucker, J. A. (2014) · Educational Researcher · doi:10.3102/0013189x14545513 · Cited by 609 · see citing works
Psychology's Renaissance
Nelson, L. D., Simmons, J., & Simonsohn, U. (2018) · Annual Review of Psychology · doi:10.1146/annurev-psych-122216-011836 · Cited by 548 · see citing works
Why Science Is Not Necessarily Self-Correcting
Ioannidis, J. P. A. (2012) · Perspectives on Psychological Science · doi:10.1177/1745691612464056 · Cited by 563 · see citing works
Correcting for Bias in Psychology: A Comparison of Meta-Analytic Methods
Carter, E. C., Schönbrodt, F. D., Gervais, W. M., & Hilgard, J. (2019) · Advances in Methods and Practices in Psychological Science · doi:10.1177/2515245919847196 · Cited by 588 · see citing works
Registered Reports: A new publishing initiative at Cortex
Chambers, C. (2012) · Cortex · doi:10.1016/j.cortex.2012.12.016 · Cited by 574 · see citing works
Sequential hypothesis testing with Bayes factors: Efficiently testing mean differences.
Schönbrodt, F. D., Wagenmakers, E., Zehetleitner, M., & Perugini, M. (2015) · Psychological Methods · doi:10.1037/met0000061 · Cited by 562 · see citing works
The Ordinal Effects of Ostracism: A Meta-Analysis of 120 Cyberball Studies
Hartgerink, C. H. J., van Beest, I., Wicherts, J. M., & Williams, K. D. (2015) · PLOS ONE · doi:10.1371/journal.pone.0127002 · Cited by 558 · see citing works
Top 10 Replicated Findings From Behavioral Genetics
Plomin, R., DeFries, J. C., Knopik, V. S., & Neiderhiser, J. M. (2016) · Perspectives on Psychological Science · doi:10.1177/1745691615617439 · Cited by 517 · see citing works
The past, present and future of Registered Reports
Chambers, C. D., & Tzavella, L. (2021) · Nature Human Behaviour · doi:10.1038/s41562-021-01193-7 · Cited by 499 · see citing works
On doing better science: From thrill of discovery to policy implications
Antonakis, J. (2017) · The Leadership Quarterly · doi:10.1016/j.leaqua.2017.01.006 · Cited by 483 · see citing works
What You See Is What You Get? Enhancing Methodological Transparency in Management Research
Aguinis, H., Ramani, R. S., & Alabduljader, N. (2017) · Academy of Management Annals · doi:10.5465/annals.2016.0011 · Cited by 477 · see citing works
The alleged crisis and the illusion of exact replication
Stroebe, W., & Strack, F. (2014) · Perspectives on Psychological Science · doi:10.1177/1745691613514450 · Cited by 597 · see citing works
Publication Bias in Psychology: A Diagnosis Based on the Correlation between Effect Size and Sample Size
Kühberger, A., Fritz, A., & Scherndl, T. (2014) · PLoS ONE · doi:10.1371/journal.pone.0105825 · Cited by 399 · see citing works
A Practical Primer To Power Analysis for Simple Experimental Designs
Perugini, M., Gallucci, M., & Costantini, G. (2018) · International Review of Social Psychology · doi:10.5334/irsp.181 · Cited by 426 · see citing works
Does stereotype threat influence performance of girls in stereotyped domains? A meta-analysis
Flore, P. C., & Wicherts, J. M. (2015) · Journal of School Psychology · doi:10.1016/j.jsp.2014.10.002 · Cited by 410 · see citing works
Sample-Size Planning for More Accurate Statistical Power: A Method Adjusting Sample Effect Sizes for Publication Bias and Uncertainty
Anderson, S. F., Kelley, K., & Maxwell, S. E. (2017) · Psychological Science · doi:10.1177/0956797617723724 · Cited by 406 · see citing works
Is Ego Depletion Real? An Analysis of Arguments
Friese, M., Loschelder, D. D., Gieseler, K., Frankenbach, J., & Inzlicht, M. (2018) · Personality and Social Psychology Review · doi:10.1177/1088868318762183 · Cited by 404 · see citing works
Meta-analysis of psychological treatments for posttraumatic stress disorder in adult survivors of childhood abuse
Ehring, T., Welboren, R., Morina, N., Wicherts, J. M., Freitag, J., & Emmelkamp, P. M. G. (2014) · Clinical Psychology Review · doi:10.1016/j.cpr.2014.10.004 · Cited by 373 · see citing works
Implications of the Credibility Revolution for Productivity, Creativity, and Progress
Vazire, S. (2018) · Perspectives on Psychological Science · doi:10.1177/1745691617751884 · Cited by 361 · see citing works
A systematic review of teacher guidance during collaborative learning in primary and secondary education
van Leeuwen, A., & Janssen, J. (2019) · Educational Research Review · doi:10.1016/j.edurev.2019.02.001 · Cited by 355 · see citing works
The relation between statistical power and inference in fMRI
Cremers, H. R., Wager, T. D., & Yarkoni, T. (2017) · PLoS ONE · doi:10.1371/journal.pone.0184923 · Cited by 336 · see citing works
Statistical and Methodological Considerations for the Interpretation of Intranasal Oxytocin Studies
Walum, H., Waldman, I. D., & Young, L. J. (2016) · Biological Psychiatry · doi:10.1016/j.biopsych.2015.06.016 · Cited by 345 · see citing works
“From a statistical point of view this problem might not be any more serious for this field than other domains of psychology (20, 21, 31) or neuroscience in general (14).”
Background
Publication bias examined in meta-analyses from psychology and medicine: A meta-meta-analysis
van Aert, R. C. M., Wicherts, J. M., & van Assen, M. A. L. M. (2019) · PLoS ONE · doi:10.1371/journal.pone.0215052 · Cited by 345 · see citing works
Does Self-Control Training Improve Self-Control? A Meta-Analysis
Friese, M., Frankenbach, J., Job, V., & Loschelder, D. D. (2017) · Perspectives on Psychological Science · doi:10.1177/1745691617697076 · Cited by 335 · see citing works
Approach, avoidance, and affect: a meta-analysis of approach-avoidance tendencies in manual reaction time tasks
Phaf, R. H., Mohr, S. E., Rotteveel, M., & Wicherts, J. M. (2014) · Frontiers in Psychology · doi:10.3389/fpsyg.2014.00378 · Cited by 326 · see citing works
Reporting effect sizes in original psychological research: A discussion and tutorial.
Pek, J. C. X., & Flora, D. B. (2017) · Psychological Methods · doi:10.1037/met0000126 · Cited by 321 · see citing works
A Systematic Scoping Review of the Choice Architecture Movement: Toward Understanding When and Why Nudges Work
Szászi, B., Pálinkás, A., Pálfi, B., Szollosi, A., & Aczél, B. (2017) · Journal of Behavioral Decision Making · doi:10.1002/bdm.2035 · Cited by 318 · see citing works
The effects of psychological distance on abstraction: Two meta-analyses.
Soderberg, C. K., Callahan, S. P., Kochersberger, A. O., Amit, E., & Ledgerwood, A. (2014) · Psychological Bulletin · doi:10.1037/bul0000005 · Cited by 319 · see citing works
Continuously Cumulating Meta-Analysis and Replicability
Braver, S. L., Thoemmes, F. J., & Rosenthal, R. (2014) · Perspectives on Psychological Science · doi:10.1177/1745691614529796 · Cited by 298 · see citing works
Sailing From the Seas of Chaos Into the Corridor of Stability: Practical Recommendations to Increase the Informational Value of Studies
Lakens, D., & Evers, E. R. K. (2014) · Perspectives on Psychological Science · doi:10.1177/1745691614528520 · Cited by 266 · see citing works
Science or Art? How Aesthetic Standards Grease the Way Through the Publication Bottleneck but Undermine Science
Giner-Sorolla, R. (2012) · Perspectives on Psychological Science · doi:10.1177/1745691612457576 · Cited by 274 · see citing works
It’s Time to Broaden the Replicability Conversation: Thoughts for and From Clinical Psychological Science
Tackett, J. L., Lilienfeld, S. O., Patrick, C. J., Johnson, S. L., Krueger, R. F., Miller, J. D., Oltmanns, T. F., & Shrout, P. E. (2017) · Perspectives on Psychological Science · doi:10.1177/1745691617690042 · Cited by 251 · see citing works
Qualitative Research Methods
Kazdin, A. E. (2021) · Cambridge University Press eBooks · doi:10.1017/9781108993647.010 · Cited by 279 · see citing works
Replication in Second Language Research: Narrative and Systematic Reviews and Recommendations for the Field
Marsden, E. J., Morgan‐Short, K., Thompson‐Lee, S., & Abugaber, D. (2018) · Language Learning · doi:10.1111/lang.12286 · Cited by 270 · see citing works
Meta-analysis of parent-mediated interventions for young children with autism spectrum disorder
Nevill, R. E., Lecavalier, L., & Stratis, E. A. (2018) · Autism · doi:10.1177/1362361316677838 · Cited by 265 · see citing works
Meta-analysis using effect size distributions of only statistically significant studies.
van Assen, M. A. L. M., van Aert, R. C. M., & Wicherts, J. M. (2015) · Psychological Methods · doi:10.1037/met0000025 · Cited by 265 · see citing works
How WEIRD is CHI?
Linxen, S., Sturm, C., Brühlmann, F., Cassau, V., Opwis, K., & Reinecke, K. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445488 · Cited by 254 · see citing works
Are bilingual advantages dependent upon specific tasks or specific bilingual experiences?
Paap, K. R., Johnson, H. A., & Sawi, O. M. (2014) · Journal of Cognitive Psychology · doi:10.1080/20445911.2014.944914 · Cited by 254 · see citing works
Bilingual advantages in executive functioning: problems in convergent validity, discriminant validity, and the identification of the theoretical constructs
Paap, K. R. (2014) · Frontiers in Psychology · doi:10.3389/fpsyg.2014.00962 · Cited by 252 · see citing works
A Short (Personal) Future History of Revolution 2.0
Spellman, B. A. (2015) · Perspectives on Psychological Science · doi:10.1177/1745691615609918 · Cited by 213 · see citing works
The Chrysalis Effect: How Ugly Initial Results Metamorphosize Into Beautiful Articles
O’Boyle, E. H., Banks, G. C., & Gonzalez-Mulé, E. (2017) · Journal of Management · doi:10.1177/0149206314527133 · Cited by 246 · see citing works
Men, women…who cares? A population-based study on sex differences and gender roles in empathy and moral cognition
Báez, S., Flichtentrei, D., Prats, M., Mastandueno, R., García, A. M., Cetkovich, M., & Ibáñez, A. M. (2017) · PLoS ONE · doi:10.1371/journal.pone.0179336 · Cited by 246 · see citing works
Publication bias and the limited strength model of self-control: has the evidence for ego depletion been overestimated?
Carter, E. C., & McCullough, M. E. (2014) · Frontiers in Psychology · doi:10.3389/fpsyg.2014.00823 · Cited by 381 · see citing works
Meaningful associations in the adolescent brain cognitive development study
Dick, A. S., Lopez, D. A., Watts, A. L., Heeringa, S. G., Reuter, C., Bartsch, H., Fan, C. C., Kennedy, D. N., Palmer, C. E., Marshall, A. T., et al. (2021) · NeuroImage · doi:10.1016/j.neuroimage.2021.118262 · Cited by 234 · see citing works
Exploratory data analysis as a foundation of inductive research
Jebb, A. T., Parrigon, S., & Woo, S. E. (2016) · Human Resource Management Review · doi:10.1016/j.hrmr.2016.08.003 · Cited by 239 · see citing works
What meta-analyses reveal about the replicability of psychological research.
Stanley, T. D., Carter, E. C., & Doucouliagos, H. (2018) · Psychological Bulletin · doi:10.1037/bul0000169 · Cited by 307 · see citing works
Do the disadvantaged legitimize the social system? A large-scale test of the status-legitimacy hypothesis.
Brandt, M. (2013) · Journal of Personality and Social Psychology · doi:10.1037/a0031751 · Cited by 213 · see citing works
“An A Is An A”: The New Bottom Line For Valuing Academic Research
Aguinis, H., Cummings, C., Ramani, R. S., & Cummings, T. G. (2019) · Academy of Management Perspectives · doi:10.5465/amp.2017.0193 · Cited by 226 · see citing works
Conducting Meta-Analyses Based on p Values: Reservations and Recommendations for Applying p -Uniform and p -Curve
van Aert, R. C. M., Wicherts, J. M., & van Assen, M. A. L. M. (2016) · Perspectives on Psychological Science · doi:10.1177/1745691616650874 · Cited by 219 · see citing works
Self-citation
A tutorial on assessing statistical power and determining sample size for structural equation models.
Jobst, L. J., Bader, M., & Moshagen, M. (2021) · Psychological Methods · doi:10.1037/met0000423 · Cited by 236 · see citing works
Current Incentives for Scientists Lead to Underpowered Studies with Erroneous Conclusions
Higginson, A. D., & Munafò, M. R. (2016) · PLOS Biology · doi:10.1371/journal.pbio.2000995 · Cited by 192 · see citing works
Outlier removal, sum scores, and the inflation of the type I error rate in independent samples t tests: The power of alternatives and recommendations.
Bakker, M., & Wicherts, J. M. (2014) · Psychological Methods · doi:10.1037/met0000014 · Cited by 208 · see citing works
Best research practices in psychology: Illustrating epistemological and pragmatic considerations with the case of relationship science.
Finkel, E. J., Eastwick, P. W., & Reis, H. T. (2015) · Journal of Personality and Social Psychology · doi:10.1037/pspi0000007 · Cited by 201 · see citing works
Safeguard Power as a Protection Against Imprecise Power Estimates
Perugini, M., Gallucci, M., & Costantini, G. (2014) · Perspectives on Psychological Science · doi:10.1177/1745691614528519 · Cited by 203 · see citing works
Psychology's Replication Crisis and Clinical Psychological Science
Tackett, J. L., Brandes, C. M., King, K. M., & Markon, K. E. (2019) · Annual Review of Clinical Psychology · doi:10.1146/annurev-clinpsy-050718-095710 · Cited by 196 · see citing works
An Assessment of the Magnitude of Effect Sizes
Paterson, T. A., Harms, P. D., Steel, P., & Credé, M. (2015) · Journal of Leadership & Organizational Studies · doi:10.1177/1548051815614321 · Cited by 196 · see citing works
Mapping brain asymmetry in health and disease through the ENIGMA consortium
Kong, X.-Z., Postema, M., Guadalupe, T., de Kovel, C., Boedhoe, P., Hoogman, M., Mathias, S., van Rooij, D., Schijven, D., Glahn, D., et al. (2019) · Human Brain Mapping · doi:10.1002/hbm.25033 · Cited by 182 · see citing works
Reproducible research in sport and exercise psychology: The role of sample sizes
Schweizer, G., & Furley, P. (2015) · Psychology of sport and exercise · doi:10.1016/j.psychsport.2015.11.005 · Cited by 186 · see citing works
Slice-to-volume medical image registration: A survey
Ferrante, E., & Nikos Paragios, A. (2017) · Medical Image Analysis · doi:10.1016/j.media.2017.04.010 · Cited by 177 · see citing works
Statistical Rituals: The Replication Delusion and How We Got There
Gigerenzer, G. (2018) · Advances in Methods and Practices in Psychological Science · doi:10.1177/2515245918771329 · Cited by 263 · see citing works
Reducing bias, increasing transparency and calibrating confidence with preregistration
Hardwicke, T. E., & Wagenmakers, E. (2023) · Nature Human Behaviour · doi:10.1038/s41562-022-01497-2 · Cited by 190 · see citing works
Scientific Utopia III: Crowdsourcing Science
Uhlmann, E. L., Ebersole, C. R., Chartier, C. R., Errington, T. M., Kidwell, M. C., Lai, C. K., McCarthy, R. J., Riegelman, A., Silberzahn, R., & Nosek, B. A. (2019) · Perspectives on Psychological Science · doi:10.1177/1745691619850561 · Cited by 182 · see citing works
“Rarely safe to assume”: Evaluating the use and interpretation of manipulation checks in experimental social psychology
Ejelöv, E., & Luke, T. J. (2019) · Journal of Experimental Social Psychology · doi:10.1016/j.jesp.2019.103937 · Cited by 175 · see citing works
The Preregistration RevolutionPreprint
Nosek, B. A., Ebersole, C. R., DeHaven, A. C., & Mellor, D. T. (2017) · doi:10.31219/osf.io/2dxu5 · Cited by 220 · see citing works
Reconceptualizing replication as a sequence of different studies: A replication typology
Hüffmeier, J., Mazei, J., & Schultze, T. (2016) · Journal of Experimental Social Psychology · doi:10.1016/j.jesp.2015.09.009 · Cited by 168 · see citing works
Questions About Questionable Research Practices in the Field of Management: A Guest Commentary
Banks, G. C., O’Boyle, E. H., Pollack, J. M., White, C. D., Batchelor, J. H., Whelpley, C. E., Abston, K. A., Bennett, A. A., & Adkins, C. L. (2016) · Journal of Management · doi:10.1177/0149206315619011 · Cited by 163 · see citing works
Science’s reproducibility and replicability crisis: International business is not immune
Aguinis, H., Cascio, W. F., & Ramani, R. S. (2017) · Journal of International Business Studies · doi:10.1057/s41267-017-0081-0 · Cited by 163 · see citing works
Questionable research practices among italian research psychologists
Agnoli, F., Wicherts, J. M., Veldkamp, C. L. S., Albiero, P., & Cubelli, R. (2017) · PLoS ONE · doi:10.1371/journal.pone.0172792 · Cited by 160 · see citing works
Problems with Traditional Science Publishing and Finding a Wider Niche for Post-Publication Peer Review
Teixeira da Silva, J. A., & Dobránszki, J. (2015) · Accountability in Research · doi:10.1080/08989621.2014.899909 · Cited by 158 · see citing works
Lessons From Pinocchio: Cues to Deception May Be Highly Exaggerated
Luke, T. (2018) · Perspectives on Psychological Science · doi:10.1177/1745691619838258 · Cited by 135 · see citing works
How (not) to measure infant Theory of Mind: Testing the replicability and validity of four non-verbal measures
Dörrenberg, S., Hannes Rakoczy, B., & Liszkowski, U. (2018) · Cognitive Development · doi:10.1016/j.cogdev.2018.01.001 · Cited by 158 · see citing works
Seven Easy Steps to Open Science
Crüwell, S., van Doorn, J. B., Etz, A. J., Makel, M. C., Moshontz, H., Niebaum, J. C., Orben, A. C., Parsons, S. D., & Schulte‐Mecklenbeck, M. (2019) · Zeitschrift für Psychologie · doi:10.1027/2151-2604/a000387 · Cited by 153 · see citing works
Clarifications on the application and interpretation of the test for excess significance and its extensions
Ioannidis, J. P. A. (2013) · Journal of Mathematical Psychology · doi:10.1016/j.jmp.2013.03.002 · Cited by 150 · see citing works
Answers to 18 Questions About Open Science Practices
Banks, G. C., Field, J. G., Oswald, F. L., O’Boyle, E. H., Landis, R. S., Rupp, D. E., & Rogelberg, S. G. (2018) · Journal of Business and Psychology · doi:10.1007/s10869-018-9547-8 · Cited by 151 · see citing works
Novel Effect Size Interpretation Guidelines and an Evaluation of Statistical Power in Rehabilitation Research
Kinney, A. R., Eakman, A. M., & Graham, J. E. (2020) · Archives of Physical Medicine and Rehabilitation · doi:10.1016/j.apmr.2020.02.017 · Cited by 148 · see citing works
Researchers’ Intuitions About Power in Psychological Research
Bakker, M., Hartgerink, C. H. J., Wicherts, J. M., & van der Maas, H. L. J. (2016) · Psychological Science · doi:10.1177/0956797616647519 · Cited by 147 · see citing works
Having your cake and eating it too: Flexibility and power with mass univariate statistics for ERP data
Fields, E. C., & Kuperberg, G. R. (2019) · Psychophysiology · doi:10.1111/psyp.13468 · Cited by 143 · see citing works
The role of test-retest reliability in measuring individual and group differences in executive functioning
Paap, K. R., & Sawi, O. M. (2016) · Journal of Neuroscience Methods · doi:10.1016/j.jneumeth.2016.10.002 · Cited by 139 · see citing works
A Primer for Conducting Experiments in Human–Robot Interaction
Hoffman, G., & Zhao, X. (2020) · ACM Transactions on Human-Robot Interaction · doi:10.1145/3412374 · Cited by 138 · see citing works
How robust are anticipatory looking measures of Theory of Mind? Replication attempts across the life span
Kulke, L., Reiß, M., Krist, H., & Hannes Rakoczy, B. (2017) · Cognitive Development · doi:10.1016/j.cogdev.2017.09.001 · Cited by 134 · see citing works

Authors: Bakker, M., van Dijk, A., & Wicherts, J. M.
Affiliations: University of Amsterdam (NL)
University of Amsterdam, The Netherlands
Tilburg University (NL)
ORCID: Bakker, Marjan
Wicherts, Jelte M.
Venue: Perspectives on Psychological Science
Publisher: SAGE Publications
Published: 2012-11
Volume: 7
Issue: 6
Pages: 543-554
ISSN: 1745-6916 (ISSN-L)
1745-6924
DOI: 10.1177/1745691612459060
OpenAlex: W2157668596
Cited by: 800
References: 75
Citations by year: 2026: 14 · 2025: 32 · 2024: 39 · 2023: 50 · 2022: 52 · 2021: 100 · 2020: 63 · 2019: 82 · 2018: 87 · 2017: 88 · 2016: 56 · 2015: 59 · 2014: 52 · 2013: 19 · 2012: 5
FWCI: 49.25
Type: journal article
Language: en
Open access: bronze
doi.org
Licence: journals.sagepub.com/page/policies/text-and-data-mining-license (tdm)
Locations: Perspectives on Psychological Science (publishedVersion)
Research portal (Tilburg University) (publishedVersion)
PubMed (publishedVersion)
Data Archiving and Networked Services (DANS) (submittedVersion)
UvA-DARE (University of Amsterdam) (submittedVersion)
Funders: Nederlandse Organisatie voor Wetenschappelijk Onderzoek (016-125-385)
Topics: Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
Social Sciences › Psychology › Experimental and Cognitive Psychology › Psychology Research and Bibliometrics
Social Sciences › Safety Research › Academic integrity and plagiarism
Keywords: null hypothesis significance testing, statistical power, publication bias, questionable research practices, false positives, meta-analysis, replication, sample size, p-values, research integrity, editorial policy

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

The rules of the game called psychological science

Abstract

Integrity signals

Claims the paper makes

Claims other papers make about this one

Cites

Cited by

Related

Record

About these checks