The DOI matches its Crossref record; 43 of the 43 references it cites were found in a registry or an index.
The Meaningfulness of Effect Sizes in Psychological Research: Differences Between Sub-Disciplines and the Impact of Potential Biases
Open accessAbstract
Effect sizes are the currency of psychological research. They quantify the results of a study to answer the research question and are used to calculate statistical power. The interpretation of effect sizes—when is an effect small, medium, or large?—has been guided by the recommendations Jacob Cohen gave in his pioneering writings starting in 1962: Either compare an effect with the effects found in past research or use certain conventional benchmarks. The present analysis shows that neither of these recommendations is currently applicable. From past publications without pre-registration, 900 effects were randomly drawn and compared with 93 effects from publications with pre-registration, revealing a large difference: Effects from the former (median r = .36) were much larger than effects from the latter (median r = .16). That is, certain biases, such as publication bias or questionable research practices, have caused a dramatic inflation in published effects, making it difficult to compare an actual effect with the real population effects (as these are unknown). In addition, there were very large differences in the mean effects between psychological sub-disciplines and between different study designs, making it impossible to apply any global benchmarks. Many more pre-registered studies are needed in the future to derive a reliable picture of real population effects.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“The multitude of challenges presented by living with chronic pain may also impact ability to regulate thoughts, feelings, and behavior (ie, self-regulation).46–48,58”
From Living well with chronic pain: a 12-month randomized controlled trial revealing impact from the digital pain self-management program EPIO (2024) · cites this paper as [48]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Statistical Power Analysis for the Behavioral Sciences
Muller, K. E., & Cohen, J. (1989) · Technometrics · doi:10.2307/1270020 · Cited by 99,603
Duval, S., & Tweedie, R. (2000) · Biometrics · doi:10.1111/j.0006-341x.2000.00455.x · Cited by 14,099
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · PLoS Medicine · doi:10.1371/journal.pmed.0020124 · Cited by 10,505
Lakens, D. (2013) · Frontiers in Psychology · doi:10.3389/fpsyg.2013.00863 · Cited by 10,303
Cohen, J. (1992) · Psychological Bulletin · doi:10.1037//0033-2909.112.1.155 · Cited by 8,771
Estimating the reproducibility of psychological science
Open Science Collaboration (2015) · Science · doi:10.1126/science.aac4716 · Cited by 8,515
Effect size estimates: Current use, calculations, and interpretation.
Fritz, C. O., Morris, P. E., & Richler, J. J. (2012) · Journal of Experimental Psychology: General · doi:10.1037/a0024338 · Cited by 5,199
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · CHANCE · doi:10.1080/09332480.2005.10722754 · Cited by 4,702
The New Statistics: Why and How
Cumming, G. (2014) · Psychological Science · doi:10.1177/0956797613504966 · Cited by 3,200
Effect size guidelines for individual differences researchers
Gignac, G. E., & Szodorai, E. T. (2016) · Personality and Individual Differences · doi:10.1016/j.paid.2016.06.069 · Cited by 2,860
Ellis, P. D. (2010) · doi:10.1017/cbo9780511761676 · Cited by 2,461
Measuring the Prevalence of Questionable Research Practices With Incentives for Truth Telling
John, L. K., Loewenstein, G., & Prelec, D. (2012) · Psychological Science · doi:10.1177/0956797611430953 · Cited by 2,239
Book Review: Design and Analysis: A Researcher's Handbook
Aaker, D. A., & Keppel, G. (1976) · Journal of Marketing Research · doi:10.2307/3150757 · Cited by 2,033
The statistical power of abnormal-social psychological research: A review.
Cohen, J. (1962) · The Journal of Abnormal and Social Psychology · doi:10.1037/h0045186 · Cited by 1,824
Things I have learned (so far).
Cohen, J. (1990) · American Psychologist · doi:10.1037/0003-066x.45.12.1304 · Cited by 1,781
Practical significance: a concept whose time has come
Kirk, R. E. (1996) · Educational and Psychological Measurement · doi:10.1177/0013164496056005002 · Cited by 1,778 · cited 3×
“While compliance in four prominent APA journals in 1995 was only 48% (Kirk, 1996), it was almost 100% in 2015 (Schäfer, 2018), for articles reporting an inferential statistic.”
BackgroundOlejnik, S., & Algina, J. (2003) · Psychological Methods · doi:10.1037/1082-989x.8.4.434 · Cited by 1,622
One Hundred Years of Social Psychology Quantitatively Described
Richard, F. D., Bond, C. F., & Stokes-Zoota, J. J. (2003) · Review of General Psychology · doi:10.1037/1089-2680.7.4.331 · Cited by 1,558
Appelbaum, M., Cooper, H., Kline, R. B., Mayo-Wilson, E., Nezu, A. M., & Rao, S. M. (2018) · American Psychologist · doi:10.1037/amp0000191 · Cited by 1,449
Statistical Power Analysis for the Behavioral Sciences
Wassertheil, S., & Cohen, J. (1970) · Biometrics · doi:10.2307/2529115 · Cited by 1,383
Negative results are disappearing from most disciplines and countries
Fanelli, D. (2012) · Scientometrics · doi:10.1007/s11192-011-0494-7 · Cited by 1,248
Interpreting the magnitudes of correlation coefficients.
Hemphill, J. F. (2003) · American Psychologist · doi:10.1037/0003-066x.58.1.78 · Cited by 1,237
Kelley, K., & Preacher, K. J. (2012) · Psychological Methods · doi:10.1037/a0028086 · Cited by 1,084
An Agenda for Purely Confirmatory Research
Wagenmakers, E.-J., Wetzels, R., Borsboom, D., van der Maas, H. L. J., & Kievit, R. A. (2012) · Perspectives on Psychological Science · doi:10.1177/1745691612463078 · Cited by 1,042
The rules of the game called psychological science
Bakker, M., van Dijk, A., & Wicherts, J. M. (2012) · Perspectives on Psychological Science · doi:10.1177/1745691612459060 · Cited by 800
The Replication Recipe: What makes for a convincing replication?
Brandt, M. J., IJzerman, H., Dijksterhuis, A., Farach, F. J., Geller, J., Giner‐Sorolla, R., Grange, J. A., Perugini, M., Spies, J. R., & Veer, A. V. '. (2013) · Journal of Experimental Social Psychology · doi:10.1016/j.jesp.2013.10.005 · Cited by 758
Do studies of statistical power have an effect on the power of studies?
Sedlmeier, P., & Gigerenzer, G. (1989) · Psychological Bulletin · doi:10.1037/0033-2909.105.2.309 · Cited by 748
p -Curve and Effect Size: Correcting for Publication Bias Using Only Significant Results
Simonsohn, U., Nelson, L. D., & Simmons, J. P. (2014) · Perspectives on Psychological Science · doi:10.1177/1745691614553988 · Cited by 717
Standardized or simple effect size: what should be reported?
Baguley, T. (2009) · British Journal of Psychology · doi:10.1348/000712608x377117 · Cited by 619
An Open, Large-Scale, Collaborative Effort to Estimate the Reproducibility of Psychological Science
Nosek, B. A., & Lakens, D. (2012) · Perspectives on Psychological Science · doi:10.1177/1745691612462588 · Cited by 592
Fraley, R. C., & Vazire, S. (2014) · PLoS ONE · doi:10.1371/journal.pone.0109019 · Cited by 409
Reporting effect sizes in original psychological research: A discussion and tutorial.
Pek, J. C. X., & Flora, D. B. (2017) · Psychological Methods · doi:10.1037/met0000126 · Cited by 356
Meta-analysis using effect size distributions of only statistically significant studies.
van Assen, M. A. L. M., van Aert, R. C. M., & Wicherts, J. M. (2015) · Psychological Methods · doi:10.1037/met0000025 · Cited by 265
(2016) · Cited by 252
Effect magnitude: A different focus
Kirk, R. (2007) · Journal of Statistical Planning and Inference · doi:10.1016/j.jspi.2006.09.011 · Cited by 101
A methodological review of meta-analyses of the effectiveness of clinical psychology treatments
Rubio‐Aparicio, M., Marín‐Martínez, F., Sánchez‐Meca, J., & López-López, J. A. (2017) · Behavior Research Methods · doi:10.3758/s13428-017-0973-8 · Cited by 93
Many Labs 2: Investigating Variation in Replicability Across Sample and SettingPreprint
Klein, R. A., Vianello, M., Hasselman, F., Adams, B. G., Adams, R. B., Alper, S., Aveyard, M. E., Axt, J., Babalola, M. T., Bahník, Š., et al. (2018) · PsyArXiv (OSF Preprints) · doi:10.31234/osf.io/9654g · Cited by 87
Haase, R. F., Waechter, D. M., & Solomon, G. S. (1982) · Journal of Counseling Psychology · doi:10.1037/0022-0167.29.1.58 · Cited by 79
Effect sizes in memory research
Morris, P. E., & Fritz, C. O. (2013) · Memory · doi:10.1080/09658211.2013.763984 · Cited by 78
Cooper, H., & Findley, M. J. (1982) · Personality and Social Psychology Bulletin · doi:10.1177/014616728281026 · Cited by 53
The Power of Replications and Replications of Power
Ottenbacher, K. J. (1996) · The American Statistician · doi:10.1080/00031305.1996.10474398 · Cited by 31
Renkewitz, F., & Keiner, M. (2018) · PsyArXiv (OSF Preprints) · doi:10.31234/osf.io/w94ep · Cited by 15
Die New Statistics in der Psychologie
Schäfer, T. (2018) · Zeitschrift für Entwicklungspsychologie und Pädagogische Psychologie · doi:10.1026/0049-8637/a000184 · Cited by 8
Flagged first, then by citations
Research works printed but not linked: 5
- 2001, Publication Manual of the American Psychological Association
- 2010, Publication Manual of the American Psychological Association
- Cohen, 1977, Statistical Power Analysis for the Behavioral Sciences (revised Edn.).
- Keppel, 1991, Design and Analysis: A Researcher’s Handbook.
- Lehrer, 2010, The Truth Wears Off., 29
Cited by
Sorted by citations
Interpreting effect sizes of education interventions
Kraft, M. A. (2020) · Educational Researcher · doi:10.3102/0013189x20912798 · Cited by 1,281
Effect Size Guidelines, Sample Size Calculations, and Statistical Power in Gerontology
Brydges, C. R. (2019) · Innovation in Aging · doi:10.1093/geroni/igz036 · Cited by 826
Elliott, M. L., Knodt, A. R., Ireland, D., Morris, M. L., Poulton, R., Ramrakha, S., Sison, M. L., Moffitt, T. E., Caspi, A., & Hariri, A. R. (2020) · Psychological Science · doi:10.1177/0956797620916786 · Cited by 714
Empirically derived guidelines for effect size interpretation in social psychology
Lovakov, A. V., & Рафиковна, А. Е. (2021) · European Journal of Social Psychology · doi:10.1002/ejsp.2752 · Cited by 755
Small Effects: The Indispensable Foundation for a Cumulative Psychological Science
Götz, F. M., Gosling, S. D., & Rentfrow, P. J. (2021) · Perspectives on Psychological Science · doi:10.1177/1745691620984483 · Cited by 610
Metadata
- Authors
- Schäfer, T., & Schwarz, M. A.
- Affiliations
- Chemnitz University of Technology (DE)
- Corresponding author
- Schäfer, Thomas
- Venue
- Frontiers in Psychology
- Publisher
- Frontiers Media SA
- Published
- 2019-04-11
- Volume
- 10
- Pages
- 813-813
- ISSN
- 1664-1078 (ISSN-L)
- PMC
- PMC6470248
- OpenAlex
- W2937326640
- Cited by
- 774
- References
- 43
- Citations by year
- 2026: 73 · 2025: 105 · 2024: 101 · 2023: 123 · 2022: 149 · 2021: 127 · 2020: 75 · 2019: 18
- FWCI
- 61.73
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- Frontiers in Psychology (publishedVersion, cc-by)
- Europe PMC (PubMed Central) (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
- Qucosa (Saxon State and University Library Dresden) (submittedVersion, cc-by)
- Funders
- Technische Universität Chemnitz
- Deutsche Forschungsgemeinschaft
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Psychology Research and Bibliometrics
- Social Sciences › Decision Sciences › Management Science and Operations Research › Psychometric Methodologies and Testing
- Keywords
- effect size, Cohen's d, publication bias, questionable research practices, preregistration, effect size inflation, statistical power
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
