CoChat
Sign Up
Cited by 774Meta-analysis and systematic reviewsPsychology Research and BibliometricsPsychometric Methodologies and Testing

The DOI matches its Crossref record; 43 of the 43 references it cites were found in a registry or an index.

The Meaningfulness of Effect Sizes in Psychological Research: Differences Between Sub-Disciplines and the Impact of Potential Biases

Open access
Schäfer, Thomas, Schwarz, Marcus A.·2019·Frontiers in Psychology
DOI 10.3389/fpsyg.2019.00813PMC PMC6470248
Open this paper in CoChat — free accountOpen PDF

Abstract

Effect sizes are the currency of psychological research. They quantify the results of a study to answer the research question and are used to calculate statistical power. The interpretation of effect sizes—when is an effect small, medium, or large?—has been guided by the recommendations Jacob Cohen gave in his pioneering writings starting in 1962: Either compare an effect with the effects found in past research or use certain conventional benchmarks. The present analysis shows that neither of these recommendations is currently applicable. From past publications without pre-registration, 900 effects were randomly drawn and compared with 93 effects from publications with pre-registration, revealing a large difference: Effects from the former (median r = .36) were much larger than effects from the latter (median r = .16). That is, certain biases, such as publication bias or questionable research practices, have caused a dramatic inflation in published effects, making it difficult to compare an actual effect with the real population effects (as these are unknown). In addition, there were very large differences in the mean effects between psychological sub-disciplines and between different study designs, making it impossible to apply any global benchmarks. Many more pre-registered studies are needed in the future to derive a reliable picture of real population effects.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.3389/fpsyg.2019.00813

Cites

43 of 43 references verified0 citation errors · 0 not checkable

Statistics

Figures

FIGURE 1 Distributions of effects (absolute values) from articles published with (N = 89) and without (N = 684) pre-registration. The distributions contain all effects that were extracted as or could be transformed into a correlation coefficient r.
FIGURE 1 Distributions of effects (absolute values) from articles published with (N = 89) and without (N = 684) pre-registration. The distributions contain all effects that were extracted as or could be transformed into a correlation coefficient r.CC BY · PubMed Central
FIGURE 2 Cohen’s conventions for small, medium, and large effects compared with empirical effects (lower, grand, and upper median) from studies published with and without pre-registration.
FIGURE 2 Cohen’s conventions for small, medium, and large effects compared with empirical effects (lower, grand, and upper median) from studies published with and without pre-registration.CC BY · PubMed Central
FIGURE 3 Medians (with 95% bootstrap confidence intervals) of effects published without pre-registration (absolute values) for the nine Social Sciences Citation Index psychological sub-disciplines. The bars contain all effects that were extracted as or could be transformed into a correlation coefficient r. The vertical line is the grand median.
FIGURE 3 Medians (with 95% bootstrap confidence intervals) of effects published without pre-registration (absolute values) for the nine Social Sciences Citation Index psychological sub-disciplines. The bars contain all effects that were extracted as or could be transformed into a correlation coefficient r. The vertical line is the grand median.CC BY · PubMed Central
FIGURE 4 Relationship (Loess curve1) between sample size and effect size r for studies published without pre-registration. N = 684.
FIGURE 4 Relationship (Loess curve1) between sample size and effect size r for studies published without pre-registration. N = 684.CC BY · PubMed Central
FIGURE 5 Relationship (Loess curve) between sample size and effect size r for studies published with pre-registration. N = 89.
FIGURE 5 Relationship (Loess curve) between sample size and effect size r for studies published with pre-registration. N = 89.CC BY · PubMed Central
FIGURE 6 Relationship (Loess curve) between year of publication and effect size r for studies published without pre-registration. N = 684.
FIGURE 6 Relationship (Loess curve) between year of publication and effect size r for studies published without pre-registration. N = 684.CC BY · PubMed Central

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 43 references

Research works printed but not linked: 5

  1. 2001, Publication Manual of the American Psychological Association
  2. 2010, Publication Manual of the American Psychological Association
  3. Cohen, 1977, Statistical Power Analysis for the Behavioral Sciences (revised Edn.).
  4. Keppel, 1991, Design and Analysis: A Researcher’s Handbook.
  5. Lehrer, 2010, The Truth Wears Off., 29

Cited by

Sorted by citations

See all 774

Related

Metadata

Authors
Schäfer, T., & Schwarz, M. A.
Affiliations
Chemnitz University of Technology (DE)
Corresponding author
Schäfer, Thomas
ORCID
Venue
Frontiers in Psychology
Publisher
Frontiers Media SA
Published
2019-04-11
Volume
10
Pages
813-813
ISSN
1664-1078 (ISSN-L)
DOI
10.3389/fpsyg.2019.00813
PMC
PMC6470248
OpenAlex
W2937326640
Cited by
774
References
43
Citations by year
2026: 73 · 2025: 105 · 2024: 101 · 2023: 123 · 2022: 149 · 2021: 127 · 2020: 75 · 2019: 18
FWCI
61.73
Type
journal article
Language
en
Open access
Licence
Locations
  • Frontiers in Psychology (publishedVersion, cc-by)
  • Europe PMC (PubMed Central) (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
  • Qucosa (Saxon State and University Library Dresden) (submittedVersion, cc-by)
Funders
  • Technische Universität Chemnitz
  • Deutsche Forschungsgemeinschaft
Topics
  • Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
  • Social Sciences › Psychology › Experimental and Cognitive Psychology › Psychology Research and Bibliometrics
  • Social Sciences › Decision Sciences › Management Science and Operations Research › Psychometric Methodologies and Testing
Keywords
effect size, Cohen's d, publication bias, questionable research practices, preregistration, effect size inflation, statistical power

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review