CoChat
Sign Up
Cited by 5,553Meta-analysis and systematic reviewsStatistical Methods in EpidemiologyEducational Research and Methods

Using Effect Size—or Why the P Value Is Not Enough

Open access
Sullivan, Gail M., Feinn, Richard·2012·Journal of Graduate Medical Education
DOI 10.4300/jgme-d-12-00156.1PMC PMC3444174
Open this paper in CoChat — free accountOpen PDF

Abstract

These statements about the importance of effect sizes were made by two of the most influential statistician-researchers of the past half-century. Yet many submissions to Journal of Graduate Medical Education omit mention of the effect size in quantitative studies while prominently displaying the P value. In this paper, we target readers with little or no statistical background in order to encourage you to improve your comprehension of the relevance of effect size for planning, analyzing, reporting, and understanding education research studies.In medical education research studies that compare different educational interventions, effect size is the magnitude of the difference between groups. The absolute effect size is the difference between the average, or mean, outcomes in two different intervention groups. For example, if an educational intervention resulted in the improvement of subjects' examination scores by an average total of 15 of 50 questions as compared to that of another intervention, the absolute effect size is 15 questions or 3 grade levels (30%) better on the examination. Absolute effect size does not take into account the variability in scores, in that not every subject achieved the average outcome.In another example, residents' self-assessed confidence in performing a procedure improved an average of 0.4 point on a Likert-type scale ranging from 1 to 5, after simulation training. While the absolute effect size in the first example appears clear, the effect size in the second example is less apparent. Is a 0.4 change a lot or trivial? Accounting for variability in the measured improvement may aid in interpreting the magnitude of the change in the second example.Thus, effect size can refer to the raw difference between group means, or absolute effect size, as well as standardized measures of effect, which are calculated to transform the effect to an easily understood scale. Absolute effect size is useful when the variables under study have intrinsic meaning (eg, number of hours of sleep). Calculated indices of effect size are useful when the measurements have no intrinsic meaning, such as numbers on a Likert scale; when studies have used different scales so no direct comparison is possible; or when effect size is examined in the context of variability in the population under study.Calculated effect sizes can also quantitatively compare results from different studies and thus are commonly used in meta-analyses.The effect size is the main finding of a quantitative study. While a P value can inform the reader whether an effect exists, the P value will not reveal the size of the effect. In reporting and interpreting studies, both the substantive significance (effect size) and statistical significance (P value) are essential results to be reported.For this reason, effect sizes should be reported in a paper's Abstract and Results sections. In fact, an estimate of the effect size is often needed before starting the research endeavor, in order to calculate the number of subjects likely to be required to avoid a Type II, or β, error, which is the probability of concluding there is no effect when one actually exists. In other words, you must determine what number of subjects in the study will be sufficient to ensure (to a particular degree of certainty) that the study has acceptable power to support the null hypothesis. That is, if no difference is found between the groups, then this is a true finding.Statistical significance is the probability that the observed difference between two groups is due to chance. If the P value is larger than the alpha level chosen (eg, .05), any observed difference is assumed to be explained by sampling variability. With a sufficiently large sample, a statistical test will almost always demonstrate a significant difference, unless there is no effect whatsoever, that is, when the effect size is exactly zero; yet very small differences, even if significant, are often meaningless. Thus, reporting only the significant P value for an analysis is not adequate for readers to fully understand the results.For example, if a sample size is 10 000, a significant P value is likely to be found even when the difference in outcomes between groups is negligible and may not justify an expensive or time-consuming intervention over another. The level of significance by itself does not predict effect size. Unlike significance tests, effect size is independent of sample size. Statistical significance, on the other hand, depends upon both sample size and effect size. For this reason, P values are considered to be confounded because of their dependence on sample size. Sometimes a statistically significant result means only that a huge sample size was used.3A commonly cited example of this problem is the Physicians Health Study of aspirin to prevent myocardial infarction (MI).4 In more than 22 000 subjects over an average of 5 years, aspirin was associated with a reduction in MI (although not in o

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.4300/jgme-d-12-00156.1

Cites

0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • The effect size for the procedural knowledge difference was medium (r=0.34).

    “Concerning effect size, the r-value was 0.34.This would be counted as medium effect size (Sullivan & Feinn, 2012).”

    From Teaching and Learning Science Using YouTube Videos and Discovery Learning in Primary School (2020) · cites this paper as Sullivan & Feinn, 2012

    Table 1 in “How to Calculate Effect Size” gives Pearson’s r benchmarks of 0.2 for small and 0.5 for medium; it does not classify r = 0.34 as medium. The “Additional information” section cautions that these categories are ballpark guides, so the paper provides a framework for interpretation but does not establish the citing sentence’s specific classification. (LLM, conf=0.83)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 1

  1. Coe R. It's the effect size, stupid: what ''effect size'' is and why it is important. Paper presented at the 2002 Annual Conference of the British Educational Research Association, University of Exeter, Exeter, Devon, England, September 12-14, 2002. http://www.leeds.ac.uk/educol/ documents/00002182.htm. Accessed March 23, 2012.

Cited by

Sorted by citations

Authors
Sullivan, G. M., & Feinn, R.
Affiliations
University of Connecticut (US)
Corresponding author
Sullivan, Gail M.
ORCID
Venue
Journal of Graduate Medical Education
Publisher
Journal of Graduate Medical Education
Published
2012-09
Volume
4
Issue
3
Pages
279-282
ISSN
  • 1949-8349 (ISSN-L)
  • 1949-8357
DOI
10.4300/jgme-d-12-00156.1
PMC
PMC3444174
OpenAlex
W2107026277
Cited by
5,553
References
6
Citations by year
2026: 607 · 2025: 877 · 2024: 759 · 2023: 547 · 2022: 574 · 2021: 499 · 2020: 416 · 2019: 419 · 2018: 320 · 2017: 222 · 2016: 215 · 2015: 83 · 2014: 9 · 2013: 4 · 2012: 2
FWCI
35.75
Type
journal article
Language
en
Open access
Locations
  • Journal of Graduate Medical Education (publishedVersion)
  • Europe PMC (PubMed Central) (publishedVersion)
  • PubMed Central (publishedVersion)
  • PubMed (publishedVersion)
Topics
  • Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
  • Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
  • Social Sciences › Education › Educational Research and Methods
Keywords
effect size, p-values, statistical significance, substantive significance, standardized mean difference, medical education research, educational intervention, meta-analysis, simulation-based training, resident confidence, test scores, interpretation of study results, graduate medical education, reporting guidelines

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review