The DOI matches its Crossref record; 104 of the 104 references it cites were found in a registry or an index.
A comprehensive review of effect size reporting and interpreting practices in academic journals in education and psychology.
Open accessAbstract
Null hypothesis significance testing has dominated quantitative research in education and psychology. However, the statistical significance of a test as indicated by a p-value does not speak to the practical significance of the study. Thus, reporting effect size to supplement p-value is highly recommended by scholars, journal editors, and academic associations. As a measure of practical significance, effect size quantifies the size of mean differences or strength of associations and directly answers the research questions. Furthermore, a comparison of effect sizes across studies facilitates meta-analytic assessment of the effect size and accumulation of knowledge. In the current comprehensive review, we investigated the most recent effect size reporting and interpreting practices in 1,243 articles published in 14 academic journals from 2005 to 2007. Overall, 49 % of the articles reported effect size—57 % of which interpreted effect size. As an empirical study for the sake of good research methodology in education and psychology, in the present study we provide an illustrative example of reporting and interpreting effect size in a published study. Furthermore, a 7-step guideline for quantitative researchers is also summarized along with some recommended resources on how to understand and interpret effect size.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Statistical power analysis for the behavioral sciences
(1990) · Computers, Environment and Urban Systems · doi:10.1016/0198-9715(90)90050-4 · Cited by 65,269
Statistical Power Analysis for the Behavioral Sciences (2nd ed.).
Lachenbruch, P. A., & Cohen, J. (1989) · Journal of the American Statistical Association · doi:10.2307/2290095 · Cited by 37,395
(1990) · Cited by 24,217
Statistical Methods for Meta-Analysis
Wasserman, S., Hedges, L. V., & Olkin, I. (1988) · Journal of Educational Statistics · doi:10.2307/1164953 · Cited by 10,326
(1990) · Cited by 8,425
Distribution Theory for Glass's Estimator of Effect Size and Related Estimators
Hedges, L. V. (1981) · Journal of Educational Statistics · doi:10.2307/1164588 · Cited by 5,695
Primary, Secondary, and Meta-Analysis of Research
GLASS, G. V. (1976) · Educational Researcher · doi:10.3102/0013189x005010003 · Cited by 4,890
Enhancing the quality and credibility of qualitative analysis.
(1999) · PubMed Central · Cited by 4,133
Cohen, J. (1994) · American Psychologist · doi:10.1037/0003-066x.49.12.997 · Cited by 4,003
Meta-analysis in social research
(1982) · Evaluation and Program Planning · doi:10.1016/0149-7189(82)90011-8 · Cited by 3,853
Effect size, confidence interval and statistical significance: a practical guide for biologists
Nakagawa, S., & Cuthill, I. C. (2007) · Biological Reviews · doi:10.1111/j.1469-185x.2007.00027.x · Cited by 3,835
The Handbook of Research Synthesis.
Howel, D., Cooper, H., & Hedges, L. V. (1995) · Biometrics · doi:10.2307/2533295 · Cited by 3,642
Statistical methods in psychology journals: Guidelines and explanations.
Wilkinson, L., & Task Force on Statistical Inference (1999) · American Psychologist · doi:10.1037/0003-066x.54.8.594 · Cited by 2,791
The Abuse of Power: The Pervasive Fallacy of Power Calculations for Data Analysis
Hoenig, J. M., & Heisey, D. M. (2001) · The American Statistician · doi:10.1198/000313001300339897 · Cited by 2,046
Null Hypothesis Testing: Problems, Prevalence, and an Alternative
Anderson, D. R., Burnham, K. P., & Thompson, W. L. (2000) · The Journal of Wildlife Management · doi:10.2307/3803199 · Cited by 1,930
Practical significance: a concept whose time has come
Kirk, R. E. (1996) · Educational and Psychological Measurement · doi:10.1177/0013164496056005002 · Cited by 1,778 · cited 23×
“On the basis of the classification of effect size measures by Kirk (1996), the effect size measures were coded into three categories: (a) measures of mean differences that include, but are not limited to, Cohen’s d, f, h; (b) measures of strength of association that include, but are not limit to,…”
BackgroundMethodResultInfluentialInference by Eye: Confidence Intervals and How to Read Pictures of Data.
Cumming, G., & Finch, S. (2005) · American Psychologist · doi:10.1037/0003-066x.60.2.170 · Cited by 1,449 · cited 1×
“…placed in the context of additional information, such as confidence intervals and effect size (AERA, 2006; Ander- son et al., 2000; APA, 2001, 2010; Cumming & Finch, 2002; Fan, 2001; Kirk, 1996, 2001; McLean & Ernest, 1998; Snyder & Lawson, 1993; Thompson, 1996, 1997, 2002b; Vacha-Haase &…”
Publication Manual of the American Psychological Association (4th ed.).
Gullickson, T. (1996) · Contemporary Psychology: A Journal of Reviews · doi:10.1037/004700 · Cited by 1,240
Designing Experiments and Analysing Data: A Model Comparison Perspective.
Freeman, G. H., Maxwell, S. E., & Delaney, H. D. (1991) · Biometrics · doi:10.2307/2532173 · Cited by 1,229
Beyond significance testing: Reforming data analysis methods in behavioral research.
Kline, R. B. (2004) · doi:10.1037/10693-000 · Cited by 1,034
Best Practices in Quantitative Methods
Osborne, J. (2008) · Sage eBooks · doi:10.4135/9781412995627 · Cited by 942
The case against statistical significance testing
Carver, R. (1978) · Harvard Educational Review · doi:10.17763/haer.48.3.t490261645281841 · Cited by 931
Contrasts and Effect Sizes in Behavioral Research: A Correlational Approach
Rosenthal, R., Rosnow, R. L., & Rubin, D. B. (1999) · Cited by 854
Schmidt, F. L. (1996) · Psychological Methods · doi:10.1037//1082-989x.1.2.115 · Cited by 783
What If There Were No Significance Tests?
(2013) · Psychology Press eBooks · doi:10.4324/9781315827353 · Cited by 667
Keselman, H. J., Huberty, C. J., Lix, L. M., Olejnik, S., Cribbie, R. A., Donahue, B., Kowalchuk, R. K., Lowman, L. L., Petoskey, M. D., Keselman, J. C., et al. (1998) · Review of Educational Research · doi:10.2307/1170601 · Cited by 627
Things I have learned (so far).
Cohen, J. (1992) · Methodological issues & strategies in clinical research. · doi:10.1037/10109-028 · Cited by 553
How to Estimate and Interpret Various Effect Sizes.
Vacha-Haase, T., & Thompson, B. (2004) · Journal of Counseling Psychology · doi:10.1037/0022-0167.51.4.473 · Cited by 539
Thompson, B. (2002) · Educational Researcher · doi:10.3102/0013189x031003025 · Cited by 538
Cumming, G., & Finch, S. (2001) · Educational and Psychological Measurement · doi:10.1177/0013164401614002 · Cited by 528
Hojat, M., & Xu, G. (2004) · Advances in Health Sciences Education · doi:10.1023/b:ahse.0000038173.00909.f6 · Cited by 427
Some Difficulties of Interpretation Encountered in the Application of the Chi-Square Test
Berkson, J. (1938) · Journal of the American Statistical Association · doi:10.1080/01621459.1938.10502329 · Cited by 411
Thomas, L. (1997) · Conservation Biology · doi:10.1046/j.1523-1739.1997.96102.x · Cited by 403
Thompson, B. (2002) · Journal of Counseling & Development · doi:10.1002/j.1556-6678.2002.tb00167.x · Cited by 398
Effect sizes, confidence intervals, and confidence intervals for effect sizes
Thompson, B. (2007) · Psychology in the Schools · doi:10.1002/pits.20234 · Cited by 386
A systematic review of effect size in software engineering experiments
Kampenes, V. B., Dybå, T., Hannay, J. E., & Sjøberg, D. I. K. (2007) · Information and Software Technology · doi:10.1016/j.infsof.2007.02.015 · Cited by 370
AERA Editorial Policies regarding Statistical Significance Testing: Three Suggested Reforms
Thompson, B. (1996) · Educational Researcher · doi:10.2307/1176337 · Cited by 339
Statistical significance tests.
Cox, D. (1982) · British Journal of Clinical Pharmacology · doi:10.1111/j.1365-2125.1982.tb01987.x · Cited by 334
Significance Tests Die Hard: The Amazing Persistence of a Probabilistic Misconception
Falk, R., & Greenbaum, C. W. (1995) · Theory & Psychology · doi:10.1177/0959354395051004 · Cited by 279
The Industrial-Organizational Psychologist: (578882011-001)
(2009) · PsycEXTRA Dataset · doi:10.1037/e578882011-001 · Cited by 274
Smithson, M. (2001) · Educational and Psychological Measurement · doi:10.1177/00131640121971392 · Cited by 274
Evaluating Results Using Corrected and Uncorrected Effect Size Estimates
Snyder, P., & Lawson, S. (1993) · The Journal of Experimental Education · doi:10.1080/00220973.1993.10806594 · Cited by 265
Colegrave, N. (2003) · Behavioral Ecology · doi:10.1093/beheco/14.3.446 · Cited by 250
Measures of effect size for categorical data.
Fleiss, J. L. (1994) · Cited by 248
Statistical Significance and Effect Size in Education Research: Two Sides of a Coin
Fan, X. (2001) · The Journal of Educational Research · doi:10.1080/00220670109598763 · Cited by 220
Effect-Size Measures and Meta-Analytic Thinking in Counseling Psychology Research
Henson, R. K. (2006) · The Counseling Psychologist · doi:10.1177/0011000005283558 · Cited by 199
What Statistical Significance Testing Is, and What It Is Not
Shaver, J. P. (1993) · The Journal of Experimental Education · doi:10.1080/00220973.1993.10806592 · Cited by 196
Promoting good statistical practices: Some suggestions
Kirk, R. E. (2001) · Educational and Psychological Measurement · doi:10.1177/00131640121971185 · Cited by 184
The Use of Statistical Significance Tests in Research: Bootstrap and Other Alternatives
Thompson, B. (1993) · The Journal of Experimental Education · doi:10.1080/00220973.1993.10806596 · Cited by 183
Reporting effect size estimates in school psychology research
Volker, M. A. (2006) · Psychology in the Schools · doi:10.1002/pits.20176 · Cited by 182
On the logic and purpose of significance testing.
Cortina, J. M., & Dunlap, W. P. (1997) · Psychological Methods · doi:10.1037/1082-989x.2.2.161 · Cited by 166 · cited 2×
“Furthermore, blindly interpreting confidence intervals only against the standard of whether zero point is included is nothing more than rejecting or not rejecting on the basis of p-value (Cortina & Dunlap, 1997; Thompson, 1999a, 2008).”
BackgroundJitendra, A. K., Griffin, C. C., Haria, P., Leh, J. M., Adams, A., & Kaduvettoor, A. (2007) · Journal of Educational Psychology · doi:10.1037/0022-0663.99.1.115 · Cited by 163
On the Post Hoc Power in Testing Mean Differences
Yuan, K.-H., & Maxwell, S. (2005) · Journal of Educational and Behavioral Statistics · doi:10.3102/10769986030002141 · Cited by 160
Yates, F. (1951) · Journal of the American Statistical Association · doi:10.1080/01621459.1951.10500764 · Cited by 154
Misuse of statistical tests in three decades of psychotherapy research.
Dar, R., Serlin, R. C., & Omer, H. (1994) · Journal of Consulting and Clinical Psychology · doi:10.1037//0022-006x.62.1.75 · Cited by 152
Schneider, B. (2006) · Cited by 149
Trusty, J., Thompson, B., & Petrocelli, J. V. (2004) · Journal of Counseling & Development · doi:10.1002/j.1556-6678.2004.tb00291.x · Cited by 144
Vaske, J. J. (2002) · Human Dimensions of Wildlife · doi:10.1080/10871200214752 · Cited by 142
Thompson, B. (1999) · Theory & Psychology · doi:10.1177/095935439992006 · Cited by 138
Reporting Practices and APA Editorial Policies Regarding Statistical Significance and Effect Size
Vacha-haase, T., Nilsson, J. E., Reetz, D. R., Lance, T. S., & Thompson, B. (2000) · Theory & Psychology · doi:10.1177/0959354300103006 · Cited by 138
Statistical Significance: Rationale, Validity and Utility
Nayak, T. K., & Chow, S. L. (1998) · Technometrics · doi:10.2307/1270661 · Cited by 136
Significance test or effect size?
Chow, S. L. (1988) · Psychological Bulletin · doi:10.1037/0033-2909.103.1.105 · Cited by 129
Enhancing mathematical problem solving among third-grade students with schema-based instruction.
Fuchs, L. S., Fuchs, D., Prentice, K., Hamlett, C. L., Finelli, R., & Courey, S. J. (2004) · Journal of Educational Psychology · doi:10.1037/0022-0663.96.4.635 · Cited by 126
Reflections on Statistical and Substantive Significance, with a Slice of Replication
Robinson, D. H., & Levin, J. R. (1997) · Educational Researcher · doi:10.2307/1176544 · Cited by 125
Planning Educational Research: Determining the Necessary Sample Size
Olejnik, S. F. (1984) · The Journal of Experimental Education · doi:10.1080/00220973.1984.10806360 · Cited by 123
Fuchs, L. S., Fuchs, D., Finelli, R., Courey, S. J., & Hamlett, C. L. (2004) · American Educational Research Journal · doi:10.3102/00028312041002419 · Cited by 112
Statistical Sirens: The Allure of Nonparametrics
Johnson, D. H. (1995) · Ecology · doi:10.2307/1940733 · Cited by 110
The Role of Statistical Significance Testing In Educational Research
McLean, J. E., & Ernest, J. M. (1998) · Cited by 105
Group Overlap as a Basis for Effect Size
Huberty, C. J., & Lowman, L. L. (2000) · Educational and Psychological Measurement · doi:10.1177/0013164400604004 · Cited by 101
On the Past and Future of Null Hypothesis Significance Testing
Robinson, D. H., & Wainer, H. (2002) · Journal of Wildlife Management · doi:10.2307/3803158 · Cited by 90
Statistical Significance and Effect Size Reporting: Portrait of a Possible Future.
Thompson, B. (1998) · Cited by 89
Zientek, L. R., Capraro, M. M., & Capraro, R. M. (2008) · Educational Researcher · doi:10.3102/0013189x08319762 · Cited by 75
Thompson, B. (1999) · Exceptional Children · doi:10.1177/001440299906500304 · Cited by 71
Henson, R. K., & Smith, A. D. (2000) · Journal of research and development in education · Cited by 70
Sleep paralysis-associated sensed presence as a possible manifestation of social anxiety.
Simard, V., & Nielsen, T. (2005) · Dreaming · doi:10.1037/1053-0797.15.4.245 · Cited by 68
Thompson, B. (2008) · Sage eBooks · doi:10.4135/9781412995627.d21 · Cited by 64
Case Studies for Evaluating Statistical Significance in Group Designs
Meline, T. J., & Schmitt, J. F. (1997) · American Journal of Speech-Language Pathology · doi:10.1044/1058-0360.0601.33 · Cited by 63
Rejoinder: Editorial Policies Regarding Statistical Significance Tests: Further Comments
Thompson, B. (1997) · Educational Researcher · doi:10.3102/0013189x026005029 · Cited by 63
Statistical Significance Tests, Effect Size Reporting and the Vain Pursuit of Pseudo-Objectivity
Thompson, B. (1999) · Theory & Psychology · doi:10.1177/095935439992007 · Cited by 63
Supplementing Tests of Statistical Significance: Variation Accounted For
Young, M. A. (1993) · Journal of Speech Language and Hearing Research · doi:10.1044/jshr.3604.644 · Cited by 56
In praise of brilliance: Where that praise really belongs.
Thompson, B. (1998) · American Psychologist · doi:10.1037//0003-066x.53.7.799 · Cited by 54
Research in the Schools, 2000.
McLean, J. E., & Kaufman, A. S. (2000) · Research in the schools · Cited by 51
Statistical Significance Testing Practices in The Journal of Experimental Education
Thompson, B., & Snyder, P. (1997) · The Journal of Experimental Education · doi:10.1080/00220979709601396 · Cited by 49
Treatments of Effect Sizes and Statistical Significance Tests in Textbooks
Capraro, R. M., & Capraro, M. M. (2002) · Educational and Psychological Measurement · doi:10.1177/001316402236877 · Cited by 49
An Analysis of the Use of Statistical Testing in Communication Research
Katzer, J., & Sodt, J. (1973) · Journal of Communication · doi:10.1111/j.1460-2466.1973.tb00947.x · Cited by 49
Policy for the Unpredictable (Uncertainty Research and Policy)
Glass, G. V. (1979) · Educational Researcher · doi:10.2307/1175901 · Cited by 45
Effect Size Reporting Practices in Published Articles
Alhija, F. N.-A., & Levy, A. (2008) · Educational and Psychological Measurement · doi:10.1177/0013164408315266 · Cited by 45
Statistical Significance Reporting: Current Trends and Uses in MECD.
Vacha-haase, T., & Nilsson, J. E. (1998) · Measurement and Evaluation in Counseling and Development · doi:10.1080/07481756.1998.12068949 · Cited by 44
Thompson, B. (1999) · The Journal of Psychology · doi:10.1080/00223989909599728 · Cited by 44
Effect Size Use in Studies of Learning Disabilities
Ives, B. (2003) · Journal of Learning Disabilities · doi:10.1177/00222194030360060101 · Cited by 39
May, H., & Supovitz, J. A. (2006) · Educational Evaluation and Policy Analysis · doi:10.3102/01623737028003231 · Cited by 35
Sun, S., Pan, W., & Wang, L. L. (2011) · Methodology · doi:10.1027/1614-2241/a000025 · Cited by 32
Self-citationConfidence Intervals and the Scientific Method
Serlin, R. C. (1993) · The Journal of Experimental Education · doi:10.1080/00220973.1993.10806595 · Cited by 31
Effect-Size Reporting Practices in AJSLP and Other ASHA Journals, 1999–2003
Meline, T. J., & Wang, B. (2004) · American Journal of Speech-Language Pathology · doi:10.1044/1058-0360(2004/021 · Cited by 30
The Impact of Data-Analysis Methods on Cumulative Research Knowledge
Schmidt, F. M., & Hunter, J. E. (1995) · Evaluation & the Health Professions · doi:10.1177/016327879501800405 · Cited by 28
Snyder, P., & Thompson, B. (1998) · School Psychology Quarterly · doi:10.1037/h0088990 · Cited by 26
Vachon, F., Tremblay, S., & Jones, D. M. (2007) · Journal of Experimental Psychology Human Perception & Performance · doi:10.1037/0096-1523.33.2.330 · Cited by 24
Debunking the myth of the “highly significant” result:Effect sizes in gifted education research
Plucker, J. A. (1997) · Roeper Review · doi:10.1080/02783199709553873 · Cited by 24
Two steps forward, one step back:Effect size reporting in gifted education research from 1995–2000
Paul, K. M., & Plucker, J. A. (2004) · Roeper Review · doi:10.1080/02783190409554244 · Cited by 20
Vacha‐Haase, T., & Ness, C. M. (1999) · Professional Psychology Research and Practice · doi:10.1037//0735-7028.30.1.104 · Cited by 19
Comment on Significance Testing
Biskin, B. H. (1998) · Measurement and Evaluation in Counseling and Development · doi:10.1080/07481756.1998.12068950 · Cited by 14
Highly significant findings in psychology: A power and effect size survey
Ward, R. M. (2002) · doi:10.23860/diss-1834 · Cited by 9
An Introduction to Confidence Intervals for Both Statistical Estimates and Effect Sizes.
Capraro, M. M. (2002) · Cited by 7
Reporting Effect Size: The Road Less Traveled.
McMillan, J. H., Lawson, S. R., Lewis, K. L., & Snyder, A. (2002) · Cited by 6
Cited by
Un sistema de clasificación de los diseños de investigación en psicología
Ato, M., López-García, J. J., & Benavente, A. (2013) · Anales de Psicología · doi:10.6018/analesps.29.3.178511 · Cited by 2,254
Cliff´s Delta Calculator: A non-parametric effect size program for two groups of observations
Macbeth, G., Razumiejczyk, E., & Ledesma, R. D. (2010) · Universitas Psychologica · doi:10.11144/javeriana.upsy10-2.cdcp · Cited by 442
Plonsky (2013) · doi:10.1017/s0272263113000399 · Cited by 347
Beyond small, medium, or large: points of consideration when interpreting effect sizes
Bakker, A., Cai, J., English, L., Kaiser, G., Mesa, V., & Van Dooren, W. (2019) · Educational Studies in Mathematics · doi:10.1007/s10649-019-09908-4 · Cited by 302
(2021) · Research Design in Clinical Psychology · doi:10.1017/9781108993647.010 · Cited by 276
Metadata
- Authors
- Sun, S., Pan, W., & Wang, L. L.
- Venue
- Journal of Educational Psychology
- Published
- 2010-11
- Volume
- 102
- Issue
- 4
- Pages
- 989-1004
- ISSN
- 0022-0663 (ISSN-L)
- 1939-2176
- OpenAlex
- W2108656311
- Cited by
- 342
- References
- 104
- FWCI
- 16.87
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.