The DOI matches its Crossref record; 124 of the 124 references it cites were found in a registry or an index.
Estimating the Difference Between Published and Unpublished Effect Sizes: A Meta-Review
Abstract
Practitioners and policymakers rely on meta-analyses to inform decision making around the allocation of resources to individuals and organizations. It is therefore paramount to consider the validity of these results. A well-documented threat to the validity of research synthesis results is the presence of publication bias, a phenomenon where studies with large and/or statistically significant effects, relative to studies with small or null effects, are more likely to be published. We investigated this phenomenon empirically by reviewing meta-analyses published in top-tier journals between 1986 and 2013 that quantified the difference between effect sizes from published and unpublished research. We reviewed 383 meta-analyses of which 81 had sufficient information to calculate an effect size. Results indicated that published studies yielded larger effect sizes than those from unpublished studies ([Formula: see text] = 0.18, 95% confidence interval [0.10, 0.25]). Moderator analyses revealed that the difference was larger in meta-analyses that included a wide range of unpublished literature. We conclude that intervention researchers require continued support to publish null findings and that meta-analyses should include unpublished studies to mitigate the potential bias from publication status.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Citations
Cites
Bias in meta-analysis detected by a simple, graphical test
Egger, M., Smith, G. D., Schneider, M., & Minder, C. (1997) · BMJ · doi:10.1136/bmj.315.7109.629 · Cited by 56,722
Quantifying heterogeneity in a meta‐analysis
Higgins, J. P. T., & Thompson, S. G. (2002) · Statistics in Medicine · doi:10.1002/sim.1186 · Cited by 36,763
Borenstein, M., Hedges, L. V., Higgins, J. P. T., & Rothstein, H. R. (2009) · doi:10.1002/9780470743386 · Cited by 14,225
Duval, S., & Tweedie, R. (2000) · Biometrics · doi:10.1111/j.0006-341x.2000.00455.x · Cited by 14,099
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · PLoS Medicine · doi:10.1371/journal.pmed.0020124 · Cited by 10,505
Deci, E. L., Koestner, R., & Ryan, R. M. (1999) · Psychological Bulletin · doi:10.1037/0033-2909.125.6.627 · Cited by 7,176
The job satisfaction–job performance relationship: A qualitative and quantitative review.
Judge, T. A., Thoresen, C. J., Bono, J. E., & Patton, G. K. (2001) · Psychological Bulletin · doi:10.1037/0033-2909.127.3.376 · Cited by 4,760
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · CHANCE · doi:10.1080/09332480.2005.10722754 · Cited by 4,702 · cited 2×
“208 published literature detecting a statistically significant effect is often the rule rather than exception (Ioannidis, 2005; Ioannidis & Trikalinos, 2007).”
BackgroundThe Handbook of Research Synthesis and Meta-Analysis
(2019) · doi:10.7758/9781610448864 · Cited by 4,553
Perceived discrimination and health: A meta-analytic review.
Pascoe, E. A., & Smart Richman, L. (2009) · Psychological Bulletin · doi:10.1037/a0016059 · Cited by 4,427
Research works printed but not linked: 12
- 10.1037/00332909.125.1.64 · doi:10.1037/00332909.125.1.64
- 10.1037//0033-2909.128.2.203 · DOI · Paper page
- 10.1037/0033-2909.104.2.226 · DOI · Paper page
- Hopewell S., 2005, Publication bias in meta-analysis: Prevention, assessment and adjustments, 48
- 10.1037/0003-066X.48.12.1181 · DOI · Paper page
- Lipsey M. W., 2001, Applied social research methods: Vol. 49. Practical meta-analysis
- 10.3102/00346543065002129 · doi:10.3102/00346543065002129 · Paper page
- 10.1037/0033-2909.124.1.22 · DOI · Paper page
- Sterne J. A. C., 2005, Publication bias in meta-analysis: Prevention, assessment and adjustments, 75
- Sutton A. J., 2009, The handbook of research synthesis and meta-analysis, 435
- *Wood W., 1994, Psychological Bulletin, 115, 323, 10.1037/0033-2909.115.3.323 · doi:10.1037/0033-2909.115.3.323 · Paper page
- 10.1037/0033-2909.109.3.371 · DOI · Paper page
Cited by
Environmental education outcomes for conservation: A systematic review
Ardoin, N. M., Bowers, A. W., & Gaillard, E. (2020) · Biological Conservation · doi:10.1016/j.biocon.2019.108224 · Cited by 924
Long-term outcomes of childhood sexual abuse: an umbrella review
Hailes, H. P., Yu, R., Danese, A., & Fazel, S. (2019) · The Lancet Psychiatry · doi:10.1016/s2215-0366(19)30286-x · Cited by 827
Cipriano, C., Strambler, M. J., Naples, L. H., Ha, C., Kirk, M., Wood, M., Sehgal, K., Zieher, A. K., Eveleigh, A., McCarthy, M., et al. (2023) · Child Development · doi:10.1111/cdev.13968 · Cited by 636
“Further, given evidence that average effect sizes differ based on the nature of a publication (Chow & Ekholm, 2018; Polanin et al., 2016), we compared overall effect sizes for published to unpub-lished studies.”
Methodological guidance paper: High-quality meta- analysis in a systematic review
Pigott, T. D., & Polanin, J. R. (2020) · Review of Educational Research · doi:10.3102/0034654319877153 · Cited by 568
“Many researchers have documented the existence of publication bias in medicine and psychology (Dickersin, 1990; Ferguson & Brannick, 2012) as well as in education (Polanin et al., 2016).”
BackgroundSelf-citationEffects of flipping the classroom on learning outcomes and satisfaction: A meta-analysis
van Alten, D. C. D., Phielix, C., Janssen, J., & Kester, L. (2019) · Educational Research Review · doi:10.1016/j.edurev.2019.05.003 · Cited by 546
“In addition, Spanjers et al. (2015) found in their meta-analysis on blended learning that the inclusion of quizzes positively moderated the effects of blended learning on learning outcomes and satisfaction.”
Background
Metadata
- Authors
- Polanin, J. R., Tanner-Smith, E. E., & Hennessy, E. A.
- Affiliations
- Vanderbilt University (US)
- Vanderbilt University
- Venue
- Review of Educational Research
- Published
- 2016-03
- Volume
- 86
- Issue
- 1
- Pages
- 207-236
- ISSN
- 0034-6543 (ISSN-L)
- 1935-1046
- OpenAlex
- W2126293099
- Cited by
- 234
- References
- 123
- Citations by year
- 2026: 9 · 2025: 18 · 2024: 20 · 2023: 26 · 2022: 24 · 2021: 45 · 2020: 25 · 2019: 27 · 2018: 17 · 2017: 11 · 2016: 7
- FWCI
- 11.23
- Type
- journal article
- Language
- en
- Open access
- closed
- Licence
- journals.sagepub.com/page/policies/text-and-data-mining-license (tdm)
- Locations
- Review of Educational Research (publishedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Psychology Research and Bibliometrics
- Social Sciences › Library and Information Sciences › scientometrics and bibliometrics research
- Keywords
- publication bias, unpublished research, meta-analysis, negative results, research validity, publication status, evidence synthesis, intervention research
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Estimating the Difference Between Published and Unpublished Effect Sizes: A Meta-Review. https://app.cochat.ai/papers/10.3102/0034654315582067
- The paper
- Polanin, J. R., Tanner-Smith, E. E., & Hennessy, E. A. (2016). Estimating the Difference Between Published and Unpublished Effect Sizes: A Meta-Review. Review of Educational Research. https://doi.org/10.3102/0034654315582067
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.