The DOI matches its Crossref record; 124 of the 124 references it cites were found in a registry or an index.
Estimating the Difference Between Published and Unpublished Effect Sizes: A Meta-Review
Abstract
Practitioners and policymakers rely on meta-analyses to inform decision making around the allocation of resources to individuals and organizations. It is therefore paramount to consider the validity of these results. A well-documented threat to the validity of research synthesis results is the presence of publication bias, a phenomenon where studies with large and/or statistically significant effects, relative to studies with small or null effects, are more likely to be published. We investigated this phenomenon empirically by reviewing meta-analyses published in top-tier journals between 1986 and 2013 that quantified the difference between effect sizes from published and unpublished research. We reviewed 383 meta-analyses of which 81 had sufficient information to calculate an effect size. Results indicated that published studies yielded larger effect sizes than those from unpublished studies ([Formula: see text] = 0.18, 95% confidence interval [0.10, 0.25]). Moderator analyses revealed that the difference was larger in meta-analyses that included a wide range of unpublished literature. We conclude that intervention researchers require continued support to publish null findings and that meta-analyses should include unpublished studies to mitigate the potential bias from publication status.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Citations
Cites
Bias in meta-analysis detected by a simple, graphical test
Egger, M., Smith, G. D., Schneider, M., & Minder, C. (1997) · BMJ · doi:10.1136/bmj.315.7109.629 · Cited by 56,722
Quantifying heterogeneity in a meta‐analysis
Higgins, J. P. T., & Thompson, S. G. (2002) · Statistics in Medicine · doi:10.1002/sim.1186 · Cited by 36,763
Borenstein, M., Hedges, L. V., Higgins, J. P. T., & Rothstein, H. R. (2009) · doi:10.1002/9780470743386 · Cited by 14,225
Duval, S., & Tweedie, R. (2000) · Biometrics · doi:10.1111/j.0006-341x.2000.00455.x · Cited by 14,099
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · PLoS Medicine · doi:10.1371/journal.pmed.0020124 · Cited by 10,505
Deci, E. L., Koestner, R., & Ryan, R. M. (1999) · Psychological Bulletin · doi:10.1037/0033-2909.125.6.627 · Cited by 7,176
The job satisfaction–job performance relationship: A qualitative and quantitative review.
Judge, T. A., Thoresen, C. J., Bono, J. E., & Patton, G. K. (2001) · Psychological Bulletin · doi:10.1037/0033-2909.127.3.376 · Cited by 4,760
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · CHANCE · doi:10.1080/09332480.2005.10722754 · Cited by 4,702 · cited 2×
“208 published literature detecting a statistically significant effect is often the rule rather than exception (Ioannidis, 2005; Ioannidis & Trikalinos, 2007).”
BackgroundThe Handbook of Research Synthesis and Meta-Analysis
(2019) · doi:10.7758/9781610448864 · Cited by 4,553
Perceived discrimination and health: A meta-analytic review.
Pascoe, E. A., & Smart Richman, L. (2009) · Psychological Bulletin · doi:10.1037/a0016059 · Cited by 4,427
Webb, T. L., & Sheeran, P. (2006) · Psychological Bulletin · doi:10.1037/0033-2909.132.2.249 · Cited by 3,732
The New Statistics: Why and How
Cumming, G. (2014) · Psychological Science · doi:10.1177/0956797613504966 · Cited by 3,200
Publication bias in clinical research
Easterbrook, P. J., Gopalan, R., Berlin, J. A., & Matthews, D. R. (1991) · The Lancet · doi:10.1016/0140-6736(91)90201-y · Cited by 3,109
Eagly, A. H., Johannesen-Schmidt, M. C., & van Engen, M. L. (2003) · Psychological Bulletin · doi:10.1037/0033-2909.129.4.569 · Cited by 2,547
Publication Bias in Meta‐Analysis: Prevention, Assessment and Adjustments
(2005) · doi:10.1002/0470870168 · Cited by 2,368
Comparison of two methods to detect publication bias in meta-analysis.
Peters, J. L. (2006) · JAMA · doi:10.1001/jama.295.6.676 · Cited by 2,278
A modified test for small‐study effects in meta‐analyses of controlled trials with binary endpoints
Harbord, R. M., Egger, M., & Sterne, J. A. C. (2006) · Statistics in Medicine · doi:10.1002/sim.2380 · Cited by 2,267
Does low self-esteem predict depression and anxiety? A meta-analysis of longitudinal studies.
Sowislo, J. F., & Orth, U. (2013) · Psychological Bulletin · doi:10.1037/a0028931 · Cited by 2,123
Robust variance estimation in meta‐regression with dependent effect size estimates
Hedges, L. V., Tipton, E., & Johnson, M. C. (2010) · Research Synthesis Methods · doi:10.1002/jrsm.5 · Cited by 2,086
A Fail-Safe N for Effect Size in Meta-Analysis
Orwin, R. G. (1983) · Journal of Educational Statistics · doi:10.2307/1164923 · Cited by 2,064 · cited 1×
“Rosenthal (1991) and Orwin (1983) proposed various measures of “file drawer” or “failsafe N” statistics, which estimate the number of presumably missing studies that could be added to a meta-analysis before it yielded a null mean effect size.”
MethodGrabe, S., Ward, L. M., & Hyde, J. S. (2008) · Psychological Bulletin · doi:10.1037/0033-2909.134.3.460 · Cited by 2,053
On the universality and cultural specificity of emotion recognition: a meta-analysis
Elfenbein, H. A., & Ambady, N. (2002) · Psychological Bulletin · doi:10.1037/0033-2909.128.2.203 · Cited by 2,049
Lipsey, M. W., & Wilson, D. B. (1993) · American Psychologist · doi:10.1037/0003-066x.48.12.1181 · Cited by 2,033 · cited 1×
“Lipsey and Wilson (1993) also examined publication bias in education by reviewing 302 meta-analyses of education-based studies.”
BackgroundThe malleability of spatial skills: A meta-analysis of training studies.
Uttal, D. H., Meadow, N. G., Tipton, E., Hand, L. L., Alden, A. R., Warren, C., & Newcombe, N. S. (2013) · Psychological Bulletin · doi:10.1037/a0028446 · Cited by 1,899
Eagly, A. H., Ashmore, R. D., Makhijani, M. G., & Longo, L. C. (1991) · Psychological Bulletin · doi:10.1037/0033-2909.110.1.109 · Cited by 1,875
Chan, A.-W., Hróbjartsson, A., Haahr, M. T., Gøtzsche, P. C., & Altman, D. G. (2004) · JAMA · doi:10.1001/jama.291.20.2457 · Cited by 1,845 · cited 1×
“For instance, we advocate for greater use of cohort studies in the social and behavioral sciences that track the research process of federally funded primary studies (e.g., Chan et al., 2004).”
MethodLearner-Centered Teacher-Student Relationships Are Effective: A Meta-Analysis
Cornelius-White, J. (2007) · Review of Educational Research · doi:10.3102/003465430298563 · Cited by 1,812
Are leader stereotypes masculine? A meta-analysis of three research paradigms
Koenig, A. M., Eagly, A. H., Mitchell, A. A., & Ristikari, T. (2011) · Psychological Bulletin · doi:10.1037/a0023557 · Cited by 1,748
Is caregiving hazardous to one’s physical health? A meta-analysis
Vitaliano, P. P., Zhang, J., & Scanlan, J. M. (2003) · Psychological Bulletin · doi:10.1037/0033-2909.129.6.946 · Cited by 1,734
Interrelatedness of marital relations and parent-child relations: A meta-analytic review.
Erel, O., & Burman, B. (1995) · Psychological Bulletin · doi:10.1037/0033-2909.118.1.108 · Cited by 1,603
Systematic Review of the Empirical Evidence of Study Publication Bias and Outcome Reporting Bias
Dwan, K., Altman, D. G., Arnaiz, J. A., Bloom, J., Chan, A.-W., Cronin, E., Decullier, E., Easterbrook, P. J., Von Elm, E., Gamble, C., et al. (2008) · PLoS ONE · doi:10.1371/journal.pone.0003081 · Cited by 1,546
Experimental disclosure and its moderators: A meta-analysis.
Frattaroli, J. (2006) · Psychological Bulletin · doi:10.1037/0033-2909.132.6.823 · Cited by 1,502
P-curve: A key to the file-drawer.
Simonsohn, U., Nelson, L. D., & Simmons, J. P. (2014) · Journal of Experimental Psychology: General · doi:10.1037/a0033242 · Cited by 1,500
Forming attitudes that predict future behavior: A meta-analysis of the attitude-behavior relation.
Glasman, L. R., & Albarracín, D. (2006) · Psychological Bulletin · doi:10.1037/0033-2909.132.5.778 · Cited by 1,459
The instructional effect of feedback in test-like events.
Bangert-Drowns, R. L., Kulik, C.-L. C., Kulik, J. A., & Morgan, M. (1991) · Review of Educational Research · doi:10.3102/00346543061002213 · Cited by 1,313
Viechtbauer, W. (2007) · Psychometrika · doi:10.1007/s11336-006-1450-y · Cited by 1,250
Does Homework Improve Academic Achievement? A Synthesis of Research, 1987–2003
Cooper, H., Robinson, J. C., & Patall, E. A. (2006) · Review of Educational Research · doi:10.3102/00346543076001001 · Cited by 1,230
Parents' differential socialization of boys and girls: a meta-analysis
Lytton, H., & Romney, D. M. (1991) · Psychological Bulletin · doi:10.1037/0033-2909.109.2.267 · Cited by 1,210
Smith, T. B., McCullough, M. E., & Poll, J. (2003) · Psychological Bulletin · doi:10.1037/0033-2909.129.4.614 · Cited by 1,150
Effects of learning skills interventions on student learning: a meta-analysis.
Hattie, J., Biggs, J., & Purdie, N. (1996) · Review of Educational Research · doi:10.3102/00346543066002099 · Cited by 1,106
Patall, E. A., Cooper, H., & Robinson, J. C. (2008) · Psychological Bulletin · doi:10.1037/0033-2909.134.2.270 · Cited by 1,101
Self-focused attention and negative affect: A meta-analysis.
Mor, N., & Winquist, J. (2002) · Psychological Bulletin · doi:10.1037/0033-2909.128.4.638 · Cited by 1,069
McClure, E. B. (2000) · Psychological Bulletin · doi:10.1037/0033-2909.126.3.424 · Cited by 1,032
Consequences of prejudice against the null hypothesis.
Greenwald, A. G. (1975) · Psychological Bulletin · doi:10.1037/h0076157 · Cited by 1,022 · cited 1×
“Greenwald (1975), in a landmark study on the “prejudice against the null hypothesis” (p. 1), posited that null research findings are often interpreted as a failing of the researcher (or design), instead of the probable hypothesis that successfully detecting a significant effect is much less likely.”
BackgroundLee, C. J., Sugimoto, C. R., Zhang, G., & Cronin, B. (2013) · Journal of the American Society for Information Science and Technology · doi:10.1002/asi.22784 · Cited by 1,000
A meta-analytic examination of assumed properties of child sexual abuse using college samples.
Rind, B., Tromovitch, P., & Bauserman, R. L. (1998) · Psychological Bulletin · doi:10.1037//0033-2909.124.1.22 · Cited by 988
Effects of psychotherapy for depression in children and adolescents: A meta-analysis
Weisz, J. R., McCarty, C. A., & Valeri, S. M. (2006) · Psychological Bulletin · doi:10.1037/0033-2909.132.1.132 · Cited by 942
Learning With Concept and Knowledge Maps: A Meta-Analysis
Nesbit, J. C., & Adesope, O. O. (2006) · Review of Educational Research · doi:10.3102/00346543076003413 · Cited by 932
Sex differences in self-disclosure: A meta-analysis.
Dindia, K., & Allen, M. (1992) · Psychological Bulletin · doi:10.1037//0033-2909.112.1.106 · Cited by 919
The psychological effects of meditation: a meta-analysis
Sedlmeier, P., Eberth, J., Schwarz, M., Zimmermann, D., Haarig, F., Jaeger, S., & Kunze, S. (2012) · Psychological Bulletin · doi:10.1037/a0028168 · Cited by 910
Publication bias in clinical trials due to statistical significance or direction of trial results
Hopewell, S., Loudon, K., Clarke, M. J., Oxman, A. D., & Dickersin, K. (2009) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.mr000006.pub3 · Cited by 898
Gender and aggressive behavior: a meta-analytic review of the social psychological literature.
Eagly, A. H., & Steffen, V. J. (1986) · Psychological Bulletin · doi:10.1037/0033-2909.100.3.309 · Cited by 872
The Effects of School-Based Writing-to-Learn Interventions on Academic Achievement: A Meta-Analysis
Bangert-Drowns, R. L., Hurley, M. M., & Wilkinson, B. (2004) · Review of Educational Research · doi:10.3102/00346543074001029 · Cited by 802
An exploratory test for an excess of significant findings
Ioannidis, J. P., & Trikalinos, T. A. (2007) · Clinical Trials · doi:10.1177/1740774507079441 · Cited by 751 · cited 1×
“published literature detecting a statistically significant effect is often the rule rather than exception (Ioannidis, 2005; Ioannidis & Trikalinos, 2007).”
BackgroundGender differences in moral orientation: A meta-analysis.
Jaffee, S., & Hyde, J. S. (2000) · Psychological Bulletin · doi:10.1037/0033-2909.126.5.703 · Cited by 739
Within-Class Grouping: A Meta-Analysis
Lou, Y., Abrami, P. C., Spence, J. C., Poulsen, C., Chambers, B., & d’Apollonia, S. (1996) · Review of Educational Research · doi:10.3102/00346543066004423 · Cited by 727
Regulatory focus and work-related outcomes: A review and meta-analysis.
Lanaj, K., Chang, C.-H. “., & Johnson, R. E. (2012) · Psychological Bulletin · doi:10.1037/a0027723 · Cited by 725
Seto, M. C., & Lalumière, M. L. (2010) · Psychological Bulletin · doi:10.1037/a0019700 · Cited by 721
Small Group and Individual Learning with Technology: A Meta-Analysis
Lou, Y., Abrami, P. C., & d’Apollonia, S. (2001) · Review of Educational Research · doi:10.3102/00346543071003449 · Cited by 721
Astill, R. G., Van der Heijden, K. B., Van IJzendoorn, M. H., & Van Someren, E. J. W. (2012) · Psychological Bulletin · doi:10.1037/a0028204 · Cited by 707
Timing of feedback and verbal learning.
Kulik, J. A., & Kulik, C.-L. C. (1988) · Review of Educational Research · doi:10.3102/00346543058001079 · Cited by 703
Stice, E., Shaw, H., & Marti, C. N. (2006) · Psychological Bulletin · doi:10.1037/0033-2909.132.5.667 · Cited by 690
Ethnicity and body dissatisfaction among women in the United States: A meta-analysis.
Grabe, S., & Hyde, J. S. (2006) · Psychological Bulletin · doi:10.1037/0033-2909.132.4.622 · Cited by 655
The Effects of Vocabulary Intervention on Young Children’s Word Learning: A Meta-Analysis
Marulis, L. M., & Neuman, S. B. (2010) · Review of Educational Research · doi:10.3102/0034654310377087 · Cited by 655
Murray, J., Farrington, D. P., & Sekol, I. (2012) · Psychological Bulletin · doi:10.1037/a0026407 · Cited by 653
Ioannidis, J. P. A. (1998) · JAMA · doi:10.1001/jama.279.4.281 · Cited by 632 · cited 1×
“In addition, favorable outcomes receive quicker time to publication (Ioannidis, 1998) and a higher rate of citation (Tanner-Smith & Polanin, 2014).”
BackgroundThoresen, C. J., Kaplan, S. A., Barsky, A. P., Warren, C. R., & de Chermont, K. (2003) · Psychological Bulletin · doi:10.1037/0033-2909.129.6.914 · Cited by 617
Ferguson, C. J., & Heene, M. (2012) · Perspectives on Psychological Science · doi:10.1177/1745691612459059 · Cited by 603 · cited 1×
“Indeed, prior research has documented that (a) studies with nonsignificant findings are often left unpublished (Ferguson & Heene, 2012), (b) authors have a tendency to report results for outcomes with statistically significant effects (Pigott, Valentine, Polanin, Williams, & Canada, 2013), and (c)…”
BackgroundParent Involvement in Homework: A Research Synthesis
Patall, E. A., Cooper, H., & Robinson, J. C. (2008) · Review of Educational Research · doi:10.3102/0034654308325185 · Cited by 602
Tanner-Smith, E. E., & Tipton, E. (2014) · Research Synthesis Methods · doi:10.1002/jrsm.1091 · Cited by 588
Self-citationOut-of-School-Time Programs: A Meta-Analysis of Effects for At-Risk Students
Lauer, P. A., Akiba, M., Wilkerson, S. B., Apthorp, H. S., Snow, D., & Martin-Glenn, M. L. (2006) · Review of Educational Research · doi:10.3102/00346543076002275 · Cited by 579
When do gestures communicate? A meta-analysis.
Hostetter, A. B. (2011) · Psychological Bulletin · doi:10.1037/a0022128 · Cited by 558
The contingent smile: A meta-analysis of sex differences in smiling.
LaFrance, M., Hecht, M. A., & Paluck, E. L. (2003) · Psychological Bulletin · doi:10.1037/0033-2909.129.2.305 · Cited by 514
Ferguson, C. J., & Brannick, M. T. (2012) · Psychological Methods · doi:10.1037/a0024445 · Cited by 493
van IJzendoorn, M. H., Juffer, F., & Poelhuis, C. W. K. (2005) · Psychological Bulletin · doi:10.1037/0033-2909.131.2.301 · Cited by 488
Cooperative versus Competitive Efforts and Problem Solving
Qin, Z., Johnson, D. W., & Johnson, R. T. (1995) · Review of Educational Research · doi:10.2307/1170710 · Cited by 488
Publication and other reporting biases in cognitive sciences: detection, prevalence, and prevention
Ioannidis, J. P. A., Munafò, M. R., Fusar-Poli, P., Nosek, B. A., & David, S. P. (2014) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2014.02.010 · Cited by 471
Minority influence: A meta-analytic review of social influence processes.
Wood, W., Lundgren, S. R., Ouellette, J. A., Busceme, S., & Blackstone, T. (1994) · Psychological Bulletin · doi:10.1037//0033-2909.115.3.323 · Cited by 455
Arcsine test for publication bias in meta‐analyses with binary outcomes
Rücker, G., Schwarzer, G., & Carpenter, J. (2008) · Statistics in Medicine · doi:10.1002/sim.2971 · Cited by 444
Currier, J. M., Neimeyer, R. A., & Berman, J. S. (2008) · Psychological Bulletin · doi:10.1037/0033-2909.134.5.648 · Cited by 431
Psychosis as a risk factor for violence to others: A meta-analysis.
Douglas, K. S., Guy, L. S., & Hart, S. D. (2009) · Psychological Bulletin · doi:10.1037/a0016311 · Cited by 415
Swanson, H. L., & Hoskyn, M. (1998) · Review of Educational Research · doi:10.3102/00346543068003277 · Cited by 409
Mechanisms of masked priming: A meta-analysis.
Van den Bussche, E., Van den Noortgate, W., & Reynvoet, B. (2009) · Psychological Bulletin · doi:10.1037/a0015329 · Cited by 350
The own-age bias in face recognition: a meta-analytic and theoretical review
Rhodes, M. G., & Anastasi, J. S. (2012) · Psychological Bulletin · doi:10.1037/a0025750 · Cited by 346
Sterne, J. A. C., Becker, B. J., & Egger, M. (2005) · Publication Bias in Meta‐Analysis · doi:10.1002/0470870168.ch5 · Cited by 341
The (non)relation between empathy and aggression: Surprising results from a meta-analysis.
Vachon, D. D., Lynam, D. R., & Johnson, J. A. (2014) · Psychological Bulletin · doi:10.1037/a0035236 · Cited by 334
Rosenthal, R. (1991) · Psychosomatic Medicine · doi:10.1097/00006842-199105000-00001 · Cited by 315
Bowman, N. A. (2011) · Review of Educational Research · doi:10.3102/0034654310383047 · Cited by 307
Effects of media violence on viewers' aggression in unconstrained social interaction.
Wood, W., Wong, F. Y., & Chachere, J. G. (1991) · Psychological Bulletin · doi:10.1037//0033-2909.109.3.371 · Cited by 303
Feingold, A. (1988) · Psychological Bulletin · doi:10.1037//0033-2909.104.2.226 · Cited by 280
Bangert-Drowns, R. L. (1993) · Review of Educational Research · doi:10.3102/00346543063001069 · Cited by 273
The influence of delaying judgments of learning on metacognitive accuracy: a meta-analytic review
Rhodes, M. G., & Tauber, S. K. (2011) · Psychological Bulletin · doi:10.1037/a0021705 · Cited by 230
Publication bias and meta-analysis
Smith, M. L. (1980) · Evaluation in Education · doi:10.1016/0191-765x(80)90004-x · Cited by 214
Kim, J. S., & Quinn, D. M. (2013) · Review of Educational Research · doi:10.3102/0034654313483906 · Cited by 213
Juffer, F., & van IJzendoorn, M. H. (2007) · Psychological Bulletin · doi:10.1037/0033-2909.133.6.1067 · Cited by 199
Effectiveness of Consultation on Student Ratings Feedback: A Meta-Analysis
Penny, A. R., & Coe, R. (2004) · Review of Educational Research · doi:10.3102/00346543074002215 · Cited by 199
Moving from cognition to behavior: What the research says.
Johnson, R. E., Chang, C.-H., & Lord, R. G. (2006) · Psychological Bulletin · doi:10.1037/0033-2909.132.3.381 · Cited by 193
Quantitative reanalysis of aggregate data on IQ in sexual offenders.
Cantor, J. M., Blanchard, R., Robichaud, L. K., & Christensen, B. K. (2005) · Psychological Bulletin · doi:10.1037/0033-2909.131.4.555 · Cited by 193
Bösch, H., Steinkamp, F., & Boller, E. (2006) · Psychological Bulletin · doi:10.1037/0033-2909.132.4.497 · Cited by 182
Ritter, G. W., Barnett, J. H., Denny, G. S., & Albin, G. R. (2009) · Review of Educational Research · doi:10.3102/0034654308325690 · Cited by 180
Failsafe N or File‐Drawer Number
Becker, B. J. (2005) · Publication Bias in Meta‐Analysis · doi:10.1002/0470870168.ch7 · Cited by 179
A Review of Meta-Analyses in Education: Methodological Strengths and Weaknesses
Ahn, S., Ames, A. J., & Myers, N. D. (2012) · Review of Educational Research · doi:10.3102/0034654312458162 · Cited by 179
PUBLICATION BIAS: THE ACHILLES’ HEEL OF SYSTEMATIC REVIEWS?
Torgerson, C. J. (2006) · British Journal of Educational Studies · doi:10.1111/j.1467-8527.2006.00332.x · Cited by 175
Forewarned and forearmed? Two meta-analysis syntheses of forewarnings of influence appeals.
Wood, W., & Quinn, J. M. (2003) · Psychological Bulletin · doi:10.1037/0033-2909.129.1.119 · Cited by 168
Self-statement modification in the treatment of child behavior disorders: A meta-analysis.
Dush, D. M., Hirt, M. L., & Schrøeder, H. E. (1989) · Psychological Bulletin · doi:10.1037/0033-2909.106.1.97 · Cited by 166
Dyadic interracial interactions: A meta-analysis.
Toosi, N. R., Babbitt, L. G., Ambady, N., & Sommers, S. R. (2011) · Psychological Bulletin · doi:10.1037/a0025767 · Cited by 164
Fate of biomedical research protocols and publication bias in France: retrospective cohort study
Decullier, É., Lhéritier, V., & Chapuis, F. R. (2005) · BMJ · doi:10.1136/bmj.38488.385995.8f · Cited by 162
Effects of Full-Day Kindergarten on Academic Achievement and Social Development
Cooper, H. M., Allen, A. B., Patall, E. A., & Dent, A. L. (2010) · Review of Educational Research · doi:10.3102/0034654309359185 · Cited by 144
Grey Literature and Systematic Reviews
Hopewell, S., Clarke, M. J., & Mallett, S. (2005) · Wiley eBooks · doi:10.1002/0470870168.ch4 · Cited by 141
Cooper, H., Valentine, J. C., Charlton, K., & Melson, A. (2003) · Review of Educational Research · doi:10.3102/00346543073001001 · Cited by 138
Levine, T. R., Asada, K. J., & Carpenter, C. (2009) · Communication Monographs · doi:10.1080/03637750903074685 · Cited by 131
Effects of Multigrade and Multi-Age Classes Reconsidered
Veenman, S. (1996) · Review of Educational Research · doi:10.3102/00346543066003323 · Cited by 130
Effectiveness of Computer-Assisted Instruction in Statistics: A Meta-Analysis
Sosa, G. W., Berger, D. E., Saw, A. T., & Mary, J. (2010) · Review of Educational Research · doi:10.3102/0034654310378174 · Cited by 129
Helping behavior in rural and urban environments: A meta-analysis.
Steblay, N. M. (1987) · Psychological Bulletin · doi:10.1037/0033-2909.102.3.346 · Cited by 127
Outcome-Reporting Bias in Education Research
Pigott, T. D., Valentine, J. C., Polanin, J. R., Williams, R. T., & Canada, D. D. (2013) · Educational Researcher · doi:10.3102/0013189x13507104 · Cited by 127
Self-citationHedges, L. V., & Vevea, J. (2005) · Publication Bias in Meta‐Analysis · doi:10.1002/0470870168.ch9 · Cited by 123
Multivariate Meta-Analysis as Structural Equation Models
Cheung, M. W. ‐. (2013) · Structural Equation Modeling A Multidisciplinary Journal · doi:10.1080/10705511.2013.797827 · Cited by 120
Controlling Variables: A Meta-Analysis of Training Studies
Ross, J. A. (1988) · Review of Educational Research · doi:10.3102/00346543058004405 · Cited by 100
Metzger, S. A., & Wu, M. (2008) · Review of Educational Research · doi:10.3102/0034654308323035 · Cited by 84
The use of meta-analytic statistical significance testing
Polanin, J. R., & Pigott, T. D. (2015) · Research Synthesis Methods · doi:10.1002/jrsm.1124 · Cited by 78 · cited 2×
“As noted previously, one of the most common methods for assessing the possibility of publication bias in a meta-analysis is to test for the differences in mean effect sizes across the published versus unpublished studies included in the review (Polanin & Pigott, 2015).”
BackgroundMethodSelf-citationSchild, A. H. E., & Voracek, M. (2014) · Research Synthesis Methods · doi:10.1002/jrsm.1125 · Cited by 43
Bax, L., & Moons, K. G. M. (2010) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2010.09.003 · Cited by 43
A retrospective analysis of dissemination biases in the brief alcohol intervention literature.
Tanner-Smith, E. E., & Polanin, J. R. (2015) · Psychology of Addictive Behaviors · doi:10.1037/adb0000014 · Cited by 26 · cited 2×
“The current study is one in a growing body of literature that suggests publication bias is prevalent in the social and behavioral sciences also (Pigott et al., 2013; Tanner-Smith & Polanin, 2014).”
BackgroundSelf-citationVariables Influencing the Acquisition of a Generic Teaching Skill
Gliessman, D. H., Pugh, R. C., Dowden, D. E., & Hutchins, T. F. (1988) · Review of Educational Research · doi:10.3102/00346543058001025 · Cited by 20
Research works printed but not linked: 12
- 10.1037/00332909.125.1.64 · doi:10.1037/00332909.125.1.64
- 10.1037//0033-2909.128.2.203 · DOI · Paper page
- 10.1037/0033-2909.104.2.226 · DOI · Paper page
- Hopewell S., 2005, Publication bias in meta-analysis: Prevention, assessment and adjustments, 48
- 10.1037/0003-066X.48.12.1181 · DOI · Paper page
- Lipsey M. W., 2001, Applied social research methods: Vol. 49. Practical meta-analysis
- 10.3102/00346543065002129 · doi:10.3102/00346543065002129 · Paper page
- 10.1037/0033-2909.124.1.22 · DOI · Paper page
- Sterne J. A. C., 2005, Publication bias in meta-analysis: Prevention, assessment and adjustments, 75
- Sutton A. J., 2009, The handbook of research synthesis and meta-analysis, 435
- *Wood W., 1994, Psychological Bulletin, 115, 323, 10.1037/0033-2909.115.3.323 · doi:10.1037/0033-2909.115.3.323 · Paper page
- 10.1037/0033-2909.109.3.371 · DOI · Paper page
Cited by
Environmental education outcomes for conservation: A systematic review
Ardoin, N. M., Bowers, A. W., & Gaillard, E. (2020) · Biological Conservation · doi:10.1016/j.biocon.2019.108224 · Cited by 924
Long-term outcomes of childhood sexual abuse: an umbrella review
Hailes, H. P., Yu, R., Danese, A., & Fazel, S. (2019) · The Lancet Psychiatry · doi:10.1016/s2215-0366(19)30286-x · Cited by 827
Cipriano, C., Strambler, M. J., Naples, L. H., Ha, C., Kirk, M., Wood, M., Sehgal, K., Zieher, A. K., Eveleigh, A., McCarthy, M., et al. (2023) · Child Development · doi:10.1111/cdev.13968 · Cited by 636
“Further, given evidence that average effect sizes differ based on the nature of a publication (Chow & Ekholm, 2018; Polanin et al., 2016), we compared overall effect sizes for published to unpub-lished studies.”
Methodological guidance paper: High-quality meta- analysis in a systematic review
Pigott, T. D., & Polanin, J. R. (2020) · Review of Educational Research · doi:10.3102/0034654319877153 · Cited by 568
“Many researchers have documented the existence of publication bias in medicine and psychology (Dickersin, 1990; Ferguson & Brannick, 2012) as well as in education (Polanin et al., 2016).”
BackgroundSelf-citationEffects of flipping the classroom on learning outcomes and satisfaction: A meta-analysis
van Alten, D. C. D., Phielix, C., Janssen, J., & Kester, L. (2019) · Educational Research Review · doi:10.1016/j.edurev.2019.05.003 · Cited by 546
“In addition, Spanjers et al. (2015) found in their meta-analysis on blended learning that the inclusion of quizzes positively moderated the effects of blended learning on learning outcomes and satisfaction.”
Background
Metadata
- Authors
- Polanin, J. R., Tanner-Smith, E. E., & Hennessy, E. A.
- Affiliations
- Vanderbilt University (US)
- Vanderbilt University
- Venue
- Review of Educational Research
- Published
- 2016-03
- Volume
- 86
- Issue
- 1
- Pages
- 207-236
- ISSN
- 0034-6543 (ISSN-L)
- 1935-1046
- OpenAlex
- W2126293099
- Cited by
- 234
- References
- 123
- Citations by year
- 2026: 9 · 2025: 18 · 2024: 20 · 2023: 26 · 2022: 24 · 2021: 45 · 2020: 25 · 2019: 27 · 2018: 17 · 2017: 11 · 2016: 7
- FWCI
- 11.23
- Type
- journal article
- Language
- en
- Open access
- closed
- Licence
- journals.sagepub.com/page/policies/text-and-data-mining-license (tdm)
- Locations
- Review of Educational Research (publishedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Psychology Research and Bibliometrics
- Social Sciences › Library and Information Sciences › scientometrics and bibliometrics research
- Keywords
- publication bias, unpublished research, meta-analysis, negative results, research validity, publication status, evidence synthesis, intervention research
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.