The DOI matches its Crossref record; 4 of the 4 references it cites were found in a registry or an index.
Estimating Effect Size under Publication Bias: Small Sample Properties and Robustness of a Random Effects Selection Model
Abstract
When there is publication bias, studies yielding large p values, and hence small effect estimates, are less likely to be published, which leads to biased estimates of effects in meta-analysis. We investigate a selection model based on one-tailed p values in the context of a random effects model. The procedure both models the selection process and corrects for the consequences of selection on estimates of the mean and variance of effect parameters. A test of the statistical significance of selection is also provided. The small sample properties of the method are evaluated by means of simulations, and the asymptotic theory is found to be reasonably accurate under correct model specification and plausible conditions. The method substantially reduces bias due to selection when model specification is correct, but the variance of estimates is increased; thus mean squared error is reduced only when selection produces substantial bias. The robustness of the method to violations of assumptions about the form of the distribution of the random effects is also investigated via simulation, and the model-corrected estimates of the mean effect are generally found to be much less biased than the uncorrected estimates. The significance test for selection bias, however, is found to be highly nonrobust, rejecting at up to 10 times the nominal rate when there is no selection but the distribution of the effects is incorrectly specified.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Publication bias in clinical research
Easterbrook, P. J., Gopalan, R., Berlin, J. A., & Matthews, D. R. (1991) · The Lancet · doi:10.1016/0140-6736(91)90201-y · Cited by 3,109 · cited 1×
“The models studied by Lane and Dunlap (1978) and by Hedges (1984) may be viewed as extreme cases of weighted distributions where only significant (i.”
MethodSterling, T. D. (1959) · Journal of the American Statistical Association · doi:10.1080/01621459.1959.10501497 · Cited by 964
Modeling Publication Selection Effects in Meta-Analysis
Hedges, L. V. (1992) · Statistical Science · doi:10.1214/ss/1177011364 · Cited by 359
Self-citationHedges, L. V. (1984) · Journal of Educational Statistics · doi:10.3102/10769986009001061 · Cited by 241
Self-citation
Cited by
The Handbook of Research Synthesis and Meta-Analysis
(2019) · doi:10.7758/9781610448864 · Cited by 4,553
Kowalski, R. M., Giumetti, G. W., Schroeder, A. N., & Lattanner, M. R. (2014) · Psychological Bulletin · doi:10.1037/a0035618 · Cited by 3,266
Comparison of two methods to detect publication bias in meta-analysis.
Peters, J. L. (2006) · JAMA · doi:10.1001/jama.295.6.676 · Cited by 2,278
Peters, J. L., Sutton, A. J., Jones, D. R., Abrams, K. R., & Rushton, L. (2008) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2007.11.010 · Cited by 2,053
GRADE guidelines: 5. Rating the quality of evidence—publication bias
Guyatt, G. H., Oxman, A. D., Montori, V., Vist, G., Kunz, R., Brozek, J., Alonso-Coello, P., Djulbegovic, B., Atkins, D., Falck-Ytter, Y., et al. (2011) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2011.01.011 · Cited by 1,819
Andrews, G., Cuijpers, P., Craske, M. G., McEvoy, P., & Titov, N. (2010) · PLoS ONE · doi:10.1371/journal.pone.0013196 · Cited by 1,311
Rosenberg, M. S. (2005) · Evolution · doi:10.1111/j.0014-3820.2005.tb01004.x · Cited by 1,285
Gender differences in temperament: A meta-analysis.
Else-Quest, N. M., Hyde, J. S., Goldsmith, H. H., & Van Hulle, C. A. (2006) · Psychological Bulletin · doi:10.1037/0033-2909.132.1.33 · Cited by 1,169
Andrews, G., Basu, A., Cuijpers, P., Craske, M. G., McEvoy, P., English, C. L., & Newby, J. M. (2018) · Journal of Anxiety Disorders · doi:10.1016/j.janxdis.2018.01.001 · Cited by 922
Initial Scale Development: Sample Size for Pilot Studies
Johanson, G. A., & Brooks, G. P. (2010) · Educational and Psychological Measurement · doi:10.1177/0013164409355692 · Cited by 1,012
Linde, K., Clausius, N., Ramirez, G., Melchart, D., Eitel, F., Hedges, L. V., & Jonas, W. B. (1997) · The Lancet · doi:10.1016/s0140-6736(97)02293-9 · Cited by 979
Self-citationAn exploratory test for an excess of significant findings
Ioannidis, J. P., & Trikalinos, T. A. (2007) · Clinical Trials · doi:10.1177/1740774507079441 · Cited by 751
Stanley, T. D. (2005) · Journal of Economic Surveys · doi:10.1111/j.0950-0804.2005.00250.x · Cited by 742
Peters, J. L., Sutton, A. J., Jones, D. R., Abrams, K. R., & Rushton, L. (2007) · Statistics in Medicine · doi:10.1002/sim.2889 · Cited by 716
The power of statistical tests for moderators in meta-analysis
Hedges, L. V., & Pigott, T. D. (2004) · Psychological Methods · doi:10.1037/1082-989x.9.4.426 · Cited by 672
Self-citationOutcomes of Meaningful Work: A Meta‐Analysis
Allan, B. A., Batz-Barbarich, C., Sterling, H. M., & Tay, L. (2019) · Journal of Management Studies · doi:10.1111/joms.12406 · Cited by 659
Maxwell, S. E. (2004) · Psychological Methods · doi:10.1037/1082-989x.9.2.147 · Cited by 654
Stanley, T. D. (2008) · Oxford Bulletin of Economics and Statistics · doi:10.1111/j.1468-0084.2007.00487.x · Cited by 605
Correcting for Bias in Psychology: A Comparison of Meta-Analytic Methods
Carter, E. C., Schönbrodt, F. D., Gervais, W. M., & Hilgard, J. (2019) · Advances in Methods and Practices in Psychological Science · doi:10.1177/2515245919847196 · Cited by 588
Siegel, E. H., Sands, M. K., Van den Noortgate, W., Condon, P., Chang, Y., Dy, J., Quigley, K. S., & Barrett, L. F. (2018) · Psychological Bulletin · doi:10.1037/bul0000128 · Cited by 515
Potential of forest thinning to mitigate drought stress: A meta-analysis
Sohn, J. A., Saha, S., & Bauhus, J. (2016) · Forest Ecology and Management · doi:10.1016/j.foreco.2016.07.046 · Cited by 503
Davies, E. B., Morriss, R., & Glazebrook, C. (2014) · Journal of Medical Internet Research · doi:10.2196/jmir.3142 · Cited by 499
Publication and other reporting biases in cognitive sciences: detection, prevalence, and prevention
Ioannidis, J. P. A., Munafò, M. R., Fusar-Poli, P., Nosek, B. A., & David, S. P. (2014) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2014.02.010 · Cited by 471
Gender differences in self-conscious emotional experience: A meta-analysis.
Else-Quest, N. M., Higgins, A., Allison, C., & Morton, L. C. (2012) · Psychological Bulletin · doi:10.1037/a0027930 · Cited by 464
Transparency, Reproducibility, and the Credibility of Economics Research
Christensen, G., & Miguel, E. (2018) · Journal of Economic Literature · doi:10.1257/jel.20171350 · Cited by 446
Efficacy of psychotherapeutic interventions to promote forgiveness: A meta-analysis.
Wade, N. G., Hoyt, W. T., Kidwell, J. E. M., & Worthington, E. L. (2014) · Journal of Consulting and Clinical Psychology · doi:10.1037/a0035268 · Cited by 444
Publication and related biases.
Song, Eastwood, Gilbody, Duley, & Sutton (2000) · Health Technology Assessment · doi:10.3310/hta4100 · Cited by 427
A General Linear Model for Estimating Effect Size in the Presence of Publication Bias
Vevea, J. L., & Hedges, L. V. (1995) · Psychometrika · doi:10.1007/bf02294384 · Cited by 413
Self-citationVideo Improves Learning in Higher Education: A Systematic Review
Noetel, M., Griffith, S., Delaney, O., Sanders, T., Parker, P., del Pozo Cruz, B., & Lonsdale, C. (2021) · Review of Educational Research · doi:10.3102/0034654321990713 · Cited by 378
Moreno, S. G., Sutton, A. J., Ades, A., Stanley, T. D., Abrams, K. R., Peters, J. L., & Cooper, N. J. (2009) · BMC Medical Research Methodology · doi:10.1186/1471-2288-9-2 · Cited by 373
A meta-analysis of six-month studies of antiplaque and antigingivitis agents
Gunsolley, J. C. (2006) · The Journal of the American Dental Association · doi:10.14219/jada.archive.2006.0110 · Cited by 357
Testing for funnel plot asymmetry of standardized mean differences
Pustejovsky, J. E., & Rodgers, M. A. (2019) · Research Synthesis Methods · doi:10.1002/jrsm.1332 · Cited by 354
Ioannidis, J. P. A., & Trikalinos, T. A. (2005) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2004.10.019 · Cited by 325
Adherence to Highly Active Antiretroviral Therapy (HAART): A Meta-Analysis
Ortego, C., Huedo-Medina, T. B., Llorca, J., Sevilla, L., Santos, P., Rodríguez, E., Warren, M. R., & Vejo, J. (2011) · AIDS and Behavior · doi:10.1007/s10461-011-9942-x · Cited by 310
Gender Differences in Responses to Moral Dilemmas: A Process Dissociation Analysis
Friesdorf, R., Conway, P., & Gawronski, B. (2015) · Personality and Social Psychology Bulletin · doi:10.1177/0146167215575731 · Cited by 300
Pellegrino, E., Öpik, M., Bonari, E., & Ercoli, L. (2015) · Soil Biology and Biochemistry · doi:10.1016/j.soilbio.2015.02.020 · Cited by 299
Swedberg, R. (2020) · The Production of Knowledge · doi:10.1017/9781108762519.002 · Cited by 293
Modelling publication bias in meta-analysis: a review
Sutton, A. J., Song, F., Gilbody, S. M., & Abrams, K. R. (2000) · Statistical Methods in Medical Research · doi:10.1177/096228020000900503 · Cited by 288
Detecting Selection Bias in Meta-Analyses with Multiple Outcomes: A Simulation Study
Fernández-Castilla, B., Declercq, L., Jamshidi, L., Beretvas, S. N., Onghena, P., & Van den Noortgate, W. (2021) · The Journal of Experimental Education · doi:10.1080/00220973.2019.1582470 · Cited by 285
Shadish, W. R., Hedges, L. V., & Pustejovsky, J. E. (2014) · Journal of School Psychology · doi:10.1016/j.jsp.2013.11.005 · Cited by 269
Self-citationMeta-analysis using effect size distributions of only statistically significant studies.
van Assen, M. A. L. M., van Aert, R. C. M., & Wicherts, J. M. (2015) · Psychological Methods · doi:10.1037/met0000025 · Cited by 265
Batz-Barbarich, C., Tay, L., Kuykendall, L., & Cheung, H. K. (2018) · Psychological Science · doi:10.1177/0956797618774796 · Cited by 253
Publication bias as a threat to the validity of meta-analytic results
Rothstein, H. R. (2008) · Journal of Experimental Criminology · doi:10.1007/s11292-007-9046-9 · Cited by 216
Differential efficacy of group and individual/couple psychotherapy with infertile patients
de Liz, T. M., & Strauss, B. (2005) · Human Reproduction · doi:10.1093/humrep/deh743 · Cited by 168
A historical review of publication bias
Marks‐Anglin, A., & Chen, Y. (2020) · Research Synthesis Methods · doi:10.1002/jrsm.1452 · Cited by 165
ADHD and personality: a meta-analytic review
Gomez, R., & Corr, P. J. (2014) · Clinical Psychology Review · doi:10.1016/j.cpr.2014.05.002 · Cited by 151
Multimedia Design for Learning: An Overview of Reviews With Meta-Meta-Analysis
Noetel, M., Griffith, S., Delaney, O., Harris, N. R., Sanders, T., Parker, P., del Pozo Cruz, B., & Lonsdale, C. (2022) · Review of Educational Research · doi:10.3102/00346543211052329 · Cited by 148
“Few used selection methods (Hedges & Vevea, 1996), which are less prone to these biases (Pustejovsky & Rodgers, 2019).”
Finding the power to reduce publication bias
Stanley, T. D., Doucouliagos, H., & Ioannidis, J. P. A. (2017) · Statistics in Medicine · doi:10.1002/sim.7228 · Cited by 131
Shadish, W. R., Zelinsky, N. A. M., Vevea, J. L., & Kratochwill, T. R. (2016) · Journal of Applied Behavior Analysis · doi:10.1002/jaba.308 · Cited by 130
Pair-bonding, fatherhood, and the role of testosterone: a meta-analytic review
Grebe, N. M., Sarafin, R. E., Strenth, C. R., & Zilioli, S. (2019) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2019.01.010 · Cited by 116
Grist, R., & Cavanagh, K. (2013) · Journal of Contemporary Psychotherapy · doi:10.1007/s10879-013-9243-y · Cited by 100
A Review of Meta-Analysis Packages in R
Polanin, J. R., Hennessy, E. A., & Tanner-Smith, E. E. (2017) · Journal of Educational and Behavioral Statistics · doi:10.3102/1076998616674315 · Cited by 92
Estimating Population Mean Power Under Conditions of Heterogeneity and Selection for Significance
Brunner, J., & Schimmack, U. (2020) · Meta-Psychology · doi:10.15626/mp.2018.874 · Cited by 76
Mueller, K. F., Meerpohl, J. J., Briel, M., Antes, G., von Elm, E., Lang, B., Motschall, E., Schwarzer, G., & Bassler, D. (2016) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2016.04.015 · Cited by 48
Schindler, S., Hilgard, J., Fritsche, I., Burke, B., & Pfattheicher, S. (2023) · Personality and Social Psychology Review · doi:10.1177/10888683221107267 · Cited by 43
“We applied p-uniform (van Assen et al., 2015; as implemented in the puniform package, van Aert, 2019), the three-parameter selection model (3PSM; Hedges & Vevea, 1996; McShane et al., 2016; as implemented in the weightr package, Coburn & Vevea, 2019), and p-curve (Simonsohn et al., 2014; p-curve…”
MethodHas the evidence for moral licensing been inflated by publication bias?
Kuper, N., & Bott, A. (2019) · Meta-Psychology · doi:10.15626/mp.2018.878 · Cited by 30
Metadata
- Authors
- Hedges, L. V., & Vevea, J. L.
- Venue
- Journal of Educational and Behavioral Statistics
- Published
- 1996
- Volume
- 21
- Issue
- 4
- Pages
- 299
- ISSN
- 1076-9986 (ISSN-L)
- OpenAlex
- W2168599342
- Cited by
- 221
- References
- 4
- FWCI
- 0.60
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.