Selection Models and the File Drawer Problem
Open accessThe DOI matches its Crossref record.
Abstract
Meta-analysis consists of quantitative methods for combining evidence from different studies about a particular issue. A frequent criticism of meta-analysis is that it may be based on a biased sample of all studies that were done. In this paper, we use selection models, or weighted distributions, to deal with one source of bias, namely, the failure to report studies that do not yield statistically significant results. We apply selection models to two approaches that have been suggested for correcting the bias. The fail-safe sample size approach calculates the minimum number of unpublished studies showing nonsignificant results that must have been carried out in order to overturn the conclusion reached from the published studies. The maximum likelihood approach uses a weighted distribution to model the selection bias in the generation of the data and estimates various parameters of interest. We suggest the use of families of weight functions to model plausible biasing mechanisms to study the sensitivity of inferences about effect sizes. By using an example, we show that the maximum likelihood approach has several advantages over the fail-safe sample size approach.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cited by
Operating characteristics of a rank correlation test for publication bias
Begg, C. B., & Mazumdar, M. (1994) · Biometrics · doi:10.2307/2533446 · Cited by 16,955
Duval, S., & Tweedie, R. (2000) · Biometrics · doi:10.1111/j.0006-341x.2000.00455.x · Cited by 14,099
The Handbook of Research Synthesis and Meta-Analysis
(2019) · doi:10.7758/9781610448864 · Cited by 4,553
Publication bias in clinical research
Easterbrook, P. J., Gopalan, R., Berlin, J. A., & Matthews, D. R. (1991) · The Lancet · doi:10.1016/0140-6736(91)90201-y · Cited by 3,109
A Nonparametric “Trim and Fill” Method of Accounting for Publication Bias in Meta-Analysis
Duval, S., & Tweedie, R. (2000) · Journal of the American Statistical Association · doi:10.1080/01621459.2000.10473905 · Cited by 2,955
“In all of these contexts, one well-documented concern is the need to collect all relevant studies, both published and unpublished, if the subsequent inferences are to be valid (Begg 1994; Begg and Mazumdar 1994; Dear and Begg 1992; Egger, Smith, Schneider, and Minder 1997; Gleser and Olkin 1996; Hedges 1992; Iyengar a…”
BackgroundEllis, P. D. (2010) · doi:10.1017/cbo9780511761676 · Cited by 2,461
A modified test for small‐study effects in meta‐analyses of controlled trials with binary endpoints
Harbord, R. M., Egger, M., & Sterne, J. A. C. (2006) · Statistics in Medicine · doi:10.1002/sim.2380 · Cited by 2,267
Null Hypothesis Testing: Problems, Prevalence, and an Alternative
Anderson, D. R., Burnham, K. P., & Thompson, W. L. (2000) · The Journal of Wildlife Management · doi:10.2307/3803199 · Cited by 1,930
Chida, Y., & Steptoe, A. (2008) · Psychosomatic Medicine · doi:10.1097/psy.0b013e31818105ba · Cited by 1,592
Data-Snooping Biases in Tests of Financial Asset Pricing Models
Lo, A. W., & MacKinlay, A. C. (1990) · Review of Financial Studies · doi:10.1093/rfs/3.3.431 · Cited by 1,134
Dissemination and publication of research findings: an updated review of related biases
Song, F., Parekh, Hooper, L., Loke, Y., Ryder, J., Sutton, A., Hing, C., Kwok, C., Pang, C., & Harvey, I. (2010) · Health Technology Assessment · doi:10.3310/hta14080 · Cited by 1,039
Adjusting for publication bias in the presence of heterogeneity
Terrin, N., Schmid, C. H., Lau, J., & Olkin, I. (2003) · Statistics in Medicine · doi:10.1002/sim.1461 · Cited by 809
Publication bias—a problem in interpreting medical data
Begg, C. B., & Berlin, J. A. (1988) · Journal of the Royal Statistical Society. Series A (Statistics in Society) · doi:10.2307/2982993 · Cited by 814
Chida, Y., & Steptoe, A. (2010) · Hypertension · doi:10.1161/hypertensionaha.109.146621 · Cited by 787
The Association of Anger and Hostility With Future Coronary Heart Disease
Chida, Y., & Steptoe, A. (2009) · Journal of the American College of Cardiology · doi:10.1016/j.jacc.2008.11.044 · Cited by 774
Stanley, T. D. (2005) · Journal of Economic Surveys · doi:10.1111/j.0950-0804.2005.00250.x · Cited by 742
Peters, J. L., Sutton, A. J., Jones, D. R., Abrams, K. R., & Rushton, L. (2007) · Statistics in Medicine · doi:10.1002/sim.2889 · Cited by 716
O'Dea, R. E., Lagisz, M., Jennions, M. D., Koricheva, J., Noble, D. W. A., Parker, T. H., Gurevitch, J., Page, M. J., Stewart, G., Moher, D., et al. (2021) · Biological Reviews · doi:10.1111/brv.12721 · Cited by 697
Chida, Y., & Hamer, M. (2008) · Psychological Bulletin · doi:10.1037/a0013342 · Cited by 667
Ferguson, C. J., & Heene, M. (2012) · Perspectives on Psychological Science · doi:10.1177/1745691612459059 · Cited by 603
Correcting for Bias in Psychology: A Comparison of Meta-Analytic Methods
Carter, E. C., Schönbrodt, F. D., Gervais, W. M., & Hilgard, J. (2019) · Advances in Methods and Practices in Psychological Science · doi:10.1177/2515245919847196 · Cited by 588
“We employed the threeparameter selection model (3PSM) as developed by Iyengar and Greenhouse (1988) and recommended by McShane et al. (2016).”
MethodMeta-analysis: synthesizing research findings in ecology and evolution
Arnqvist, G., & Wooster, D. (1995) · Trends in Ecology & Evolution · doi:10.1016/s0169-5347(00)89073-4 · Cited by 570
Some Statistical Methods for Combining Experimental Results
Laird, N. M., & Mosteller, F. (1990) · International Journal of Technology Assessment in Health Care · doi:10.1017/s0266462300008916 · Cited by 568
Weight management using a meal replacement strategy: meta and pooling analysis from six studies.
Heymsfield, S. B., van Mierlo, C. A. J., van der Knaap, H. C. M., Heo, M., & Frier, H. I. (2003) · International Journal of Obesity · doi:10.1038/sj.ijo.0802258 · Cited by 557
Meta-analysis, funnel plots and sensitivity analysis
Copas, J. (2000) · Biostatistics · doi:10.1093/biostatistics/1.3.247 · Cited by 503
Cicchetti, D. V. (1991) · Behavioral and Brain Sciences · doi:10.1017/s0140525x00065675 · Cited by 496
Publication Bias in Research Synthesis: Sensitivity Analysis Using A Priori Weight Functions.
Vevea, J. L., & Woods, C. M. (2005) · Psychological Methods · doi:10.1037/1082-989x.10.4.428 · Cited by 461
Sterling, T. D., Rosenbaum, W. L., & Weinkam, J. J. (1995) · The American Statistician · doi:10.1080/00031305.1995.10476125 · Cited by 453
Investigating and Dealing with Publication and Other Biases
Sterne, J. A. C., Egger, M., & Smith, G. D. (2001) · Systematic Reviews in Health Care · doi:10.1002/9780470693926.ch11 · Cited by 449
Publication and related biases.
Song, Eastwood, Gilbody, Duley, & Sutton (2000) · Health Technology Assessment · doi:10.3310/hta4100 · Cited by 427
McShane, B. B., Böckenholt, U., & Hansen, K. T. (2016) · Perspectives on Psychological Science · doi:10.1177/1745691616662243 · Cited by 398
A General Linear Model for Estimating Effect Size in the Presence of Publication Bias
Vevea, J. L., & Hedges, L. V. (1995) · Psychometrika · doi:10.1007/bf02294384 · Cited by 413
Religiosity/Spirituality and Mortality
Chida, Y., Steptoe, A., & Powell, L. H. (2009) · Psychotherapy and Psychosomatics · doi:10.1159/000190791 · Cited by 409
Fleiss, J. L., & Gross, A. J. (1991) · Journal of Clinical Epidemiology · doi:10.1016/0895-4356(91)90261-7 · Cited by 404
Is Ego Depletion Real? An Analysis of Arguments
Friese, M., Loschelder, D. D., Gieseler, K., Frankenbach, J., & Inzlicht, M. (2019) · Personality and Social Psychology Review · doi:10.1177/1088868318762183 · Cited by 393
Rating reliability and representation validity in scenic landscape assessments
Palmer, J. F., & Hoffman, R. E. (2001) · Landscape and Urban Planning · doi:10.1016/s0169-2046(01)00133-5 · Cited by 367
Modeling Publication Selection Effects in Meta-Analysis
Hedges, L. V. (1992) · Statistical Science · doi:10.1214/ss/1177011364 · Cited by 359
Sharpe, D. (1997) · Clinical Psychology Review · doi:10.1016/s0272-7358(97)00056-1 · Cited by 334
Publication bias examined in meta-analyses from psychology and medicine: A meta-meta-analysis
van Aert, R. C. M., Wicherts, J. M., & van Assen, M. A. L. M. (2019) · PLOS ONE · doi:10.1371/journal.pone.0215052 · Cited by 334
Advancing Maternal Age Is Associated With Increasing Risk for Autism: A Review and Meta-Analysis
Sandin, S., Hultman, C. M., Kolevzon, A., Gross, R., MacCabe, J. H., & Reichenberg, A. (2012) · Journal of the American Academy of Child & Adolescent Psychiatry · doi:10.1016/j.jaac.2012.02.018 · Cited by 333
Michels, K. B., & Yusuf, S. (1995) · Circulation · doi:10.1161/01.cir.91.2.476 · Cited by 326
Understanding the Role and Methods of Meta-Analysis in IS Research
King, W. R., & He, J. (2005) · Communications of the Association for Information Systems · doi:10.17705/1cais.01632 · Cited by 319
Publication Bias (The "File-Drawer Problem") in Scientific Inference
Scargle, J. D. (1999) · arXiv · doi:10.48550/arxiv.physics/9909033 · Cited by 313
Modelling publication bias in meta-analysis: a review
Sutton, A. J., Song, F., Gilbody, S. M., & Abrams, K. R. (2000) · Statistical Methods in Medical Research · doi:10.1177/096228020000900503 · Cited by 288
A Solution to Separation in Binary Response Models
Zorn, C. (2005) · Political Analysis · doi:10.1093/pan/mpi009 · Cited by 280
8. Data-Snooping Biases in Tests of Financial Asset Pricing Models
(2011) · A Non-Random Walk Down Wall Street · doi:10.1515/9781400829095.213 · Cited by 279
Comparing meta-analyses and preregistered multiple-laboratory replication projects
Kvarven, A., Strømland, E., & Johannesson, M. (2019) · Nature Human Behaviour · doi:10.1038/s41562-019-0787-z · Cited by 271
Palmer, A. R. (1999) · The American Naturalist · doi:10.1086/303223 · Cited by 264
Invited review: Use of meta-analysis in animal health and reproduction: Methods and applications
Lean, I. J., Rabiee, A. R., Duffield, T. F., & Dohoo, I. R. (2009) · Journal of Dairy Science · doi:10.3168/jds.2009-2140 · Cited by 257
Statistical methods for dealing with publication bias in meta‐analysis
Jin, Z., Zhou, X., & He, J. (2015) · Statistics in Medicine · doi:10.1002/sim.6342 · Cited by 252
PIETRZAK, R., MOLLICA, C., MARUFF, P., & SNYDER, P. (2006) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2006.10.002 · Cited by 214
Outcome selection bias in meta-analysis
Williamson, P. R., Gamble, C., Altman, D. G., & Hutton, J. L. (2005) · Statistical Methods in Medical Research · doi:10.1191/0962280205sm415oa · Cited by 188
Hedges, L. V. (1992) · Journal of Educational Statistics · doi:10.3102/10769986017004279 · Cited by 186
“One promising approach to the problem involves modeling the selection process as part of the estimation procedure (see Hedges, 1992; Iyengar & Greenhouse, 1988).”
MethodA historical review of publication bias
Marks‐Anglin, A., & Chen, Y. (2020) · Research Synthesis Methods · doi:10.1002/jrsm.1452 · Cited by 165
Publication Bias in Meta-Analysis: Confidence Intervals for Rosenthal’s Fail-Safe Number
Fragkos, K. C., Tsagris, M., & Frangos, C. C. (2014) · International Scholarly Research Notices · doi:10.1155/2014/825383 · Cited by 159
Kraemer, H. C., Gardner, C., Brooks, J. O., & Yesavage, J. A. (1998) · Psychological Methods · doi:10.1037/1082-989x.3.1.23 · Cited by 141
A kinked meta‐regression model for publication bias correction
Bom, P. R. D., & Rachinger, H. (2019) · Research Synthesis Methods · doi:10.1002/jrsm.1352 · Cited by 133
Adjustment for reporting bias in network meta-analysis of antidepressant trials
Trinquart, L., Chatellier, G., & Ravaud, P. (2012) · BMC Medical Research Methodology · doi:10.1186/1471-2288-12-150 · Cited by 122
P-hacking and publication bias interact to distort meta-analytic effect size estimates
Friese, M., & Frankenbach, J. (2020) · Psychological Methods · doi:10.1037/met0000246 · Cited by 122
Wilson, D. B. (2009) · Journal of Experimental Criminology · doi:10.1007/s11292-009-9085-5 · Cited by 86
“The possibility that a collection of studies represents a biased sample of all relevant empirical work on a topic has long been a concern among those conducting a meta-analysis or more broadly any form of systematic review (e.g., Hedges 1992; Iyengar and Greenhouse 1988; Rosenthal 1979; Rothstein et al. 2005; Smith 19…”
BackgroundGuidelines for Meta-Analyses of Counseling Psychology Research
Quintana, S. M., & Minami, T. (2006) · The Counseling Psychologist · doi:10.1177/0011000006286991 · Cited by 83
Combining independent p values: Extensions of the Stouffer and binomial methods.
Darlington, R. B., & Hayes, A. F. (2000) · Psychological Methods · doi:10.1037/1082-989x.5.4.496 · Cited by 66
Meta-analysis, can it ever fail?
Kotiaho, J. S., & Tomkins, J. L. (2002) · Oikos · doi:10.1034/j.1600-0706.2002.960316.x · Cited by 52
Mueller, K. F., Meerpohl, J. J., Briel, M., Antes, G., von Elm, E., Lang, B., Motschall, E., Schwarzer, G., & Bassler, D. (2016) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2016.04.015 · Cited by 48
Has the evidence for moral licensing been inflated by publication bias?
Kuper, N., & Bott, A. (2019) · Meta-Psychology · doi:10.15626/mp.2018.878 · Cited by 30
Design characteristics that influence attrition in geriatric antidepressant trials: meta‐analysis
Heo, M., Papademetriou, E., & Meyers, B. S. (2009) · International Journal of Geriatric Psychiatry · doi:10.1002/gps.2211 · Cited by 18
Not Published Is Not Perished: Addressing Publication Bias in Meta-Analytic Studies in Communication
Sun, Y., & Pan, Z. (2020) · Human Communication Research · doi:10.1093/hcr/hqz015 · Cited by 18
“The three-parameter selection models (3PSM), for example, contain parameters of effect size and heterogeneity (data model) and a weight function that specifies the form and severity of significance-based selection (selection model; e.g., Iyengar & Greenhouse, 1988; Vevea & Hedges, 1995).”
MethodChen, L., Benjamin, R., Lai, A., & Heine, S. (2022) · doi:10.31234/osf.io/kuhy6 · Cited by 14
Chen, L., Benjamin, R., Guo, Y., Lai, A., & Heine, S. J. (2023) · Research Square · doi:10.21203/rs.3.rs-1254756/v1 · Cited by 5
Does Retrieval Practice Protect Memory Against Stress? A Meta-Analysis
Mihaylova, M., Zocca, M., Kliegel, M., & Rothen, N. (2025) · Swiss Psychology Open · doi:10.5334/spo.101 · Cited by 1
Xiao, Q., Li, L. C., Au, Y. L., Tan, S. N., Chung, W. T., & Feldman, G. (2024) · International Review of Social Psychology · doi:10.5334/irsp.945
Metadata
- Authors
- Iyengar, S., & Greenhouse, J. B.
- Venue
- Statistical Science
- Publisher
- Institute of Mathematical Statistics
- Published
- 1988-02-01
- Volume
- 3
- Issue
- 1
- ISSN
- 0883-4237 (ISSN-L)
- OpenAlex
- W2054305681
- Cited by
- 384
- References
- 0
- FWCI
- 35.21
- Type
- journal article
- Language
- en
- Open access
- bronze
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.