The DOI matches its Crossref record; 40 of the 40 references it cites were found in a registry or an index.
A Nonparametric “Trim and Fill” Method of Accounting for Publication Bias in Meta-Analysis
Abstract
Meta-analysis collects and synthesizes results from individual studies to estimate an overall effect size. If published studies are chosen, say through a literature review, then an inherent selection bias may arise, because, for example, studies may tend to be published more readily if they are statistically significant, or deemed to be more “interesting” in terms of the impact of their outcomes. We develop a simple rank-based data augmentation technique, formalizing the use of funnel plots, to estimate and adjust for the numbers and outcomes of missing studies. Several nonparametric estimators are proposed for the number of missing studies, and their properties are developed analytically and through simulations. We apply the method to simulated and epidemiological datasets and show that it is both effective and consistent with other criteria in the literature.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Publication bias was investigated by applying the trim and fill method, where effect sizes are recalculated to adjust for supposed publication biases (Duval & Tweedie, 2000).”
From Taking stock: A meta-analysis of the effects of foreign language enjoyment (2022) · cites this paper as Duval & Tweedie, 2000
In the citing paper
“The trim‐and‐fill procedure determined that nine studies should be added to create symmetry.”
From Problem-oriented policing for reducing crime and disorder: An updated systematic review and meta-analysis (2020) · cites this paper as Duval and Tweedie (2000)
In the citing paper
“Like many systematic reviews, our meta‐analyses used the trim‐and‐fill procedure to explore whether publication bias might be affecting the results and to estimate how the reported effects would change if the bias were to be removed (Duval & Tweedie, 2000; Duval, 2005).”
From Hot spots policing of small geographic areas effects on crime (2019) · cites this paper as Duval & Tweedie, 2000
In the citing paper
“When we encountered publication bias, we conducted a trim and fill adjusted analysis 25 to remove the most extreme small studies from the positive side of the funnel plot, and recalculated the effect size iteratively, until the funnel plot was symmetrical around the (new) effect size.”
From Diabetes mellitus in people with schizophrenia, bipolar disorder and major depressive disorder: a systematic review and large scale meta‐analysis (2016) · cites this paper as [25]
In the citing paper
“Variations in the risk for glucose abnormalities are evident in the literature, with the highest risk being associated with clozapine, olanzapine and quetiapine in carefully designed studies 25,34,35 .”
From Diabetes mellitus in people with schizophrenia, bipolar disorder and major depressive disorder: a systematic review and large scale meta‐analysis (2016) · cites this paper as [25]
Citations
Cites
Bias in meta-analysis detected by a simple, graphical test
Egger, M., Smith, G. D., Schneider, M., & Minder, C. (1997) · BMJ · doi:10.1136/bmj.315.7109.629 · Cited by 56,722
Meta-analysis in clinical trials
DerSimonian, R., & Laird, N. (1986) · Controlled Clinical Trials · doi:10.1016/0197-2456(86)90046-2 · Cited by 39,552
Operating characteristics of a rank correlation test for publication bias
Begg, C. B., & Mazumdar, M. (1994) · Biometrics · doi:10.2307/2533446 · Cited by 16,955
Duval, S., & Tweedie, R. (2000) · Biometrics · doi:10.1111/j.0006-341x.2000.00455.x · Cited by 14,099
Self-citationWasserman, S. S., Hedges, L. V., & Olkin, I. (1988) · Journal of Educational Statistics · doi:10.2307/1164953 · Cited by 10,377
Casella, G., & Berger, R. L. (1993) · Chapman & Hall/CRC eBooks · doi:10.1201/9781003456285 · Cited by 5,835
The Handbook of Research Synthesis.
Howel, D., Cooper, H., & Hedges, L. V. (1995) · Biometrics · doi:10.2307/2533295 · Cited by 3,657
Publication bias in clinical research
Easterbrook, P. J., Gopalan, R., Berlin, J. A., & Matthews, D. R. (1991) · The Lancet · doi:10.1016/0140-6736(91)90201-y · Cited by 3,109 · cited 1×
“The negative binomial forms here are reminiscent of those used by Eberly and Casella (1999) and Givens et al.”
MethodSumming up: The Science of Reviewing Research
Wilkinson, I., Light, R. J., & Pillemer, D. B. (1987) · Journal of Educational Statistics · doi:10.2307/1164691 · Cited by 1,486
Light, R. J., & Pillemer, D. B. (1984) · Harvard University Press eBooks · doi:10.4159/9780674040243 · Cited by 1,422
Monastyrsky, M. (1999) · Birkhäuser Boston eBooks · doi:10.1007/978-0-8176-4779-7_2 · Cited by 1,081
Nonparametric and Distribution-Free Methods for the Social Sciences.
Bassler, J. F., Marascuilo, L. A., & McSweeney, M. (1978) · Journal of the American Statistical Association · doi:10.2307/2286625 · Cited by 981
The Handbook of Research Synthesis
Ziegel, E. R., Cooper, H., & Hedges, L. V. (1995) · Technometrics · doi:10.2307/1269651 · Cited by 936
Factors Influencing Publication of Research Results
Dickersin, K. (1992) · JAMA · doi:10.1001/jama.1992.03480030052036 · Cited by 931
A note on graphical presentation of estimated odds ratios from several clinical trials
Galbraith, R. F. (1988) · Statistics in Medicine · doi:10.1002/sim.4780070807 · Cited by 663
Selection Models and the File Drawer Problem
Iyengar, S., & Greenhouse, J. B. (1988) · Statistical Science · doi:10.1214/ss/1177013012 · Cited by 388
Modeling Publication Selection Effects in Meta-Analysis
Hedges, L. V. (1992) · Statistical Science · doi:10.1214/ss/1177011364 · Cited by 359
Combining Information: Statistical Issues and Opportunities for Research
Draper, D. L., Gaver, D. P., Goel, P., Greenhouse, J. B., Hedges, L. V., Morris, C., Tucker, J., & Waternaux, C. (1993) · Cited by 279
INCORPORATING VARIABILITY IN ESTIMATES OF HETEROGENEITY IN THE RANDOM EFFECTS MODEL IN META-ANALYSIS
Biggerstaff, B. J., & Tweedie, R. L. (1997) · Statistics in Medicine · doi:10.1002/(sici)1097-0258(19970415)16:7 · Cited by 272
Angus, J. E., Casella, G., & Berger, R. L. (1991) · Technometrics · doi:10.2307/1269442 · Cited by 220
Meta-Analysis in Medicine and Health Policy
Stangl, D. K. (2000) · CRC Press eBooks · doi:10.1201/9780203909935 · Cited by 213
An Approach for Assessing Publication Bias Prior to Performing a Meta-Analysis
Dear, K., & Begg, C. (1992) · Statistical Science · doi:10.1214/ss/1177011363 · Cited by 202
Dickersin, K. (1992) · JAMA · doi:10.1001/jama.267.3.374 · Cited by 193
Givens, G. H., Smith, D. D., & Tweedie, R. L. (1997) · Statistical Science · doi:10.1214/ss/1030037958 · Cited by 127
Self-citationPublication Bias and Research on Passive Smoking
Misakian, A. L., & Bero, L. A. (1998) · JAMA · doi:10.1001/jama.280.3.250 · Cited by 126
Pastor, M., & Mollenkopf, J. H. (2016) · Cornell University Press eBooks · doi:10.7591/cornell/9781501702662.003.0010 · Cited by 94
Environmental Tobacco Smoke and Mortality
Strachan, D. P. (1993) · Journal of Epidemiology & Community Health · doi:10.1136/jech.47.2.163-a · Cited by 77
Some Progress and Problems in Meta-Analysis of Clinical Trials
Mosteller, F. C., & Chalmers, T. C. (1992) · Statistical Science · doi:10.1214/ss/1177011362 · Cited by 64
Bayesian meta-analysis, with application to studies of ETS and lung cancer
Tweedie, R. L., Scott, D. J., Biggerstaff, B. J., & Mengersen, K. L. (1996) · Lung Cancer · doi:10.1016/s0169-5002(96)90222-6 · Cited by 51
Self-citationTHE IMPACT OF METHOD CHOICE ON META‐ANALYSIS
Mengersen, K. L., Tweedie, R. L., & Biggerstaff, B. J. (1995) · Australian Journal of Statistics · doi:10.1111/j.1467-842x.1995.tb00869.x · Cited by 42
Self-citationDIFFERENTIAL RECALL BIAS AND SPURIOUS ASSOCIATIONS IN CASE/CONTROL STUDIES
Barry, D. (1996) · Statistics in Medicine · doi:10.1002/(sici)1097-0258(19961215)15:23 · Cited by 39
Environmental Tobacco Smoke and Mortality
Law, M. (1993) · British Journal of Cancer · doi:10.1038/bjc.1993.77 · Cited by 25
Olkin, I. (1992) · Statistical Science · doi:10.1214/ss/1177011361 · Cited by 19
Estimating a Summarized Odds Ratio whilst Eliminating Publication Bias in Meta-analysis *
Sugita, M., Kanamor, M., Izuno, T., & Miyakawa, M. (1992) · Japanese Journal of Clinical Oncology · doi:10.1093/oxfordjournals.jjco.a039577 · Cited by 17
JAPANESE JOURNAL OF CLINICAL ONCOLOGY
(2011) · JNCI Journal of the National Cancer Institute · doi:10.1093/jnci/djr110 · Cited by 12
Sugita, M., Yamaguchi, N., Izuno, T., Kanamori, M., & Kasuga, H. (1994) · PubMed · Cited by 8
Begg, C., DuMouchel, W., Harris, J., Dobson, A. P., Dear, K. B. G., Givens, G. H., Smith, D., & Tweedle, R. (1997) · Statistical Science · Cited by 8
Bayesian Estimation of the Number of Unseen Studies in a Meta-Analysis
Eberly, L. E., & Casella, G. (1999) · Journal of Official Statistics · Cited by 7
Gibson, T. (1994) · Lara D. Veeken · doi:10.1093/rheumatology/33.6.589 · Cited by 6
Estimating the Number of Unseen Studies in a Meta-Analysis
Eberly, L. E., & Casella, G. (1997) · Cited by 3
Research works printed but not linked: 10
- 1983, British Medical Journal, 280, 508
- Cooper H., 1998, Synthesizing Research
- Dear K., 1997, Statistical Science, 12, 245
- DuMouchel W., 1997, Statistical Science, 12, 244
- Duval S. J., 1999, “Effects of Publication Bias in Meta-Analysis,”
- Duval S. J., “Robustness of the ‘Trim and Fill’ publication bias approach to different mechanisms for study suppression,”
- Eberly L. E., 1999, Journal of Official Statistics, 15
- Hedges L., 1985, Statistical Methods for Meta-Analysis
- Sugita M., 1994, Tokai Journal of Experimental and Clinical Medicine, 19, 29
- Sutton A. J., 1999, “The Impact of Publication Bias on Meta-Analyses Within the Cochrane Database of Systematic Reviews,”
Cited by
Sorted by citations
Conducting Meta-Analyses in R with the metafor Package
Viechtbauer, W. (2010) · Journal of Statistical Software · doi:10.18637/jss.v036.i03 · Cited by 20,464
Hao (2015) · doi:10.1002/14651858 · Cited by 11,321
A meta-analytic test of intergroup contact theory.
Pettigrew, T. F., & Tropp, L. R. (2006) · Journal of personality and social psychology · doi:10.1037/0022-3514.90.5.751 · Cited by 9,708
Social Relationships and Mortality Risk: A Meta-analytic Review
Holt-Lunstad, J., Smith, T. B., & Layton, J. B. (2010) · PLoS Medicine · doi:10.1371/journal.pmed.1000316 · Cited by 7,272
“Second, we employed the ‘‘trim and fill’’ methodology described by Duval and Tweedie [178,179] to estimate the number of studies missing due to publication bias, but this analysis failed to reveal any studies that would need to be created on the opposite side of the distribution, meaning that adjustment to the omnibus…”
MethodHow to perform a meta-analysis with R: a practical tutorial
Balduzzi, S., Rücker, G., & Schwarzer, G. (2019) · Evidence Based Mental Health · doi:10.1136/ebmental-2019-300117 · Cited by 5,827
Metadata
- Authors
- Duval, S., & Tweedie, R.
- Affiliations
- Colorado State University (US)
- University of Minnesota (US)
- ORCID
- Duval, Sue
- Venue
- Journal of the American Statistical Association
- Publisher
- Informa UK Limited
- Published
- 2000-03
- Volume
- 95
- Issue
- 449
- Pages
- 89-98
- ISSN
- 0162-1459 (ISSN-L)
- 1537-274X
- OpenAlex
- W2043565183
- Cited by
- 2,955
- References
- 40
- Citations by year
- 2026: 106 · 2025: 167 · 2024: 189 · 2023: 171 · 2022: 233 · 2021: 275 · 2020: 259 · 2019: 179 · 2018: 184 · 2017: 160 · 2016: 142 · 2015: 149 · 2014: 128 · 2013: 91 · 2012: 101
- FWCI
- 21.09
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Journal of the American Statistical Association (publishedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
- Social Sciences › Library and Information Sciences › scientometrics and bibliometrics research
- Keywords
- publication bias, trim and fill, missing studies, funnel plot, nonparametric estimators, selection bias, meta-analysis, effect size estimation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
