The DOI matches its Crossref record; 40 of the 40 references it cites were found in a registry or an index.
A Nonparametric “Trim and Fill” Method of Accounting for Publication Bias in Meta-Analysis
Abstract
Meta-analysis collects and synthesizes results from individual studies to estimate an overall effect size. If published studies are chosen, say through a literature review, then an inherent selection bias may arise, because, for example, studies may tend to be published more readily if they are statistically significant, or deemed to be more “interesting” in terms of the impact of their outcomes. We develop a simple rank-based data augmentation technique, formalizing the use of funnel plots, to estimate and adjust for the numbers and outcomes of missing studies. Several nonparametric estimators are proposed for the number of missing studies, and their properties are developed analytically and through simulations. We apply the method to simulated and epidemiological datasets and show that it is both effective and consistent with other criteria in the literature.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Publication bias was investigated by applying the trim and fill method, where effect sizes are recalculated to adjust for supposed publication biases (Duval & Tweedie, 2000).”
From Taking stock: A meta-analysis of the effects of foreign language enjoyment (2022) · cites this paper as Duval & Tweedie, 2000
In the citing paper
“The trim‐and‐fill procedure determined that nine studies should be added to create symmetry.”
From Problem-oriented policing for reducing crime and disorder: An updated systematic review and meta-analysis (2020) · cites this paper as Duval and Tweedie (2000)
In the citing paper
“Like many systematic reviews, our meta‐analyses used the trim‐and‐fill procedure to explore whether publication bias might be affecting the results and to estimate how the reported effects would change if the bias were to be removed (Duval & Tweedie, 2000; Duval, 2005).”
From Hot spots policing of small geographic areas effects on crime (2019) · cites this paper as Duval & Tweedie, 2000
In the citing paper
“When we encountered publication bias, we conducted a trim and fill adjusted analysis 25 to remove the most extreme small studies from the positive side of the funnel plot, and recalculated the effect size iteratively, until the funnel plot was symmetrical around the (new) effect size.”
From Diabetes mellitus in people with schizophrenia, bipolar disorder and major depressive disorder: a systematic review and large scale meta‐analysis (2016) · cites this paper as [25]
In the citing paper
“Variations in the risk for glucose abnormalities are evident in the literature, with the highest risk being associated with clozapine, olanzapine and quetiapine in carefully designed studies 25,34,35 .”
From Diabetes mellitus in people with schizophrenia, bipolar disorder and major depressive disorder: a systematic review and large scale meta‐analysis (2016) · cites this paper as [25]
Citations
Cites
Bias in meta-analysis detected by a simple, graphical test
Egger, M., Smith, G. D., Schneider, M., & Minder, C. (1997) · BMJ · doi:10.1136/bmj.315.7109.629 · Cited by 56,722
Meta-analysis in clinical trials
DerSimonian, R., & Laird, N. (1986) · Controlled Clinical Trials · doi:10.1016/0197-2456(86)90046-2 · Cited by 39,552
Operating characteristics of a rank correlation test for publication bias
Begg, C. B., & Mazumdar, M. (1994) · Biometrics · doi:10.2307/2533446 · Cited by 16,955
Duval, S., & Tweedie, R. (2000) · Biometrics · doi:10.1111/j.0006-341x.2000.00455.x · Cited by 14,099
Self-citationWasserman, S. S., Hedges, L. V., & Olkin, I. (1988) · Journal of Educational Statistics · doi:10.2307/1164953 · Cited by 10,377
Casella, G., & Berger, R. L. (1993) · Chapman & Hall/CRC eBooks · doi:10.1201/9781003456285 · Cited by 5,835
The Handbook of Research Synthesis.
Howel, D., Cooper, H., & Hedges, L. V. (1995) · Biometrics · doi:10.2307/2533295 · Cited by 3,657
Publication bias in clinical research
Easterbrook, P. J., Gopalan, R., Berlin, J. A., & Matthews, D. R. (1991) · The Lancet · doi:10.1016/0140-6736(91)90201-y · Cited by 3,109 · cited 1×
“The negative binomial forms here are reminiscent of those used by Eberly and Casella (1999) and Givens et al.”
MethodSumming up: The Science of Reviewing Research
Wilkinson, I., Light, R. J., & Pillemer, D. B. (1987) · Journal of Educational Statistics · doi:10.2307/1164691 · Cited by 1,486
Light, R. J., & Pillemer, D. B. (1984) · Harvard University Press eBooks · doi:10.4159/9780674040243 · Cited by 1,422
Research works printed but not linked: 10
- 1983, British Medical Journal, 280, 508
- Cooper H., 1998, Synthesizing Research
- Dear K., 1997, Statistical Science, 12, 245
- DuMouchel W., 1997, Statistical Science, 12, 244
- Duval S. J., 1999, “Effects of Publication Bias in Meta-Analysis,”
- Duval S. J., “Robustness of the ‘Trim and Fill’ publication bias approach to different mechanisms for study suppression,”
- Eberly L. E., 1999, Journal of Official Statistics, 15
- Hedges L., 1985, Statistical Methods for Meta-Analysis
- Sugita M., 1994, Tokai Journal of Experimental and Clinical Medicine, 19, 29
- Sutton A. J., 1999, “The Impact of Publication Bias on Meta-Analyses Within the Cochrane Database of Systematic Reviews,”
Cited by
Sorted by citations
Items 1,001–1,059 of 1,059
A Critical Review of the Research on the Extreme Male Brain Theory and Digit Ratio (2D:4D)
Teatero, M. L., & Netley, C. (2013) · Journal of Autism and Developmental Disorders · doi:10.1007/s10803-013-1819-6 · Cited by 78
Exercise and bone mineral density in men: A meta-analysis of randomized controlled trials
Kelley, G. A., Kelley, K. S., & Kohrt, W. M. (2012) · Bone · doi:10.1016/j.bone.2012.11.031 · Cited by 75
Does caffeine added to carbohydrate provide additional ergogenic benefit for endurance?
Conger, S. A., Warren, G. L., Hardy, M. A., & Millard-Stafford, M. L. (2011) · International Journal of Sport Nutrition and Exercise Metabolism · doi:10.1123/ijsnem.21.1.71 · Cited by 71
Sitko, K., Bewick, B. M., Owens, D., & Masterson, C. (2020) · Schizophrenia Bulletin Open · doi:10.1093/schizbullopen/sgaa023 · Cited by 70
“Publication bias was examined using the Begg and Mazumdar’s Rank Order Correlation test, 32 Egger’s regression intercept test, 33 and Duval and Tweedie’s trim-and-fill procedure 34 using the Comprehensive Meta-Analysis software (version 3).”
Pearcy, C. P., Anderson, R. A., Egan, S. J., & Rees, C. S. (2016) · Journal of Behavior Therapy and Experimental Psychiatry · doi:10.1016/j.jbtep.2015.12.007 · Cited by 66
Wilson, S. J., & Tanner-Smith, E. E. (2013) · Journal of the Society for Social Work and Research · doi:10.5243/jsswr.2013.22 · Cited by 66
Indahlastari, A., Hardcastle, C., Albizu, A., Alvarez-Alvarado, S., Boutzoukas, E. M., Evangelista, N. D., Hausman, H. K., Kraft, J., Langer, K., & Woods, A. J. (2021) · Neuropsychiatric Disease and Treatment · doi:10.2147/ndt.s259499 · Cited by 64
Police stops to reduce crime: A systematic review and meta-analysis
Petersen, K., Weisburd, D., Fay, S., Eggins, E., & Mazerolle, L. (2023) · Campbell Systematic Reviews: Better Evidence for a Better World · doi:10.1002/cl2.1302 · Cited by 63
School Start Times, Sleep, and Youth Outcomes: A Meta-analysis
Yip, T., Wang, Y., Xie, M., Ip, P. S., Fowle, J., & Buckhalt, J. (2022) · Pediatrics · doi:10.1542/peds.2021-054068 · Cited by 58
Cognitive behavioral therapy for hoarding disorder: An updated meta-analysis
Rodgers, N., McDonald, S., & Wootton, B. M. (2021) · Journal of Affective Disorders · doi:10.1016/j.jad.2021.04.067 · Cited by 58
Does unstable mood increase risk of suicide? Theory, research and practice
Palmier-Claus, J. E., Taylor, P. J., Varese, F., & Pratt, D. (2012) · Journal of Affective Disorders · doi:10.1016/j.jad.2012.05.030 · Cited by 57
Tutal, Ö., & Yazar, T. (2021) · Asia Pacific Education Review · doi:10.1007/s12564-021-09706-9 · Cited by 57
Cho, M.-K., & Kim, M. Y. (2021) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph182312750 · Cited by 57
Maguire, P. N., Clark, G. I., & Wootton, B. M. (2018) · Journal of Anxiety Disorders · doi:10.1016/j.janxdis.2018.10.002 · Cited by 50
Perera, M. P. N., Mallawaarachchi, S., Miljevic, A., Bailey, N. W., Herring, S. E., & Fitzgerald, P. B. (2021) · Biological Psychiatry: Cognitive Neuroscience and Neuroimaging · doi:10.1016/j.bpsc.2021.03.010 · Cited by 51
Kotfis, K., Jamioł-Milc, D., Skonieczna-Żydecka, K., Folwarski, M., & Stachowska, E. (2020) · Nutrients · doi:10.3390/nu12103105 · Cited by 51
Juvenile Transfer and the Specific Deterrence Hypothesis: Systematic Review and Meta‐Analysis
Zane, S. N., Welsh, B. C., & Mears, D. P. (2016) · Criminology & Public Policy · doi:10.1111/1745-9133.12222 · Cited by 48
Dunn, C., & Sicouri, G. (2022) · Behaviour Change · doi:10.1017/bec.2022.13 · Cited by 48
Challenging the challenge hypothesis on testosterone in fathers: Limited meta-analytic support
Meijer, W. M., van IJzendoorn, M. H., & Bakermans - Kranenburg, M. J. (2019) · Psychoneuroendocrinology · doi:10.1016/j.psyneuen.2019.104435 · Cited by 49
Wilson, E. J., Abbott, M. J., & Norton, A. R. (2023) · Journal of Anxiety Disorders · doi:10.1016/j.janxdis.2023.102729 · Cited by 52
Ursoniu, S., Sahebkar, A., Serban, M.-C., Antal, D., Mikhailidis, D. P., Cicero, A., Athyros, V., Rizzo, M., Rysz, J., & Banach, M. (2017) · Nutrition Reviews · doi:10.1093/nutrit/nuw063 · Cited by 48
Amiri, M., Raeisi-Dehkordi, H., Sarrafzadegan, N., Forbes, S. C., & Salehi-Abargouei, A. (2020) · Nutrition, Metabolism and Cardiovascular Diseases · doi:10.1016/j.numecd.2020.06.007 · Cited by 54
Roberts, G. J., Cho, E., Garwood, J. D., Goble, G. H., Robertson, T., & Hodges, A. (2020) · Educational Psychology Review · doi:10.1007/s10648-019-09485-1 · Cited by 47
Montessori education's impact on academic and nonacademic outcomes: A systematic review
Randolph, J. J., Bryson, A., Menon, L., Henderson, D. K., Kureethara Manuel, A., Michaels, S., rosenstein, D. L. W., McPherson, W., O'Grady, R., & Lillard, A. S. (2023) · Campbell Systematic Reviews: Better Evidence for a Better World · doi:10.1002/cl2.1330 · Cited by 44
Walking and Non‐HDL‐C in Adults: A Meta‐Analysis of Randomized Controlled Trials
Kelley, G. A., Kelley, K. S., & Tran, Z. V. (2005) · Preventive Cardiology · doi:10.1111/j.1520-037x.2005.3474.x · Cited by 46
Simpson, R., Simpson, S., Ramparsad, N., Lawrence, M., Booth, J., & Mercer, S. W. (2020) · Multiple Sclerosis and Related Disorders · doi:10.1016/j.msard.2019.101493 · Cited by 45
Winkley, K., Upsher, R., Stahl, D., Pollard, D., Brennan, A., Heller, S., & Ismail, K. (2020) · Diabetic Medicine · doi:10.1111/dme.14264 · Cited by 47
Barnes, E. L., Holubar, S. D., & Herfarth, H. H. (2021) · Journal of Crohn's and Colitis · doi:10.1093/ecco-jcc/jjab025 · Cited by 42
Psychological treatment of anxiety in primary care: a meta-analysis
Seekles, W., Cuijpers, P., Kok, R., Beekman, A., van Marwijk, H., & van Straten, A. (2013) · Psychological Medicine · doi:10.1017/s0033291712000670 · Cited by 41
Winkley, K., Upsher, R., Stahl, D., Pollard, D., Brennan, A., Heller, S. R., & Ismail, K. (2020) · BMJ Open Diabetes Research & Care · doi:10.1136/bmjdrc-2019-001150 · Cited by 41
Terpstra, J. A., van der Vaart, R., van Beugen, S., van Eersel, R. A., Gkika, I., Erdős, D., Schmidt, J., Radstake, C., Kloppenburg, M., van Middendorp, H., et al. (2022) · Internet Interventions · doi:10.1016/j.invent.2022.100587 · Cited by 38
Pfefferbaum, B., Nitiéma, P., & Newman, E. (2019) · Journal of Traumatic Stress · doi:10.1002/jts.22391 · Cited by 41
Betz, L. T., Brambilla, P., Ilankovic, A., Premkumar, P., Kim, M.-S., Raffard, S., Bayard, S., Hori, H., Lee, K.-U., Lee, S. J., et al. (2019) · Schizophrenia Research · doi:10.1016/j.schres.2018.09.009 · Cited by 32
Associations of quality of life with service satisfaction in psychotic patients: a meta-analysis.
Petkari, E., & Pietschnig, J. (2015) · PLoS ONE · doi:10.1371/journal.pone.0135267 · Cited by 35
Flückiger, C., Del Re, A. C., Munder, T., Heer, S., & Wampold, B. E. (2014) · Clinical Psychology Review · doi:10.1016/j.cpr.2014.05.001 · Cited by 34
Kieselbach, B., Kimber, M., MacMillan, H. L., & Perneger, T. (2022) · BMJ Open · doi:10.1136/bmjopen-2021-051140 · Cited by 36
Misera, A., Liśkiewicz, P., Łoniewski, I., Skonieczna-Żydecka, K., & Samochowiec, J. (2021) · Pharmaceuticals · doi:10.3390/ph14100952 · Cited by 34
The Effectiveness of Concept Maps on Students’ Achievement in Science: A Meta-Analysis
Anastasiou, D., Wirngo, C. N., & Bagos, P. (2024) · Educational Psychology Review · doi:10.1007/s10648-024-09877-y · Cited by 36
Yamaoka, K., Nemoto, A., & Tango, T. (2019) · Nutrients · doi:10.3390/nu11061373 · Cited by 32
Spindle-dependent memory consolidation in healthy adults: a meta-analysis
Kumral, D., Matzerath, A., Leonhart, R., & Schönauer, M. (2023) · Neuropsychologia · doi:10.1016/j.neuropsychologia.2023.108661 · Cited by 32
Correlations of online social network size with well-being and distress: a meta-analysis
Huang, C. (2021) · Cyberpsychology: Journal of Psychosocial Research on Cyberspace · doi:10.5817/cp2021-2-3 · Cited by 33
Efficacy of mentalization‐based therapy in treating self‐harm: A systematic review and meta‐analysis
Hajek Gross, C., Oehlke, S., Prillinger, K., Goreis, A., Plener, P. L., & Kothgassner, O. D. (2024) · Suicide and Life-Threatening Behavior · doi:10.1111/sltb.13044 · Cited by 31
Remote cognitive‐behavioral therapy for generalized anxiety disorder: A preliminary meta‐analysis
Trenoska Basile, V., Newton‐John, T., & Wootton, B. M. (2022) · Journal of Clinical Psychology · doi:10.1002/jclp.23360 · Cited by 28
Exercise Effects on Bone Mineral Density in Men
Mages, M., Shojaa, M., Kohl, M., von Stengel, S., Becker, C., Gosch, M., Jakob, F., Kerschan-Schindl, K., Kladny, B., Klöckner, N., et al. (2021) · Nutrients · doi:10.3390/nu13124244 · Cited by 28
Generalized synthesis of evidence and the threat of dissemination bias
Sutton, A. J., Abrams, K. R., & Jones, D. R. (2002) · Journal of Clinical Epidemiology · doi:10.1016/s0895-4356(02)00460-2 · Cited by 27
Does Caffeine Increase Fat Metabolism? A Systematic Review and Meta-Analysis
Conger, S. A., Tuthill, L. M., & Millard-Stafford, M. L. (2023) · International Journal of Sport Nutrition and Exercise Metabolism · doi:10.1123/ijsnem.2022-0131 · Cited by 28
Tak, L. M., Meijer, A., Manoharan, A., de Jonge, P., & Rosmalen, J. G. M. (2010) · Psychosomatic Medicine · doi:10.1097/psy.0b013e3181d714e1 · Cited by 23
Risk factors for juvenile cybercrime: A meta-analytic review
Wissink, I. B., Standaert, J. C. A., Stams, G. J. J. M., Asscher, J. J., & Assink, M. (2023) · Aggression and Violent Behavior · doi:10.1016/j.avb.2023.101836 · Cited by 22
Stanyer, E. C., Baniqued, P. D. E., Awais, M., Kouara, L., Davies, A. G., Killan, E. C., & Mushtaq, F. (2022) · Journal of Sleep Research · doi:10.1111/jsr.13385 · Cited by 21
RetractedBayesian multivariate meta-analysis of multiple factors
Lin, L., & Chu, H. (2018) · Research Synthesis Methods · doi:10.1002/jrsm.1293 · Cited by 19
The sleeper effect: Artifact or phenomenon—A brief comment on Bell et al. (2013).
Flückiger, C., Del Re, A. C., & Wampold, B. E. (2015) · Journal of Consulting and Clinical Psychology · doi:10.1037/a0037220 · Cited by 19
Polito, M. D., Papst, R. R., & Farinatti, P. (2021) · Journal of Sports Sciences · doi:10.1080/02640414.2021.1924978 · Cited by 14
Steubl, L., Sachser, C., Baumeister, H., & Domhardt, M. (2019) · Systematic Reviews · doi:10.1186/s13643-019-1190-6 · Cited by 12
Crane, M. F., Hazel, G., Kunzelmann, A., Kho, M., Gucciardi, D. F., Rigotti, T., Kalisch, R., & Karin, E. (2024) · Anxiety, Stress, & Coping · doi:10.1080/10615806.2023.2228707 · Cited by 10
Stocchero, B. A., Rothmann, L. M., Portolan, E. T., Lopes, T. G., Ferraz-Rodrigues, C., Garcia, M. G., de Magalhães Narvaez, J. C., Grassi-Oliveira, R., & Viola, T. W. (2024) · Child Abuse & Neglect · doi:10.1016/j.chiabu.2024.107085 · Cited by 9
Moore, M. N., Climie, R. E., Otahal, P., & Schultz, M. G. (2021) · Journal of Hypertension · doi:10.1097/hjh.0000000000002962 · Cited by 23
Zhang, X., Chang, Z., Zhao, S., & Ning, Z. (2025) · Frontiers in Psychology · doi:10.3389/fpsyg.2025.1662868 · Cited by 5
Borthwick, C., Norton, A., Trenoska Basile, V., & Wootton, B. (2026) · British Journal of Clinical Psychology · doi:10.1111/bjc.70089
Lo, J. C., Groeger, J. A., Cheng, G. H., Dijk, D.-J., & Chee, M. W. L. (2016) · Sleep Medicine · doi:10.1016/j.sleep.2015.08.021 · Cited by 438
Metadata
- Authors
- Duval, S., & Tweedie, R.
- Affiliations
- Colorado State University (US)
- University of Minnesota (US)
- ORCID
- Duval, Sue
- Venue
- Journal of the American Statistical Association
- Publisher
- Informa UK Limited
- Published
- 2000-03
- Volume
- 95
- Issue
- 449
- Pages
- 89-98
- ISSN
- 0162-1459 (ISSN-L)
- 1537-274X
- OpenAlex
- W2043565183
- Cited by
- 2,955
- References
- 40
- Citations by year
- 2026: 106 · 2025: 167 · 2024: 189 · 2023: 171 · 2022: 233 · 2021: 275 · 2020: 259 · 2019: 179 · 2018: 184 · 2017: 160 · 2016: 142 · 2015: 149 · 2014: 128 · 2013: 91 · 2012: 101
- FWCI
- 21.09
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Journal of the American Statistical Association (publishedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
- Social Sciences › Library and Information Sciences › scientometrics and bibliometrics research
- Keywords
- publication bias, trim and fill, missing studies, funnel plot, nonparametric estimators, selection bias, meta-analysis, effect size estimation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
