A Nonparametric “Trim and Fill” Method of Accounting for Publication Bias in Meta-Analysis
Abstract
Meta-analysis collects and synthesizes results from individual studies to estimate an overall effect size. If published studies are chosen, say through a literature review, then an inherent selection bias may arise, because, for example, studies may tend to be published more readily if they are statistically significant, or deemed to be more “interesting” in terms of the impact of their outcomes. We develop a simple rank-based data augmentation technique, formalizing the use of funnel plots, to estimate and adjust for the numbers and outcomes of missing studies. Several nonparametric estimators are proposed for the number of missing studies, and their properties are developed analytically and through simulations. We apply the method to simulated and epidemiological datasets and show that it is both effective and consistent with other criteria in the literature.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Publication bias was investigated by applying the trim and fill method.
“Publication bias was investigated by applying the trim and fill method, where effect sizes are recalculated to adjust for supposed publication biases (Duval & Tweedie, 2000).”
From Taking stock: A meta-analysis of the effects of foreign language enjoyment (2022) · cites this paper as Duval & Tweedie, 2000
The trim-and-fill procedure determined that nine studies should be added to create symmetry.
“The trim‐and‐fill procedure determined that nine studies should be added to create symmetry.”
From Problem-oriented policing for reducing crime and disorder: An updated systematic review and meta-analysis (2020) · cites this paper as Duval and Tweedie (2000)
The meta-analyses used the trim-and-fill procedure to explore whether publication bias might be affecting the results and to estimate how the reported effects would change if the bias were to be removed.
“Like many systematic reviews, our meta‐analyses used the trim‐and‐fill procedure to explore whether publication bias might be affecting the results and to estimate how the reported effects would change if the bias were to be removed (Duval & Tweedie, 2000; Duval, 2005).”
From Hot spots policing of small geographic areas effects on crime (2019) · cites this paper as Duval & Tweedie, 2000
When publication bias was encountered, a trim and fill adjusted analysis was conducted.
“When we encountered publication bias, we conducted a trim and fill adjusted analysis 25 to remove the most extreme small studies from the positive side of the funnel plot, and recalculated the effect size iteratively, until the funnel plot was symmetrical around the (new) effect size.”
From Diabetes mellitus in people with schizophrenia, bipolar disorder and major depressive disorder: a systematic review and large scale meta‐analysis (2016) · cites this paper as [25]
Variations in the risk for glucose abnormalities are evident in the literature, with the highest risk being associated with clozapine, olanzapine and quetiapine in carefully designed studies.
“Variations in the risk for glucose abnormalities are evident in the literature, with the highest risk being associated with clozapine, olanzapine and quetiapine in carefully designed studies 25,34,35 .”
From Diabetes mellitus in people with schizophrenia, bipolar disorder and major depressive disorder: a systematic review and large scale meta‐analysis (2016) · cites this paper as [25]
Cites
Bias in meta-analysis detected by a simple, graphical test
Egger, M., Smith, G. D., Schneider, M., & Minder, C. (1997) · BMJ · doi:10.1136/bmj.315.7109.629 · Cited by 56,722 · see citing works
Meta-analysis in clinical trials
DerSimonian, R., & Laird, N. (1986) · Controlled Clinical Trials · doi:10.1016/0197-2456(86)90046-2 · Cited by 39,552 · see citing works
Operating characteristics of a rank correlation test for publication bias
Begg, C. B., & Mazumdar, M. (1994) · Biometrics · doi:10.2307/2533446 · Cited by 16,955 · see citing works
Duval, S., & Tweedie, R. (2000) · Biometrics · doi:10.1111/j.0006-341x.2000.00455.x · Cited by 14,099 · see citing works
Self-citationWasserman, S. S., Hedges, L. V., & Olkin, I. (1988) · Journal of Educational Statistics · doi:10.2307/1164953 · Cited by 10,347 · see citing works
Casella, G., & Berger, R. L. (1993) · Chapman & Hall/CRC eBooks · doi:10.1201/9781003456285 · Cited by 5,835 · see citing works
The Handbook of Research Synthesis.
Howel, D., Cooper, H., & Hedges, L. V. (1995) · Biometrics · doi:10.2307/2533295 · Cited by 3,657 · see citing works
Publication bias in clinical research
Easterbrook, P. J., Gopalan, R., Berlin, J. A., & Matthews, D. R. (1991) · The Lancet · doi:10.1016/0140-6736(91)90201-y · Cited by 3,109 · cited 1× · see citing works
“The negative binomial forms here are reminiscent of those used by Eberly and Casella (1999) and Givens et al.”
MethodSumming up: The Science of Reviewing Research
Wilkinson, I., Light, R. J., & Pillemer, D. B. (1987) · Journal of Educational Statistics · doi:10.2307/1164691 · Cited by 1,486 · see citing works
Light, R. J., & Pillemer, D. B. (1984) · Harvard University Press eBooks · doi:10.4159/9780674040243 · Cited by 1,422 · see citing works
Monastyrsky, M. (1999) · Birkhäuser Boston eBooks · doi:10.1007/978-0-8176-4779-7_2 · Cited by 1,081 · see citing works
Nonparametric and Distribution-Free Methods for the Social Sciences.
Bassler, J. F., Marascuilo, L. A., & McSweeney, M. (1978) · Journal of the American Statistical Association · doi:10.2307/2286625 · Cited by 997 · see citing works
The Handbook of Research Synthesis
Ziegel, E. R., Cooper, H., & Hedges, L. V. (1995) · Technometrics · doi:10.2307/1269651 · Cited by 936 · see citing works
Dickersin, K., Yuan, M.-Q., & Meinert, C. (1992) · Journal of the American Medical Association (JAMA) · doi:10.1001/jama.1992.03480030052036 · Cited by 900 · see citing works
A note on graphical presentation of estimated odds ratios from several clinical trials
Galbraith, R. F. (1988) · Statistics in Medicine · doi:10.1002/sim.4780070807 · Cited by 663 · see citing works
Selection Models and the File Drawer Problem
Iyengar, S., & Greenhouse, J. B. (1988) · Statistical Science · doi:10.1214/ss/1177013012 · Cited by 388 · see citing works
Modeling Publication Selection Effects in Meta-Analysis
Hedges, L. V. (1992) · Statistical Science · doi:10.1214/ss/1177011364 · Cited by 359 · see citing works
Combining Information: Statistical Issues and Opportunities for Research
Draper, D. L., Gaver, D. P., Goel, P., Greenhouse, J. B., Hedges, L. V., Morris, C., Tucker, J., & Waternaux, C. (1993) · Cited by 279 · see citing works
Angus, J. E., Casella, G., & Berger, R. L. (1991) · Technometrics · doi:10.2307/1269442 · Cited by 220 · see citing works
Meta-Analysis in Medicine and Health Policy
Stangl, D. K. (2000) · CRC Press eBooks · doi:10.1201/9780203909935 · Cited by 213 · see citing works
An Approach for Assessing Publication Bias Prior to Performing a Meta-Analysis
Dear, K., & Begg, C. (1992) · Statistical Science · doi:10.1214/ss/1177011363 · Cited by 202 · see citing works
Dickersin, K. (1992) · JAMA · doi:10.1001/jama.267.3.374 · Cited by 193 · see citing works
Givens, G. H., Smith, D. D., & Tweedie, R. L. (1997) · Statistical Science · doi:10.1214/ss/1030037958 · Cited by 127 · see citing works
Self-citationPublication Bias and Research on Passive Smoking
Misakian, A. L., & Bero, L. A. (1998) · JAMA · doi:10.1001/jama.280.3.250 · Cited by 126 · see citing works
Models for estimating the number of unpublished studies.
Gleser, L., & Olkin, I. (1996) · Statistics in Medicine · doi:10.1002/(sici)1097-0258(19961215)15:23 · Cited by 116 · see citing works
Pastor, M., & Mollenkopf, J. H. (2016) · Cornell University Press eBooks · doi:10.7591/cornell/9781501702662.003.0010 · Cited by 94 · see citing works
Environmental Tobacco Smoke and Mortality
Strachan, D. P. (1993) · Journal of Epidemiology & Community Health · doi:10.1136/jech.47.2.163-a · Cited by 77 · see citing works
Lee, W. (1997) · Statistics in Medicine · doi:10.1002/(sici)1097-0258(19970415)16:7 · Cited by 74 · see citing works
Some Progress and Problems in Meta-Analysis of Clinical Trials
Mosteller, F. C., & Chalmers, T. C. (1992) · Statistical Science · doi:10.1214/ss/1177011362 · Cited by 66 · see citing works
Bayesian meta-analysis, with application to studies of ETS and lung cancer
Tweedie, R. L., Scott, D. J., Biggerstaff, B. J., & Mengersen, K. L. (1996) · Lung Cancer · doi:10.1016/s0169-5002(96)90222-6 · Cited by 51 · see citing works
Self-citationTHE IMPACT OF METHOD CHOICE ON META‐ANALYSIS
Mengersen, K. L., Tweedie, R. L., & Biggerstaff, B. J. (1995) · Australian Journal of Statistics · doi:10.1111/j.1467-842x.1995.tb00869.x · Cited by 42 · see citing works
Self-citationEnvironmental Tobacco Smoke and Mortality
Law, M. (1993) · British Journal of Cancer · doi:10.1038/bjc.1993.77 · Cited by 25 · see citing works
Olkin, I. (1992) · Statistical Science · doi:10.1214/ss/1177011361 · Cited by 19 · see citing works
Estimating a Summarized Odds Ratio whilst Eliminating Publication Bias in Meta-analysis *
Sugita, M., Kanamor, M., Izuno, T., & Miyakawa, M. (1992) · Japanese Journal of Clinical Oncology · doi:10.1093/oxfordjournals.jjco.a039577 · Cited by 17 · see citing works
JAPANESE JOURNAL OF CLINICAL ONCOLOGY
(2011) · JNCI Journal of the National Cancer Institute · doi:10.1093/jnci/djr110 · Cited by 12 · see citing works
Sugita, M., Yamaguchi, N., Izuno, T., Kanamori, M., & Kasuga, H. (1994) · PubMed · Cited by 8 · see citing works
Begg, C., DuMouchel, W., Harris, J., Dobson, A. P., Dear, K. B. G., Givens, G. H., Smith, D., & Tweedle, R. (1997) · Statistical Science · Cited by 8 · see citing works
Bayesian Estimation of the Number of Unseen Studies in a Meta-Analysis
Eberly, L. E., & Casella, G. (1999) · Journal of Official Statistics · Cited by 7 · see citing works
Gibson, T. (1994) · Lara D. Veeken · doi:10.1093/rheumatology/33.6.589 · Cited by 6 · see citing works
Estimating the Number of Unseen Studies in a Meta-Analysis
Eberly, L. E., & Casella, G. (1997) · Cited by 3 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Conducting Meta-Analyses in <i>R</i> with the <b>metafor</b> Package
Viechtbauer, W. (2010) · Journal of Statistical Software · doi:10.18637/jss.v036.i03 · Cited by 20,464 · see citing works
Hao (2015) · doi:10.1002/14651858 · Cited by 11,321 · see citing works
A meta-analytic test of intergroup contact theory.
Pettigrew, T. F., & Tropp, L. R. (2006) · Journal of personality and social psychology · doi:10.1037/0022-3514.90.5.751 · Cited by 8,964 · see citing works
Social Relationships and Mortality Risk: A Meta-analytic Review
Holt-Lunstad, J., Smith, T. B., & Layton, J. B. (2010) · PLoS Medicine · doi:10.1371/journal.pmed.1000316 · Cited by 7,272 · see citing works
“Second, we employed the ‘‘trim and fill’’ methodology described by Duval and Tweedie [178,179] to estimate the number of studies missing due to publication bias, but this analysis failed to reveal any studies that would need to be created on the opposite side of the distribution, meaning that adjustment to the omnibus…”
MethodHow to perform a meta-analysis with R: a practical tutorial
Balduzzi, S., Rücker, G., & Schwarzer, G. (2019) · Evidence Based Mental Health · doi:10.1136/ebmental-2019-300117 · Cited by 5,827 · see citing works
Perceived discrimination and health: A meta-analytic review.
Pascoe, E. A., & Smart Richman, L. (2009) · Psychological Bulletin · doi:10.1037/a0016059 · Cited by 4,427 · see citing works
Roberts, B. W., Walton, K. E., & Viechtbauer, W. (2006) · Psychological Bulletin · doi:10.1037/0033-2909.132.1.1 · Cited by 3,318 · see citing works
Lorenz, M. W., Markus, H. S., Bots, M. L., Rosvall, M., & Sitzer, M. (2007) · Circulation · doi:10.1161/circulationaha.106.628875 · Cited by 2,977 · see citing works
Racism as a Determinant of Health: A Systematic Review and Meta-Analysis
Paradies, Y., Ben, J., Denson, N., Elias, A., Priest, N., Pieterse, A., Gupta, A., Kelaher, M., & Gee, G. (2015) · PLOS ONE · doi:10.1371/journal.pone.0138511 · Cited by 2,610 · see citing works
“Where we detected publication bias, we used the trim and fill method to estimate and adjust for missing (un-reported) studies to estimate what the effect size would have been in the absence of bias [66, 67].”
MethodDo Psychosocial and Study Skill Factors Predict College Outcomes? A Meta-Analysis.
Robbins, S. B., Lauver, K., Le, H., Davis, D., Langley, R., & Carlstrom, A. (2004) · Psychological Bulletin · doi:10.1037/0033-2909.130.2.261 · Cited by 2,479 · see citing works
Varese, F., Smeets, F., Drukker, M., Lieverse, R., Lataster, T., Viechtbauer, W., Read, J., van Os, J., & Bentall, R. P. (2012) · Schizophrenia Bulletin · doi:10.1093/schbul/sbs050 · Cited by 2,436 · see citing works
Effective techniques in healthy eating and physical activity interventions: A meta-regression.
Michie, S., Abraham, C., Whittington, C., McAteer, J., & Gupta, S. (2009) · Health Psychology · doi:10.1037/a0016136 · Cited by 2,229 · see citing works
Perceived organizational support: a meta-analytic evaluation of organizational support theory.
Kurtessis, J. N., Eisenberger, R., Ford, M. T., Buffardi, L. C., Stewart, K. A., & Adis, C. S. (2017) · Journal of Management · doi:10.1177/0149206315575554 · Cited by 2,354 · see citing works
Comparison of two methods to detect publication bias in meta-analysis.
Peters, J. L. (2006) · JAMA · doi:10.1001/jama.295.6.676 · Cited by 2,278 · see citing works
A modified test for small‐study effects in meta‐analyses of controlled trials with binary endpoints
Harbord, R. M., Egger, M., & Sterne, J. A. C. (2006) · Statistics in Medicine · doi:10.1002/sim.2380 · Cited by 2,267 · see citing works
Peters, J. L., Sutton, A. J., Jones, D. R., Abrams, K. R., & Rushton, L. (2008) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2007.11.010 · Cited by 2,053 · see citing works
Anderson, C. A., Shibuya, A., Ihori, N., Swing, E. L., Bushman, B. J., Sakamoto, A., Rothstein, H. R., & Saleem, M. (2010) · Psychological Bulletin · doi:10.1037/a0018251 · Cited by 2,048 · see citing works
A Global Perspective on Child Sexual Abuse: Meta-Analysis of Prevalence Around the World
Stoltenborgh, M., van IJzendoorn, M. H., Euser, E. M., & Bakermans-Kranenburg, M. J. (2011) · Child Maltreatment · doi:10.1177/1077559511403920 · Cited by 2,010 · see citing works
Effect of Aerobic Exercise on Blood Pressure: A Meta-Analysis of Randomized, Controlled Trials
Whelton, S. P., Chin, A., Xin, X., & He, J. (2002) · Annals of Internal Medicine · doi:10.7326/0003-4819-136-7-200204020-00006 · Cited by 1,970 · see citing works
Snyder, H. R. (2013) · Psychological Bulletin · doi:10.1037/a0028727 · Cited by 1,667 · see citing works
Quantifying publication bias in meta-analysis
Lin, L., & Chu, H. (2018) · Biometrics · doi:10.1111/biom.12817 · Cited by 1,674 · see citing works
Effects of occupational stress management intervention programs: A meta-analysis.
Richardson, K. M., & Rothstein, H. R. (2008) · Journal of Occupational Health Psychology · doi:10.1037/1076-8998.13.1.69 · Cited by 1,374 · see citing works
Does culture influence what and how we think? Effects of priming individualism and collectivism.
Oyserman, D., & Lee, S. W. S. (2008) · Psychological Bulletin · doi:10.1037/0033-2909.134.2.311 · Cited by 1,390 · see citing works
Andrews, G., Cuijpers, P., Craske, M. G., McEvoy, P., & Titov, N. (2010) · PLoS ONE · doi:10.1371/journal.pone.0013196 · Cited by 1,311 · see citing works
Mind-sets matter: A meta-analytic review of implicit theories and self-regulation.
Burnette, J. L., O'Boyle, E. H., VanEpps, E. M., Pollack, J. M., & Finkel, E. J. (2013) · Psychological Bulletin · doi:10.1037/a0029531 · Cited by 1,367 · see citing works
Fearon, R. P., Bakermans-Kranenburg, M. J., Van IJzendoorn, M. H., Lapsley, A.-M., & Roisman, G. I. (2010) · Child Development · doi:10.1111/j.1467-8624.2009.01405.x · Cited by 1,326 · see citing works
Fitting three-level meta-analytic models in R: A step-by-step tutorial
Assink, M., & Wibbelink, C. J. M. (2016) · The Quantitative Methods for Psychology · doi:10.20982/tqmp.12.3.p154 · Cited by 1,330 · see citing works
Kyu, H. H., Bachman, V. F., Alexander, L. T., Mumford, J. E., Afshin, A., Estep, K., Veerman, J. L., Delwiche, K., Iannarone, M. L., Moyer, M. L., et al. (2016) · BMJ · doi:10.1136/bmj.i3857 · Cited by 1,272 · see citing works
Social Psychiatry and Psychiatric Epidemiology
(2013) · Social Psychiatry and Psychiatric Epidemiology · doi:10.1007/127.1433-9285 · Cited by 1,331 · see citing works
To read or not to read: a meta-analysis of print exposure from infancy to early adulthood
Mol, S. E., & Bus, A. G. (2011) · Psychological Bulletin · doi:10.1037/a0021890 · Cited by 1,299 · see citing works
Rosenberg, M. S. (2005) · Evolution · doi:10.1111/j.0014-3820.2005.tb01004.x · Cited by 1,285 · see citing works
Vancampfort, D., Stubbs, B., Mitchell, A. J., De Hert, M., Wampers, M., Ward, P. B., Rosenbaum, S., & Correll, C. U. (2015) · World Psychiatry · doi:10.1002/wps.20252 · Cited by 1,271 · see citing works
Global ecological impacts of invasive species in aquatic ecosystems
Gallardo, B., Clavero, M., Sánchez, M. I., & Vilà, M. (2016) · Global Change Biology · doi:10.1111/gcb.13004 · Cited by 1,245 · see citing works
Mezulis, A. H., Abramson, L. Y., Hyde, J. S., & Hankin, B. L. (2004) · Psychological Bulletin · doi:10.1037/0033-2909.130.5.711 · Cited by 1,281 · see citing works
Ownby, R. L., Crocco, E., Acevedo, A., John, V., & Loewenstein, D. (2006) · Archives of General Psychiatry · doi:10.1001/archpsyc.63.5.530 · Cited by 1,493 · see citing works
Digital Games, Design, and Learning: A Systematic Review and Meta-Analysis
Clark, D. B., Tanner-Smith, E. E., & Killingsworth, S. S. (2016) · Review of Educational Research · doi:10.3102/0034654315582065 · Cited by 1,274 · see citing works
Chu, C., Buchman-Schmitt, J. M., Stanley, I. H., Hom, M. A., Tucker, R. P., Hagan, C. R., Rogers, M. L., Podlogar, M. C., Chiurliza, B., Ringer, F. B., et al. (2017) · Psychological Bulletin · doi:10.1037/bul0000123 · Cited by 1,231 · see citing works
Culturally adapted mental health intervention: a meta-analytic review
Griner, D., & Smith, T. B. (2006) · Psychotherapy: Theory, Research, Practice, Training · doi:10.1037/0033-3204.43.4.531 · Cited by 1,236 · see citing works
Men and things, women and people: A meta-analysis of sex differences in interests.
Su, R., Rounds, J., & Armstrong, P. I. (2009) · Psychological Bulletin · doi:10.1037/a0017364 · Cited by 1,206 · see citing works
Field, A. P., & Gillett, R. (2010) · British Journal of Mathematical and Statistical Psychology · doi:10.1348/000711010x502733 · Cited by 1,162 · see citing works
Smith, T. B., McCullough, M. E., & Poll, J. (2003) · Psychological Bulletin · doi:10.1037/0033-2909.129.4.614 · Cited by 1,150 · see citing works
Kyrgiou, M., Koliopoulos, G., Martin-Hirsch, P., Arbyn, M., Prendiville, W., & Paraskevaidis, E. (2006) · The Lancet · doi:10.1016/s0140-6736(06)68181-6 · Cited by 1,120 · see citing works
Patall, E. A., Cooper, H., & Robinson, J. C. (2008) · Psychological Bulletin · doi:10.1037/0033-2909.134.2.270 · Cited by 1,101 · see citing works
Andrews, G., Basu, A., Cuijpers, P., Craske, M. G., McEvoy, P., English, C. L., & Newby, J. M. (2018) · Journal of Anxiety Disorders · doi:10.1016/j.janxdis.2018.01.001 · Cited by 922 · see citing works
Elder abuse prevalence in community settings: a systematic review and meta-analysis
Yon, Y., Mikton, C. R., Gassoumis, Z. D., & Wilber, K. H. (2017) · The Lancet Global Health · doi:10.1016/s2214-109x(17)30006-2 · Cited by 1,074 · see citing works
Smetana, G. W., Lawrence, V. A., & Cornell, J. E. (2006) · Annals of Internal Medicine · doi:10.7326/0003-4819-144-8-200604180-00009 · Cited by 1,028 · see citing works
Methodological issues and advances in biological meta-analysis
Nakagawa, S., & Santos, E. S. A. (2012) · Evolutionary Ecology · doi:10.1007/s10682-012-9555-5 · Cited by 1,032 · see citing works
Outdoor Particulate Matter Exposure and Lung Cancer: A Systematic Review and Meta-Analysis
Hamra, G. B., Guha, N., Cohen, A., Laden, F., Raaschou‐Nielsen, O., Samet, J. M., Vineis, P., Forastiere, F., Saldiva, P. H. N., Yorifuji, T., et al. (2014) · Environmental Health Perspectives · doi:10.1289/ehp/1408092 · Cited by 1,028 · see citing works
Atkinson, R. W., Kang, S., Anderson, H. R., Mills, I., & Walton, H. (2014) · Thorax · doi:10.1136/thoraxjnl-2013-204492 · Cited by 1,017 · see citing works
Vazquez, G., Duval, S., Jacobs, D. R., & Silventoinen, K. (2007) · Epidemiologic Reviews · doi:10.1093/epirev/mxm008 · Cited by 976 · see citing works
Self-citationPostmenopausal Hormone Replacement Therapy: Scientific Review
Nelson, H. D., Humphrey, L. L., Nygren, P., Teutsch, S. M., & Allan, J. D. (2002) · JAMA · doi:10.1001/jama.288.7.872 · Cited by 990 · see citing works
Fischer, P., Krueger, J. I., Greitemeyer, T., Vogrincic, C., Kastenmüller, A., Frey, D., Heene, M., Wicher, M., & Kainbacher, M. (2011) · Psychological Bulletin · doi:10.1037/a0023304 · Cited by 983 · see citing works
Added value of dialogic parent–child book readings: A meta-analysis
Mol, S. E., Bus, A. G., de Jong, M. T., & Smeets, D. J. H. (2008) · Early Education and Development · doi:10.1080/10409280701838603 · Cited by 973 · see citing works
Stanley, T. D., & Doucouliagos, H. (2014) · Research Synthesis Methods · doi:10.1002/jrsm.1095 · Cited by 916 · see citing works
Cao, Z. J., & Lumineau, F. (2014) · Journal of Operations Management · doi:10.1016/j.jom.2014.09.009 · Cited by 961 · see citing works
Rueger, S. Y., Malecki, C. K., Pyun, Y., Aycock, C., & Coyle, S. (2016) · Psychological Bulletin · doi:10.1037/bul0000058 · Cited by 946 · see citing works
Effects of fibrates on cardiovascular outcomes: a systematic review and meta-analysis
Jun, M., Foote, C., Lv, J., Neal, B., Patel, A., Nicholls, S. J., Grobbee, D. E., Cass, A., Chalmers, J., & Perkovic, V. (2010) · The Lancet · doi:10.1016/s0140-6736(10)60656-3 · Cited by 933 · see citing works
Niche breadth predicts geographical range size: a general ecological pattern
Slatyer, R., Hirst, M. J., & Sexton, J. P. (2013) · Ecology Letters · doi:10.1111/ele.12140 · Cited by 916 · see citing works
Meta-analyses of factors motivating climate change adaptation behaviour
van Valkengoed, A. M., & Steg, L. (2019) · Nature Climate Change · doi:10.1038/s41558-018-0371-y · Cited by 940 · see citing works
Make, Buy, or Ally: A Transaction Cost Theory Meta-Analysis
Geyskens, I., Steenkamp, J. E. M., & Kumar, N. (2006) · Academy of Management Journal · doi:10.5465/amj.2006.21794670 · Cited by 925 · see citing works
Adam, E. K., Quinn, M. E., Tavernier, R., McQuillan, M. T., Dahlke, K. A., & Gilbert, K. E. (2017) · Psychoneuroendocrinology · doi:10.1016/j.psyneuen.2017.05.018 · Cited by 905 · see citing works
Kelley, J., Kraft-Todd, G. T., Schapira, L., Kossowsky, J., & Riess, H. (2014) · PLoS ONE · doi:10.1371/journal.pone.0094207 · Cited by 836 · see citing works
How Effective Are Mentoring Programs for Youth? A Systematic Assessment of the Evidence
DuBois, D. L., Portillo, N., Rhodes, J. E., Silverthorn, N., & Valentine, J. C. (2011) · Psychological Science in the Public Interest · doi:10.1177/1529100611414806 · Cited by 889 · see citing works
The relationship between materialism and personal well-being: A meta-analysis
Dittmar, H., Bond, R., Hurst, M., & Kasser, T. (2014) · Journal of Personality and Social Psychology · doi:10.1037/a0037409 · Cited by 890 · see citing works
Racial/ethnic discrimination and well-being during adolescence: A meta-analytic review.
Benner, A. D., Wang, Y., Shen, Y., Boyle, A. E., Polk, R., & Cheng, Y. (2018) · American Psychologist · doi:10.1037/amp0000204 · Cited by 897 · see citing works
Meng, X., & D’Arcy, C. (2012) · PLoS ONE · doi:10.1371/journal.pone.0038268 · Cited by 893 · see citing works
Contingency management for treatment of substance use disorders: a meta‐analysis
Prendergast, M., Podus, D., Finney, J., Greenwell, L., & Roll, J. (2006) · Addiction · doi:10.1111/j.1360-0443.2006.01581.x · Cited by 877 · see citing works
Leucocyte telomere length and risk of cardiovascular disease: systematic review and meta-analysis
Haycock, P. C., Heydon, E. E., Kaptoge, S., Butterworth, A. S., Thompson, A., & Willeit, P. (2014) · BMJ · doi:10.1136/bmj.g4227 · Cited by 856 · see citing works
Hadfield, J. D., & Nakagawa, S. (2010) · Journal of Evolutionary Biology · doi:10.1111/j.1420-9101.2009.01915.x · Cited by 813 · see citing works
Correll, C. U., Galling, B., Pawar, A., Krivko, A., Bonetto, C., Ruggeri, M., Craig, T. J., Nordentoft, M., Srihari, V. H., Guloksuz, S., et al. (2018) · JAMA Psychiatry · doi:10.1001/jamapsychiatry.2018.0623 · Cited by 826 · see citing works
Adjusting for publication bias in the presence of heterogeneity
Terrin, N., Schmid, C. H., Lau, J., & Olkin, I. (2003) · Statistics in Medicine · doi:10.1002/sim.1461 · Cited by 809 · see citing works
Recognition of Emotions in Autism: A Formal Meta-Analysis
Uljarevic, M., & Hamilton, A. (2013) · Journal of Autism and Developmental Disorders · doi:10.1007/s10803-012-1695-5 · Cited by 824 · see citing works
Sexual assault victimization and psychopathology: A review and meta-analysis
Dworkin, E. R., Menon, S. V., Bystrynski, J., & Allen, N. E. (2017) · Clinical Psychology Review · doi:10.1016/j.cpr.2017.06.002 · Cited by 825 · see citing works
Shi, L., & Lin, L. (2019) · Medicine · doi:10.1097/md.0000000000015987 · Cited by 818 · see citing works
Dent, A. L., & Koenka, A. C. (2015) · Educational Psychology Review · doi:10.1007/s10648-015-9320-8 · Cited by 888 · see citing works
Social influence approaches to encourage resource conservation: a meta-analysis
Abrahamse, W., & Steg, L. (2013) · Global Environmental Change · doi:10.1016/j.gloenvcha.2013.07.029 · Cited by 802 · see citing works
<i>p</i> -Curve and Effect Size: Correcting for Publication Bias Using Only Significant Results
Simonsohn, U., Nelson, L. D., & Simmons, J. P. (2014) · Perspectives on Psychological Science · doi:10.1177/1745691614553988 · Cited by 717 · see citing works
The neglect of child neglect: A meta-analytic review of the prevalence of neglect
Stoltenborgh, M., Bakermans-Kranenburg, M. J., & van IJzendoorn, M. H. (2013) · Social Psychiatry and Psychiatric Epidemiology · doi:10.1007/s00127-012-0549-y · Cited by 736 · see citing works
Yap, M. B. H., Pilkington, P. D., Ryan, S., & Jorm, A. F. (2013) · Journal of Affective Disorders · doi:10.1016/j.jad.2013.11.007 · Cited by 783 · see citing works
Long-term opioid management for chronic noncancer pain
Noble, M., Treadwell, J. R., Tregear, S., Coates, V. H., Wiffen, P. J., Akafomo, C., Schoelles, K. M., & Chou, R. (2010) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd006605.pub2 · Cited by 728 · see citing works
Peters, J. L., Sutton, A. J., Jones, D. R., Abrams, K. R., & Rushton, L. (2007) · Statistics in Medicine · doi:10.1002/sim.2889 · Cited by 716 · see citing works
Childhood maltreatment and obesity: systematic review and meta-analysis
Danese, A., & Tan, M. (2014) · Molecular Psychiatry · doi:10.1038/mp.2013.54 · Cited by 709 · see citing works
O’Dea, R. E., Lagisz, M., Jennions, M. D., Koricheva, J., Noble, D. W. A., Parker, T. H., Gurevitch, J., Page, M. J., Stewart, G. B., Moher, D., et al. (2021) · Biological reviews/Biological reviews of the Cambridge Philosophical Society · doi:10.1111/brv.12721 · Cited by 771 · see citing works
Astill, R. G., Van der Heijden, K. B., Van IJzendoorn, M. H., & Van Someren, E. J. W. (2012) · Psychological Bulletin · doi:10.1037/a0028204 · Cited by 707 · see citing works
Does Homework Improve Academic Achievement? A Synthesis of Research, 1987–2003
Cooper, H., Robinson, J. C., & Patall, E. A. (2006) · Review of Educational Research · doi:10.3102/00346543076001001 · Cited by 1,230 · see citing works
Meta-Analysis of the Reasoned Action Approach (RAA) to Understanding Health Behaviors
McEachan, R., Taylor, N. J., Harrison, R., Lawton, R., Gardner, P. H., & Conner, M. T. (2016) · Annals of Behavioral Medicine · doi:10.1007/s12160-016-9798-4 · Cited by 706 · see citing works
Food cue reactivity and craving predict eating and weight gain: A meta-analytic review
Boswell, R. G., & Kober, H. (2016) · Obesity Reviews · doi:10.1111/obr.12354 · Cited by 703 · see citing works
Molendijk, M. L., Spinhoven, P., Polak, M., Bus, B. A. A., Penninx, B. W. J. H., & Elzinga, B. M. (2014) · Molecular Psychiatry · doi:10.1038/mp.2013.105 · Cited by 702 · see citing works
The Effects of Hot Spots Policing on Crime: An Updated Systematic Review and Meta-Analysis
Braga, A. A., Papachristos, A. V., & Hureau, D. M. (2012) · Justice Quarterly · doi:10.1080/07418825.2012.673632 · Cited by 698 · see citing works
Meta‐analysis of human connection to nature and proenvironmental behavior
Whitburn, J., Linklater, W. L., & Abrahamse, W. (2019) · Conservation Biology · doi:10.1111/cobi.13381 · Cited by 700 · see citing works
Gardening is beneficial for health: A meta-analysis
Soga, M., Gaston, K. J., & Yamaura, Y. (2017) · Preventive Medicine Reports · doi:10.1016/j.pmedr.2016.11.007 · Cited by 688 · see citing works
“If publication biaswas indicated by Egger's test, we performed a trim and fill analysis (with the R0 estimator) (Duval and Tweedie, 2000).”
MethodInterpretation of tests of heterogeneity and bias in meta-analysis
Ioannidis, J. P. A. (2008) · Journal of Evaluation in Clinical Practice · doi:10.1111/j.1365-2753.2008.00986.x · Cited by 696 · see citing works
Personality and prosocial behavior: A theoretical framework and meta-analysis.
Thielmann, I., Spadaro, G., & Balliet, D. P. (2019) · Psychological Bulletin · doi:10.1037/bul0000217 · Cited by 689 · see citing works
Risk of gastrointestinal haemorrhage with long term use of aspirin: meta-analysis
Derry, S. (2000) · BMJ · doi:10.1136/bmj.321.7270.1183 · Cited by 675 · see citing works
Lench, H. C., Flores, S. A., & Bench, S. W. (2011) · Psychological Bulletin · doi:10.1037/a0024244 · Cited by 675 · see citing works
Macnamara, B. N., Hambrick, D. Z., & Oswald, F. L. (2014) · Psychological Science · doi:10.1177/0956797614535810 · Cited by 656 · see citing works
Cabral, R. R., & Smith, T. B. (2011) · Journal of Counseling Psychology · doi:10.1037/a0025266 · Cited by 651 · see citing works
Vancampfort, D., Correll, C. U., Galling, B., Probst, M., De Hert, M., Ward, P. B., Rosenbaum, S., Gaughran, F., Lally, J., & Stubbs, B. (2016) · World Psychiatry · doi:10.1002/wps.20309 · Cited by 659 · see citing works
The Effects of Vocabulary Intervention on Young Children’s Word Learning
Marulis, L. M., & Neuman, S. B. (2010) · Review of Educational Research · doi:10.3102/0034654310377087 · Cited by 668 · see citing works
Polanin, J. R., Espelage, D. L., & Pigott, T. (2012) · School Psychology Review · doi:10.1080/02796015.2012.12087375 · Cited by 659 · see citing works
- Authors
- Duval, S., & Tweedie, R.
- Affiliations
- Colorado State University (US)
- University of Minnesota (US)
- ORCID
- Duval, Sue
- Venue
- Journal of the American Statistical Association
- Publisher
- Informa UK Limited
- Published
- 2000-03
- Volume
- 95
- Issue
- 449
- Pages
- 89-98
- ISSN
- 0162-1459 (ISSN-L)
- 1537-274X
- OpenAlex
- W2043565183
- Cited by
- 2,955
- References
- 40
- Citations by year
- 2026: 106 · 2025: 167 · 2024: 189 · 2023: 171 · 2022: 233 · 2021: 275 · 2020: 259 · 2019: 179 · 2018: 184 · 2017: 160 · 2016: 142 · 2015: 149 · 2014: 128 · 2013: 91 · 2012: 101
- FWCI
- 21.09
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Journal of the American Statistical Association (publishedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › scientometrics and bibliometrics research
- Keywords
- publication bias, trim and fill, missing studies, funnel plot, nonparametric estimators, selection bias, meta-analysis, effect size estimation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
