Publication Bias in Research Synthesis: Sensitivity Analysis Using A Priori Weight Functions.

“…are surveyed about unpublished work (e.g., Chan, Sacks, & Chalmers, 1982; Coursol & Wagner, 1986; Dickersin, Chan, Chalmers, Sacks, & Smith, 1987; Easterbrook et al., 1991; Misakian & Bero, 1998; Rotton, Foos, Vanmeek, & Levitt, 1995; Scherer, Dickersin, & Langenberg, 1994; Shadish, Doherty, &…”

“In considering these analyses, it is important to note that funnel plot asymmetry could be due to factors other than publication bias and because of this the Trim and Fill method may lead to overcorrection (Vevea & Woods, 2005).”

“Yet another way to interpret the MS ES is to investigate the possibility of publication bias, the tendency for the availability of research to depend on the results (Vevea & Woods, 2005).”

“Opening this file in a text editor reveals the code that controls the weight functions: # Enter fixed weight function w1 ,- matrix(c(1.0,.99,.95,.90,.80,.75,.65,.60,.55,.50,.50,.50,.50,.50), ncol ¼ 1) w2 ,- matrix(c(1.0,.99,.90,.75,.60,.50,.40,.35,.30,.25,.10,.10,.10,.10), ncol ¼ 1) w3 ,- matrix(c(1.0,.99,.95,.90,.80,…”

“Next, we performed the bias tests using selection models (Vevea & Woods, 2005), which are weightfunction models accounting for the fact that not all effect sizes have the same probability of being published.”

“We applied the same four different weight functions in our analyses as were used by Vevea and Woods (2005).”

“Of note, Vevea and Woods’ (2005) weight function involves choice of direction and size for the weight function.”

“…Qouta, Sarraj, & Montgomery, 2006 PTSD Reaction Index 3 0.11 0.11 0.00 0.22 Smith et al., 2001 Impact of Events Scale — Revised 2 0.35 0.03 0.33 0.37 Wickrama & Kaspar, 2007 DSM-IV diagnostic 1 0.40 interview items Parental depression Meiser-Stedman, Yule, Dalgleish, Smith, & Glucksman,…”

“3) revealed an asymmetric distribution that suggested a one-tailed overrepresentation of positive effect sizes in studies with comparatively low statistical power (43).”

“The weights of all four selection methods are obtainable from Table 1 of Vevea and Woods’ (2005) article.”

“To investigate the role of publication bias in determining the actual presence of this effect, we employed a computer program designed by Vevea and Woods (2005).”

“In order to triangulate the possibility of publication bias, we report the results of the trim and fill (Duval, 2005), moderate selection models (Vevea & Woods, 2005), and cumulative meta-analysis (Kepes, Banks, McDaniel, & Whetzel, 2012).”

“While it has been suggested that correcting for publication bias is not possible without making untestable assumptions [130], a bound for the bias as a function of the fraction of missing studies has been proposed [131], as has the use of a priori specified selection models [132].”

“Indeed, meta-analyses that ignore unpublished studies likely overstate the true effect size (e.g., Vevea & Woods, 2005).”

“We used the method described by Hedges and Vevea (1996) and Vevea and Woods (2005) to compute a sensitivity analysis because it performs well in random-effects designs, can accommodate the inclusion of moderators, and allows for the flexibility to explore specific conditions of publication bias…”

“…model that calculates the adjusted metaanalytic effect size estimate by considering the statistical significance of the effect size in certain samples (Vevea and Woods 2005) shows that the availability bias in our meta–data set is not likely to be a serious issue in moderate asymmetry situations.”

“There are sophisticated methods available for such analyses (e.g., Vevea & Woods, 2005), but the authors of the meta-analyses in our sample rarely used them (the rare exceptions were Pettigrew & Tropp, 2006; Shadish & Baldwin, 2005).”

“We identified the specific weights and probability value cutoff points for our sample based on Vevea and Woods (2005).”

“…less for obtaining a single estimate that purports to adjust for publication bias ex post and more for sensitivity analysis—that is, exploring the range of estimates that result from assuming different forms of and severity of publication bias (see Vevea & Woods, 2005, and Hedges & Vevea, 2005).”

“That is, as the number of studies we found on mentalization was small, single studies may be very influential to the calculation of the pooled correlation on mentalization and attachment/ sensitivity (Vevea & Woods, 2005).”

“These analyses were fail-safe k, Egger’s test, randomeffects trim-and-fill method, and weight-function model analysis (Coburn & Vevea, 2015; Egger, Davey Smith, Schneider, & Minder, 1997; Vevea & Hedges, 1995; Vevea & Woods, 2005).”

“Unfortunately, the estimation of publication bias using selection models has been limited by data set requirements because a large number of samples is necessary to estimate the model with adequate accuracy (e.g., at least 100 samples; Vevea & Woods, 2005).”

“We estimated the parameters for the random-or mixed-effects models with unconditional maximum likelihood using a computer program developed for that purpose (Vevea & Hedges, 1995; Vevea & Woods, 2004).”

“The R-packages Weightr (Coburn & Vevea, 2016; weighting schemes provided by; Vevea & Woods, 2005) and Metafor (Viechtbauer, 2010) were used for WFM and contour-en-hanced funnel plot production, respectively.”

“…meta-analysis are only mildly attenuated even under severe publication bias, then the results might be considered fairly robust to publication bias (Vevea and Woods, 2005), whereas if the results are severely attenuated under mild publication bias the results might be considered sensitive to…”

“The Vevea and Woods [58] method estimated that the SMD for TAG measures comparing PA breaks with sitting would be reduced from − 0.”

“We used four different study weight functions as specified by Vevea and Woods (2005) assuming either moderate or severe and one- or two-tailed selection of effect sizes.”

“In general, funnel plots can be useful to help detect publication bias, but interpretation can be dubious (Vevea & Woods, 2005).”

“Some approaches estimate the selection model [74, 77] whereas others assume a known selection model [79].”

“05) effect sizes [108] or test the robustness of the results by postulating a probability for the nonsignificant effects to be published [109,110].”

“Their sensitivity analysis (4) indicates that the true effect size could be as low as d = 0.”

“Third, a priori selection models (Hedges & Vevea, 2005; Vevea & Woods, 2005) can be used to weight samples in the analysis on the basis of the probability that they are available to be included in the meta-analysis.”

“Although that was the case in the PRT dataset, it will not always hold—though in such cases the methods of Vevea and colleagues (Citkowicz & Vevea, 2011; Hedges & Vevea, 1996; Vevea & Hedges, 1995; Vevea & Woods, 2005) should be considered.”

“Although this is consistent with a publication bias against non-significant or negative results, it may also arise from phenomena such as certain patterns of heterogeneity [221].”

“As a final test of publication bias, we carried out a sensitivity analysis hypothesizing both moderate and severe one-tailed selection of studies (Vevea & Woods, 2005).”

“The main change in the modification model was imposing a set of fixed weights determined a priori, but not estimating the weights from the data [43]; yet, a priori chosen weights are usually arbitrary, so it should be used and interpreted cautiously.”

“…bias, namely, the tendency of editors, reviewers, and individual researchers to favor only statistically significant or theoretically appealing findings (Moncrieff, 1998; Torgerson, 2006; Vevea & Woods, 2005; for a comprehensive review of this issue, see Rothstein, Sutton, & Borenstein, 2005).”

“2 ) the cloud is a little sparse around zero for small studies, which indicates two-tailed publication bias (Vevea & Woods, 2005) .”

“Fifth, we employed a selection model with a priori weight functions that could model four different scenarios of publication biases: moderate one-tailed selection, severe one-tailed selection, moderate two-tailed selection, and severe two-tailed selection (Vevea & Woods, 2005).”

“To quantify the likely effect of publication bias, a sensitivity analysis based on Vevea and Woods (2005) was conducted.”

“…2005), trimand-fill analysis (Duval & Tweedie, 2000) following the approach as suggested by Peters, Sutton, Jones, Abrams and Rushton (2007), a sensitivity analysis for publications bias as suggested by Vevea and Woods (2005), and a method based on truncated normal distributions (Formann, 2008).”

“2009), the trim and fill method (Duval and Tweedie 2000a, b), Egger et al.’s (1997) regression test, failsafe N (Rosenthal 1979), or selection models (Hedges and Vevea 2005; Vevea and Woods 2005).”

“We used funnel plots and Egger tests to assess the risk of publication bias and used a priori weight functions for sensitivity analyses.(60)”

“To quantify the likely effect of publication bias, a sensitivity analysis based on Vevea and Woods (2005) was conducted which adjusts the population effect size estimate for moderate and severe one-and two-tailed selection bias.”

“, negative-exponential) [34,57], step functions with more than one cut point [34,36,58], and nonparametric functions [38, 39].”

“…plots (Peters, Sutton, Jones, Abrams, & Ruston, 2008; Sterne & Harbord, 2004), Begg’s rank correlation test (Begg & Mazumdar, 1994), and Egger’s regression test (Egger, Smith, Schnieder, & Minder, 1997), trim-and-fill analysis (Duval & Tweedie, 2000a,b), and selection models (Vevea & Woods, 2005).”

“Weights reflect the process by which studies tend to be published based on higher levels of statistical significance (Vevea and Woods 2005).”

“Fixed Weights Selection Model [61–63].”

“tion bias effects (Vevea and Woods, 2005).”

“However, it should be noted that the 3PSM sensitivity test is less robust with less than 10 data points (Vevea & Woods, 2005), which was the case for correlational amotivation studies and intervention studies of identified regulation.”

“Funnel plots as proposed, for example, by Vevea and Woods (2005) are principally suited for small meta-analyses, but at least 10 data sets should be included in each calculation (Higgins & Green, 2011), which was not the case for all calculations.”

“Third, we employed a sensitivity analysis approach described by Vevea and Woods (2005), which produces adjusted effect size estimates.”

“1, R Foundation for Statistical Computing, Vienna, Austria) using a code written byVevea andWoods [19] and applying a script from Field and Gillett to feed SPSS data into R [18].”

“We used the packages compute.es (Del Re, 2013), robumeta (Fisher & Tipton, 2015), metafor (Viechtbauer, 2010), clubSandwich (Pustejovsky, 2016) and the Shiny application weightr that implements the weight function model published by Vevea and Woods (2005).”

“Following Vevea and Woods (2005), we specify the weights at each cut point (rather than estimate them as in Hedges and Vevea (1996)).”

“A t p o st in te rv e n tio n , M B S R g ro u p sh o w e d a te n d e n c y to w a rd a lo w e r A U C G th a n d id th e in a c tiv e c o n tr o ls .”

“Vevea and Woods (2005) demonstrated that there are situations where visual examination of the funnel plot may make it difficult to detect evidence of subtle but important publication bias.”

“Although the result of Kendall’s tau is non-significant, in view of the small number of studies included in the metaanalysis, publication bias cannot be rejected and therefore a sensitivity analysis (Vevea and Woods, 2005) was conducted to estimate and account for the likelihood of publication bias.”

“This study used weightr package (Coburn & Vevea, 2019) to apply the weight-function model (Vevea & Woods, 2005).”

“de Vibe and Moum39 2006 Norway Patients with stress and chronic illness I: 47 (20-69) PC: 48 (17-63) 88.”

“The method of Vevea and Woods [42] corrects for publication bias by adjusting weights assigned to different studies based on study characteristics that could influence likeliness of publication, such as statistical significance.”

“Following Vevea and Woods (2005), a prespecified sensitivity analysis was conducted using a priori weight functions.”

“28 – 30 The new method proposed here is another useful tool to accommodate publication bias, and researchers are free to supplement it with other methods.”

“Thus, we also used a more recent approach suggested by McShane et al. (2016; see also Vevea & Woods, 2005): sensitivity analysis.”

“Given the mixed evidence on the potential presence of publication bias, we decided to apply Vevea and Woods (2005) selection model to obtain a corrected, overall effect size.”

“The selection model approach takes into consideration the probability that certain samples are included in a meta-analysis due to specific characteristics (Hedges & Vevea, 2005; Vevea & Woods, 2005).”

“It is important to consider that funnel plot asymmetry may be due to factors other than publication bias and because of this, the Trim and Fill method may lead to overcorrection (Vevea & Woods, 2005).”