Correcting for Bias in Psychology: A Comparison of Meta-Analytic Methods

In the citing paper

“The absence of publication bias is highly unlikely unless a set of preregistered reports is examined (Carter et al. 2019).”

In the citing paper

“The trim-and-fill method (Duval & Tweedie, 2000) tends to adjust too little when bias is strong (Carter et al., 2019), and Fail-Safe N (Rosenthal, 1979) neither tests nor adjusts for the presence of bias.”

In the citing paper

“Bias adjustments can improve performance relative to a meta-analysis that does not adjust for publication bias (Carter et al., 2019; McShane et al., 2016; Moreno et al., 2009; Simonsohn et al., 2014).”

In the citing paper

“negative and statistically significant SPET estimates have been observed in both simulation studies (Carter et al., 2019) and meta-analysis (Hilgard et al., 2019).”

In the citing paper

“On the one hand, this may be an underestimate of average study power—in simulation studies, p-hacking adds large p values to the p-curve, canceling out the right skew and biasing the p-curve estimate downward (Carter et al., 2019).”

In the citing paper

“SPET and SPEESE have a downward bias in the presence of p-hacking (Carter et al., 2019)”

In the citing paper

“There may also be downward bias inflicted by p-hacking (Carter et al., 2019).”

In the citing paper

“Further, PET-PEESE can underestimate the effect size when there is publication bias and when p-hacking is present (Carter et al., 2019), which are two conditions likely affecting the literature.”

In the citing paper

“However, this counterintuitive finding may be explained by the fact that PET is prone to producing imprecise and biased estimates under high heterogeneity, small sample sizes, and a large proportion of p-hacking (Stanley & Doucouliagos, 2014; Stanley 2017; Carter et al., 2019).”

In the citing paper

“Simulation studies by Carter et al. (2019) showed that WAAP-WLS can provide a reasonably unbiased estimate of the effect size with a high power to detect the effect given that a true underlying effect exists.”

In the citing paper

“This could lead selection model to underestimate effects in the presence of publication bias and p-hacking, as shown in simulation studies (Carter et al., 2019).”

In the citing paper

“Lastly, PET-PEESE can underestimate the size of small effects in the presence of p-hacking and publication bias (Carter et al., 2019).”

In the citing paper

“However, when effects are highly heterogeneous and when sample sizes are small, the PET-PEESE analysis is known to yield imprecise estimates (Stanley & Doucouliagos, 2014; Stanley, 2017; Carter et al., 2019).”

In the citing paper

“there is a danger that uncritical application can lead to inappropriate conclusions being drawn 68,73,74”

In the citing paper

“the available evidence suggests that no single method outperforms others in all scenarios”