On the predicted replicability of two decades of experimental research on system justification: A Z‐curve analysis

Abstract

We examine the predicted replicability of experimental research on system justification theory (SJT) by conducting a z‐curve analysis. Z‐curve is a meta‐analytic technique similar to p‐curve, but which performs better under conditions of heterogeneity. It estimates the predicted replication rate, average power, false discovery risk, and file drawer ratio (FDR) of a sample of studies. The z‐curve based on 116 papers and 232 unique samples suggests that the experimental SJT literature is likely to show low rates of replicability, as indicated by an overall average statistical power of 16%. Moderator analyses suggest that this may be driven in part by publication pressures, that the replicability of research in this area has improved since 2015, and that studies using system threat manipulations show particularly low estimated replication rates (ERR). Implications for the replicability and validity of the experimental SJT literature are discussed, and recommendations to increase the rigor of research are put forth.