An audit of social science survey experiments

Abstract

Survey experiments have become a popular methodology for causal inference across the social sciences. We study the efficacy of survey experiment designs by analyzing 100 social science experiments—entailing more than 1,000 hypothesis tests—that were selected by experts via a competitive process and fielded on probability samples of US adults between 2012 and 2020. Inclusion in the analysis is only conditional on the experiment qualifying for data collection, and not in any way on study results or publication. Results show that less than a third of proposed hypotheses were supported by the data, implying many more null findings than ostensibly appear in the published literature. We find that the largest predictor of positive experimental results was sample size. This is somewhat surprising, given that experimental studies typically take power considerations into account prior to data collection. In our data, the importance of sample size stemmed from small effect sizes across studies (perhaps smaller than researchers may have anticipated), highlighting a tension between commonly used power calculi and determining what constitutes a “meaningful effect.” We also find that moderation hypotheses were rarely significant, and that using multiple items for outcome measures did not affect results as expected. But indicators of research experience predicted higher rates of positive results, suggesting that there may be some room for optimizing experiment outcomes by minimizing design e…