Inaccurate Estimation of Disparities Due to Mischievous Responders: Several Suggestions to Assess Conclusions

Abstract

This article introduces novel sensitivity-analysis procedures for investigating and reducing the bias that mischievous responders (i.e., youths who provide extreme, and potentially untruthful, responses to multiple questions) often introduce in adolescent disparity estimates based on data from self-administered questionnaires (SAQs). Mischievous responders affect a wide range of disparity estimates, including those between adoptees and nonadoptees, sexual minorities and nonminorities, and individuals with and without disabilities. Thus, the procedures introduced here have broad relevance to research and can be widely, and easily, implemented. The sensitivity-analysis procedures are illustrated with SAQ data from youths in Grades 9–12 ( N = 11,829) to examine between-group disparities based on sexual identity, gender identity, and physical disability. Sensitivity analyses revealed that each disparity estimated with these data was extremely sensitive to the presence of potentially mischievous responders. Patterns were similar across multiple approaches to dealing with mischievous responders, across various outcomes, and across different between-group comparisons. Mischievous responders are ubiquitous in adolescent research using SAQs and can, even in small numbers, lead to inaccurate conclusions that substantively affect research, policy, and public discourse regarding a variety of disparities. This article calls attention to this widespread problem and provides practical suggestions for assessing it, even when data are already collected.