Re: "Dealing With Missing Outcome Data in Randomized Trials and Observational Studies"

Abstract

Groenwold et al. (1) offer advice on ways to handle missing outcome data in randomized experiments and observational studies. We agree that, regardless of the amount of effort invested in collecting complete data, it is difficult to avoid some missing values. However, we are concerned about the conclusion, stated in the abstract and the discussion, that “complete case analysis with covariate adjustment can and should be used as the analysis of choice more often” (1, pp. 210 and 217). In support of this conclusion, Groenwold et al. presented a simulation study that considered a very basic scenario: binary treatment with a constant treatment effect, 1 fully observed covariate, and 1 partially missing outcome, either binary or continuous, in which the parameters of the missingness mechanism were distinct from those that governed the distribution of the outcomes (2). In the simulations, complete case analysis with covariate adjustment (CCA-CA) yielded results similar to those from the multiple imputation method implemented using multivariate imputation by chained equations (3). For the simulations under missing-not-at-random mechanisms, CCA-CA and multiple imputation had essentially identical bias and interval coverage. This agreement is obvious because in this basic scenario, both methods used effectively the same inappropriate model. However, the fact that both methods achieved the same wrong result cannot justify a preference for one over the other. Moreover, as the authors note (1), multiple imputation has the advantage of being flexible enough to use alternative assumptions about the missing data mechanism, thereby performing sensitivity analyses (4), which is especially useful when the data may be missing not at random. For the simulations under missing-at-random mechanisms, CCA-CA performed as well as the multiple imputation method, with both demonstrating zero bias and the correct coverage for intervals. As explained by Little and Rubin (5), if the missingness mechanism is ignorable (i.e., missing at random with distinct parameters), likelihood inference on the observed data produces efficient and consistent estimates. Because in the basic scenario the maximum likelihood estimate of the constant treatment effect is identical to the CCA-CA estimate, the similarity of the results is again obvious. However, this is essentially the only scenario in which the results from CCA-CA correspond to those from multiple imputation. For a more realistic scenario with multiple covariates and/or multivariate outcomes with missingness in more than 1 variable, CCA-CA would not be optimal and could be badly biased. Perhaps this crucial observation could be inferred from the discussion in the article by Groenwold et al. (1); however, we feel that it is important to state this fact explicitly to avoid potential misuse of the stated conclusions. Moreover, the arguments offered by Groenwold et al. (1) do not apply to situations in which marginal population characteristics are of interest and individual causal effects depend upon covariates. In those cases, CCA-CA is inappropriate unless the missing data are missing completely at random, and even then it is generally not efficient (6). CCA-CA should be used with caution because it is apposite only in simple missingness cases (5). To us, multiple imputation is a more transparent method because it requires the analyst to be explicit about the procedures used to create the imputations and therefore to acknowledge the underlying assumptions. Hence, in any setting, researchers would be advised to use the multiple imputation method, for both gaining efficiency and reducing bias, with the added advantage of having the flexibility to perform sensitivity analyses. Finally, a recent substantial report by a panel of the US National Academy of Sciences reached conclusions consistent with ours: “Modern statistical analysis tools—such as maximum likelihood, multiple imputation, Bayesian methods, and methods based on generalized estimating equations—can reduce the potential bias arising from missing data by making principled use of auxiliary information available for nonrespondents. The panel encourages increased use of these methods” (7, p. 2). Conflict of interest: none declared.