Commentary: Can a Quasi-experimental Design Be a Better Idea than an Experimental One?

Abstract

Regression discontinuity works in situations where a threshold in a continuous variable is used to assign treatment.1 An example of such a rule would be assigning treatment only to participants with a systolic blood pressure (SBP) of 140 mmHg or higher. Because the probability of treatment changes discontinuously from zero below 140 mmHg to one at and above 140 mmHg, a discontinuity, or abrupt change, should also be observed in the outcome when plotted against blood pressure if treatment has a causal effect. Although the statistical machinery used to analyze regression discontinuity designs is sometimes complex, the basic task is to estimate this discontinuity in the outcome. Many have written that regression discontinuity is the observational design that most resembles a randomized control trial (RCT).2 In both designs, we have precise knowledge of the assignment mechanism. RCTs assign treatment through random allocation and regression discontinuity assigns treatment using a threshold in a measured, continuous variable often referred to as an assignment variable. Therefore, the treated and untreated in an RCT are unconditionally exchangeable, whereas in a regression discontinuity design the treated and untreated are only exchangeable conditional on the assignment variable.2 The added wrinkle of the regression discontinuity assignment mechanism is that there is no overlap in the assignment variable in the treated and untreated groups meaning there is no way to condition on the assignment variable without making important, often impossible, assumptions.3 This is the reason causal estimates in a regression discontinuity design are made at the threshold: because observations close to the threshold on either side are considered to be exchangeable. Although the regression discontinuity design and RCTs share these similarities, we are aware of only one previous attempt to compare the two in a real world setting.4 In this issue of EPIDEMIOLOGY, van Leeuwen and colleagues5 attempt to compare the real world validity and statistical efficiency of the RCT and regression discontinuity designs using both simulations and RCT data. Similar to the previous attempt to compare RCT and regression discontinuity designs, the authors restrict data from an RCT to make it appear as though it were generated from a regression discontinuity design.4 They select a baseline measurement as an assignment variable, create an assignment rule, and drop all observations that do not comply with this rule. For example, the authors use data from an RCT evaluating the effect of the nurse-led intervention targeting cardiovascular risks on the 6-year risk of dementia. They apply an assignment rule where only participants with SBP ≥ 140 mmHg received treatment by eliminating untreated observations with SBP ≥ 140 mmHg and treated observations with SBP < 140 mmHg. This is a clever way of creating a regression discontinuity dataset that should in theory yield causal estimates similar to the RCT estimates. COMPARING RCTS AND REGRESSION DISCONTINUITIES: EASIER SAID THAN DONE Even with comparable RCT and regression discontinuity datasets in hand, however, comparing RCT and regression discontinuity estimates is not straightforward. RCTs estimate an average treatment effect and the regression discontinuity designs generally estimate a local average treatment effect among participants at the threshold. Only under specific circumstances do they estimate the same quantity. The simplest case is when the assignment variable does not modify the treatment effect. In other words, the treatment effect is the same regardless of the baseline value of the assignment variable, in which case the local average treatment effect is equal to the average treatment effect. One test of whether or not this can be assumed is whether the slopes on each side of the cutoff are parallel.3 In none of the examples presented by van Leeuwen and colleagues Figure 1 in reference 5 does this seem to be the case. To the contrary, in all cases, we observe different treatment effects by assignment variable. Therefore, if the authors wish to make a direct comparison, they must find a way to estimate the local average treatment effect with RCT data or to estimate the average treatment effect with the regression discontinuity data. Estimating a local average treatment effect with RCT data can be done relatively straightforwardly by separately modeling the assignment variable–outcome relationship in the treated and untreated groups and taking the difference at the threshold selected for the regression discontinuity estimate. The authors claim to choose this approach when using blood pressure and cholesterol as the assignment variable but only report one estimate. It is not clear to which threshold this estimate corresponds. It is much more difficult to estimate an average treatment effect within the regression discontinuity design when the assignment variable is an effect measure modifier. To do