Preventing treatment spillover contamination in criminological field experiments: the case of body-worn police cameras

“Empirically, the presence of spillovers may vary widely, leading to the same question that faces ITT itself (Peto et al. 1976): how much is too much?”

“Empirically, the presence of spillovers may vary widely, leading to the same question that faces ITT itself (Peto et al. 1976): how much is too much? If only 10% of an intended treatment group is actually treated, compared to 5% of a no-treatment control group, many would think that ITT analysis is pointless.”

“When there are more shifts or other temporal units (e.g., days of the week) than police officers, especially in midsize departments, substitutes ought to be considered to satisfy the sample size problem (unless a Bayesian approach is possible; see Lenth 2001).”

“Nonetheless, we can see that the question of where to draw the line remains central (Sherman 1993).”

“Reference groups (Shibutani 1955), small-group psychology (Shaw 1911), Lucifer effects (Zimbardo 2007), social identity (Stott and Drury 1999), and a myriad of effects would either negatively or positively motivate participants in the group to act in various ways.”

“This so-called “actor-partner interdependence model” (APIM; Kashy and Kenny 2000) can be used to analyze dyadic data.”

“…also provide a useful solution to handling treatment propagation in clusters, and these are becoming more common in observational data (Lyons 2011; Shalizi and Thomas 2011) and randomized experimental designs (Aral and Walker 2011; Aronow and Samii 2013; Bapna and Umyarov 2015; Bond et al.…”

“Reference groups (Shibutani 1955), small-group psychology (Shaw 1911), Lucifer effects (Zimbardo 2007), social identity (Stott and Drury 1999), and a myriad of effects would either negatively or positively motivate participants in the group to act in various ways.”

“This so-called “actor-partner interdependence model” (APIM; Kashy and Kenny 2000) can be used to analyze dyadic data.”

“…in clusters, and these are becoming more common in observational data (Lyons 2011; Shalizi and Thomas 2011) and randomized experimental designs (Aral and Walker 2011; Aronow and Samii 2013; Bapna and Umyarov 2015; Bond et al. 2012; Eckles et al. 2017; Ichino and Schündeln 2012; Ostrovsky and…”

“For present purposes, we would prefer to describe it as “contagion effects,” or “synergistic effects,” similar to the concept of “herd immunity” in vaccinations (Anderson and May 1985).”

“Put another way, when conducting an experiment—following Cox (1958) and Rubin (1980)—we are assuming that the effect of an intervention on any 2 Recalling that the origins of experimental science were actual trials in fields conducted by Sir Ronald Fisher (see the discussion in Armitage 2003). one…”

“…we would not observe significant differences between treatment and control conditions across multiple outcomes using this research design (e.g., Ariel et al. 2015; Ariel et al. 2016a; Ariel et al. 2016b).8 To summarize, a shift-based design can create, in theory, both negative and positive…”

“There are two broad types of spillover effects: major interference and partial interference (Sobel 2006).”

“…in observational data (Lyons 2011; Shalizi and Thomas 2011) and randomized experimental designs (Aral and Walker 2011; Aronow and Samii 2013; Bapna and Umyarov 2015; Bond et al. 2012; Eckles et al. 2017; Ichino and Schündeln 2012; Ostrovsky and Schwarz 2011; Rosenbaum 2007; Toulis and Kao 2013).”

“…when the sample size is large relative to the experimenter’s resources tomanage implementation, and thus high fidelity across units is more challenging (Weisburd, Petrosino and Mason 1993; Weisburd et al. 2001).4 4 We note that an additional consideration is the size of the sample of officers.”

“A host of institutional undercurrents that are recognized in the literature (Sherman 1980), but cannot be factored into a statistical protocol without detailed information about the officers themselves, may affect the “independence” of individuals from factors affecting the deployment of officers…”

“…and formalized literature in other experimental disciplines has paid closer attention to these concerns— mainly in statistics (see an early review by Page 1978 and more recently by Bowers et al. 2018, see also Hudgens and Halloran 2008 and Rosenbaum 2007).3 Kruskal (1988), for example, discusses…”

“We conclude by reiterating that shift randomization allows researchers to maximize sample sizes, be in a better position to characterize SUTVA violations (see Sampson 2010) and minimize problems arising from spillover effects.”

“For further reading on this approach, see Ledermann and Kenny (2015), Cook and Kenny (2005), and Garcia et al. (2015).”

“…of approximately 128 officers and no covariates to increase the statistical power of the test, a study is unlikely to detect effects below d = 0.499, and the practice in some studies had been to relax some of these statistical assumptions of the power test (e.g., Jennings et al. 2015, p. 482).”

“…techniques also provide a useful solution to handling treatment propagation in clusters, and these are becoming more common in observational data (Lyons 2011; Shalizi and Thomas 2011) and randomized experimental designs (Aral and Walker 2011; Aronow and Samii 2013; Bapna and Umyarov 2015; Bond…”

“The spillovers can operate at different levels, bleeding from treatment to control, between different treatment groups, within statistical blocks or clusters or within individual treatment units (Baird et al. 2016; Campbell and Stanley 1966; Shadish et al. 2002).”

“From a theoretical perspective that would then affect the computation of the predictors, BWCs may have at least as much of an effect on citizens as they do on officers, particularly if citizens are verbally warned that cameras are being used (Ariel et al. 2016c).”

“A fundamental rule of RCTs is the axiom to “analyze as you randomize” (Boruch 1997:203; Senn 2004).”

“For a more elaborate discussion, see Aronow and Samii (2017), Bowers et al. (2013), and Toulis and Kao (2013). to recent methodological work on statistical inferences about peer effects or total average effects, when the topology of the network can be explained (Aronow and Samii 2017; Bowers et al.…”

“A fundamental rule of RCTs is the axiom to “analyze as you randomize” (Boruch 1997:203; Senn 2004).”

“…policing can purposely patrol control sites even though they were instructed otherwise (as they did in the first systematic patrol experiment; see Kelling et al. 1974), and prisoners randomly assigned to a particular rehabilitation program can engage with control prisoners, all in a way in which…”

“…paid closer attention to these concerns— mainly in statistics (see an early review by Page 1978 and more recently by Bowers et al. 2018, see also Hudgens and Halloran 2008 and Rosenbaum 2007).3 Kruskal (1988), for example, discusses the causal assumption of independence and makes a key observation.”

“The logical meaning of “what would have happened otherwise” collapses if the control group receives the treatment (Nagin and Sampson 2018).”

“Analyzing at the officer-level following shift-level randomization (i.e., ignoring the fact that officers are clustered by shift) would undermine the experimental design, becoming the exercise in self-deception against which Cornfield (1976) warns.”

“Individual vs. temporal units: statistical considerations Let us return to the notion of “analyze as you randomize” (see Senn 2004; Demir et al. 2018 ; Ariel et al. 2018; Maskaly et al. 2017).”

“The statistical literature should be consulted (Basse and Airoldi 2017; Basse and Feller 2017; and others).”

“…on any 2 Recalling that the origins of experimental science were actual trials in fields conducted by Sir Ronald Fisher (see the discussion in Armitage 2003). one individual/unit, “unit A” for example, is unrelated to the treatment assignments of other people/units in the study (units B,…”

“Individual vs. temporal units: statistical considerations Let us return to the notion of “analyze as you randomize” (see Senn 2004; Demir et al. 2018 ; Ariel et al. 2018; Maskaly et al. 2017).”

“The number of contacts per 10,000 encounters that lead to complaints against the officer, for example, or that result in the use of force, is remarkably small (see Terrill and McCluskey 2002).”

“Formally, this means that participants who function together in groups usually yield scores that are correlated (Peckham et al. 1969).”

“The statistical literature should be consulted (Basse and Airoldi 2017; Basse and Feller 2017; and others).”

“Braga et al. (2013) and Braga and Weisburd (2014, pp. 583–586) show how positive spillover effects contributed to meaningful reductions in gang violence.”

“More importantly, spillover effects often result from experiments, which indeed may be the intention (Angelucci and Maro 2016).”

“For further reading on this approach, see Ledermann and Kenny (2015), Cook and Kenny (2005), and Garcia et al. (2015).”

“While this perception may be true (Tankebe and Ariel 2016), in practice, this means there is a reduction in treatment intensity because officers are not complying with the protocol.”

“Reference groups (Shibutani 1955), small-group psychology (Shaw 1911), Lucifer effects (Zimbardo 2007), social identity (Stott and Drury 1999), and a myriad of effects would either negatively or positively motivate participants in the group to act in various ways.”

“This “analyze as you randomize” rule is the preferred method of dealing with crossover among medical scholars (e.g., Armitage 1991).”

“The issue in both cases is not whether imperfections exist, but how much tolerance the design has for such imperfections, as the history of precision engineering clearly demonstrates (Winchester 2018).”

“While policing experiments may have underemphasized the issue of spillover and interference effects (but cf. Braga et al. 2018), a relatively developed and formalized literature in other experimental disciplines has paid closer attention to these concerns— mainly in statistics (see an early review…”

“In terms of random assignment, statisticians offer a partial although convincing solution to the interference issue: model-assisted restricted randomization strategies that take into account these interference effects (see Yates 1948, but more recently see Ariel and Farrington 2010).”

“In the case of randomized designs, those units in the treatment and control groups should be exchangeable (see Hartman et al. 2015).”