Agile science: creating useful products for behavior change in the real world

In the paper

“For example, in the Fig. 1 dynamical model, the team has conducted a series of simulations to explore different ways of defining an “ambitious but doable” goal based on interactions between the variables in the model.”

In the paper

“As our ongoing suite of complementary projects illustrate, we are actively working toward testing the assumptions outlined.”

In the paper

“Multiple complementary research projects are underway that each focus on developing a different module for a complex intervention for physical activity.”

In the paper

“A third complementary project is focused on the development of a tool that can support the rapid creation and early-and-often sharing of behavior change modules.”

In the paper

“In partnership with Bob Evans at Google and his open-source Paco system (www.pacoapp.com), at the time of writing, we are building a system that allows researchers to rapidly build behavior change modules.”

In the paper

“Within the ambitious but doable and context cue/implementation intention projects, we previously engaged in formative work, simulation studies, and prototype testing and thus are currently utilizing the operations created previously in the evaluate phase.”

In the paper

“Evidence-based practices not only require careful examination of the efficacy of an intervention but also the ability of the evidence-based practice to be translated and disseminated for real-world use [2].”

What the sources say

• Not checked yetcites [2] Rotheram-Borus MJ, Swendeman D, Chorpita BF. Disruptive innovations for designing and diffusing evidence-based interventions. Am Psychol. 2012; 67: 463. source

In the paper

“Indeed, Chorpita et al. [20] have argued that the fundamental challenge in behavior change science may be an overemphasis on knowledge proliferation (i.e., the development of new treatments) at the cost of knowledge management (e.g., developing new ways to design, organize, and share existing resources) and thus suggest focusing more on algorithms to curate knowledge.”

What the sources say

• Not checked yetcites [20] Chorpita BF, Rotheram-Borus MJ, Daleiden EL, et al. The old solutions are the new problem how do we better use what we already know about reducing the burden of mental illness? Perspect Psychol Sci. 2011; 6: 493–497. source

In the paper

“This reduces independent groups reinventing similar operations and also, if the information is well-curated and modules are interoperable, could facilitate increased specialization.”

In the paper

“This testing of assumptions via MVP modules is achieved via “sprints,” which involve a rapid cycle of identifying an assumption, building an MVP module to test the assumption, and then releasing the module to relevant stakeholders to test the assumption [23].”

What the sources say

• Not checked yetcites [23] Abrahamsson P, Warsta J, Siponen MT, Ronkainen J. New directions on agile methods: a comparative analysis. Proceedings of the International Conference on Software Engineering, 2003.

In the paper

“Put differently, sharing early and often enables individuals to build on one another’s work at the beginning and thus makes it easier for the process to facilitate interoperable and repurposable modules across complex interventions.”

In the paper

“As such, the personalization algorithm is a pragmatic target for developing evidence-based recommendations of complex interventions as the personalization algorithms establish the evidence base for selecting the right intervention option for a particular person in context.”

In the paper

“(3) sharing of operations enables different research groups to use the same operations in different use cases, which enables a more robust testing of external validity.”

In the paper

“Behavioral theories have classically been how behavioral sciences describe how interventions, individuals, and context interact but these are fraught with well-documented problems such as lack of falsification, lack of information about subtle dynamics, and they are often so abstract that they are difficult to apply in practice [36, 37].”

What the sources say

• Not checked yetcites [36] Hekler EB, Klasnja P, Froehlich JE, Buman MP. Mind the theoretical gap: interpreting, using, and developing behavioral theory in HCI research. Proceedings of SIGCHI Conference on Human Factors in Computing Systems (CHI’13). 2013; 3307–3316. source
• Not checked yetcites [37] Riley WT, Rivera DE, Atienza AA, et al. Health behavior models in the age of mobile interventions: are our theories up to the task? Translat Behav Med. 2011; 1: 53–71. source

In the paper

“A plausible reason for this siloing of evidence-based protocols is simply that we are sharing our operations at the wrong time.”

In the paper

“It has been argued elsewhere that evidence-based practices often become their own separate silos of evidence, rendering it very difficult to mix and match modules across evidence-based practices [2].”

What the sources say

• Not checked yetcites [2] Rotheram-Borus MJ, Swendeman D, Chorpita BF. Disruptive innovations for designing and diffusing evidence-based interventions. Am Psychol. 2012; 67: 463. source

In the paper

“Together, these three agile science products each uniquely contribute to supporting evidence-based decision making about which intervention to use for whom and at what time.”

In the paper

“In brief, striving for small modules enables repurposability.”

In the paper

“It is likely that a researcher engages in formative work, simulation studies, and even prototyping testing iteratively and/or simultaneously.”

In the paper

“MVPs are the simplified modules that are designed to test assumptions about the module with relevant stakeholders [25].”

What the sources say

• Not checked yetcites [25] Ries E. The lean startup: how today’s entrepreneurs use continuous innovation to create radically successful businesses. Random House LLC; 2011.

In the paper

“In terms of construct specification, concrete examples, particularly multiple examples, provide richer information about a construct than a textual definition alone.”

In the paper

“Careful delineation of inputs, processes, and outputs makes it easier to draw connections between behavior change modules and thus the architecture on how modules interact (including modules for measuring attributes of the person/population, setting, or outcome measure but,”

In the paper

“Within software development, the basic structure of modules is that they have specific inputs, a process that utilizes these inputs to transform them in some way and then this process creates specific outputs [24].”

What the sources say

• Not checked yetcites [24] Garlan D, Shaw M. An introduction to software architecture. Advances in software engineering and knowledge engineering. 1993; 1. source

In the paper

“The disruptive innovation process emphasizes the focus on providing simpler and less expensive alternatives to current practices that meet the essential need for the majority of users in order to make them more easily accessible, scalable, and replicable [2].”

What the sources say

• Not checked yetcites [2] Rotheram-Borus MJ, Swendeman D, Chorpita BF. Disruptive innovations for designing and diffusing evidence-based interventions. Am Psychol. 2012; 67: 463. source

In the paper

“With the value of sharing operations specified, there are three behavior change intervention product types that we view as particularly important for targeting as they are the conceptual building blocks for personalized complex interventions: behavior change intervention modules, computational models, and personalization algorithms.”

In the paper

“The micro-randomized trial is a sequential factorial design for modeling effectiveness of treatment components over time.”

What the sources say

• Not checked yetcites [16] Klasnja P, Hekler EB, Shiffman S, et al. Micro-randomized trials: an experimental design for developing just-in-time adaptive interventions. Health Psychol. 2016. source
• Not checked yetcites [17] Martin CA, Desphande S, Hekler EB, Rivera DE. A system identification approach for improving behavioral interventions based on social cognitive theory. American Control Conference (ACC). 2015; 5878–5883. doi: 10.1109/ACC.2015.7172261 . source
• Not checked yetcites [18] Martin CA, Rivera DE, Hekler EB. An identification test monitoring procedure for MIMO systems based on statistical uncertainty estimation. IEEE Conference on Decision and Control (CDC); 2015. source

In the paper

“By contender, we explicitly acknowledge that the products produced via the generate phase are not evidence-based.”

In the paper

“As discussed earlier, Riley et al. utilized simulation to generate a computational model of social cognitive theory [39].”

What the sources say

• Not checked yetcites [39] Riley WT, Martin CA, Rivera DE, et al. The development of a control systems model of social cognitive theory. Translat Behav Med. 2016.

In the paper

“As this example illustrates, good external validity requires variations in who uses an intervention, how the intervention might be operationalized, the outcomes the intervention is being targeted to influence, and the settings in which the intervention is used.”

What the sources say

• Not checked yetcites [28] Shadish WR, Cook TD, Campbell DT. Experimental and quasi-experimental designs for generalized causal inference. Wadsworth Cengage learning; 2002.

In the paper

“(3) algorithms for supporting decisions about matching interventions and modules with specific individuals in context is a second valuable target product and could be useful for curation.”

In the paper

“Greater value should be placed on concrete operations, even if they have not been empirically validated because:”

In the paper

“Model predictive control [40–43], recommender systems [44], agent-based modeling [45], or Bayesian network analysis [46] are all plausible personalization algorithm strategies.”

What the sources say

• Not checked yetcites [40] Dong Y, Rivera DE, Downs DS, et al. Hybrid model predictive control for optimizing gestational weight gain behavioral interventions. American Control Conference (ACC). 2013; 1970–1975. source
• Not checked yetcites [41] Nandola NN, Rivera DE. An improved formulation of hybrid model predictive control with application to production-inventory systems. IEEE Trans Control Syst Technol. 2013; 1: 121–135. source
• Not checked yetcites [42] Deshpande S, Rivera DE, Younger JW, Nandola NN. A control systems engineering approach for adaptive behavioral interventions: illustration with a fibromyalgia intervention. Translat Behav Med. 2014; 4: 275–289. Errratum in 274(273), p439. source
• Not checked yetcites [43] Martin CA, Rivera DE, Hekler EB. A decision framework for an adaptive behavioral intervention for physical activity using hybrid model predictive control. American Control Conference (ACC). 2016. source
• Not checked yetcites [44] Resnick P, Varian HR. Recommender systems. Commun ACM. 1997; 40: 56–58. source
• Not checked yetcites [45] Gilbert N. Agent-based models. Sage; 2008. source

In the paper

“For example, the founders of Dropbox made the assumption that there would be demand for a service that makes it easy to sync and share files. To test this assumption, they could have built the technology but that would have required considerable resources. Instead, they tested their assumption with a video (http://techcrunch.com/2011/10/19/dropbox-minimal-viable-product/). The video was a test on”

In the paper

“It can be used to test proximal main effects of components over time, time-varying moderation [16], and with variations in study design based on strategies from system identification, the development of idiographic computational models [17, 18].”

What the sources say

• Not checked yetcites [16] Klasnja P, Hekler EB, Shiffman S, et al. Micro-randomized trials: an experimental design for developing just-in-time adaptive interventions. Health Psychol. 2016. source
• Not checked yetcites [17] Martin CA, Desphande S, Hekler EB, Rivera DE. A system identification approach for improving behavioral interventions based on social cognitive theory. American Control Conference (ACC). 2015; 5878–5883. doi: 10.1109/ACC.2015.7172261 . source
• Not checked yetcites [18] Martin CA, Rivera DE, Hekler EB. An identification test monitoring procedure for MIMO systems based on statistical uncertainty estimation. IEEE Conference on Decision and Control (CDC); 2015. source

In the paper

“Within the behavioral sciences, reliable effect size estimates for constructs are more important to support evidence-based decisions [1].”

What the sources say

• Not checked yetcites [1] Kelly M, Morgan A, Ellis S, et al. Evidence based public health: a review of the experience of the National Institute of Health and Clinical Excellence (NICE) of developing public health guidance in England. Soc Sci Med. 2010; 71: 1056–1062. source

In the paper

“(2) while complex interventions might be desired in practice, modules of complex interventions are plausibly more valuable for science;”

In the paper

“These methods are acknowledged as critical for a wide range of evidence-based practice recommendations related to the research enterprise [5].”

What the sources say

• Not checked yetcites [5] Craig P, Dieppe P, Macintyre S, et al. Developing and evaluating complex interventions: the new medical research council guidance. Int J Nurs Stud. 2013; 50: 587–592. source

In the paper

“we see these two as complementary with, roughly, HCI processes conforming to the generate phase and behavioral science processes roughly conforming to the evaluate phase, though obviously there is overlap.”

In the paper

“It is essential to note that while modules are often built initially for a specific use case, an advantage of software module structure is that the module is built to be self-contained and thus repurposable for any use case with the requisite inputs (e.g., personal identification information, permission) and desires the requisite output (e.g., a secure bridge to share data across systems).”

In the paper

“As something is learned with one action, that information is fed into the others until contender operations and constructs emerge that show promise for being useful in the real-world.”

In the paper

“For example, extreme programming (XP), which is one agile method [23], emphasizes well-specified modules that are shared early-and-often with relevant stakeholders when building complex multi-component systems.”

What the sources say

• Not checked yetcites [23] Abrahamsson P, Warsta J, Siponen MT, Ronkainen J. New directions on agile methods: a comparative analysis. Proceedings of the International Conference on Software Engineering, 2003.

In the paper

“(1) operations are highly valuable for better specifying constructs;”

In the paper

“Modules, computational models, and personalization algorithms provide the evidence-based “building blocks” for generating practice recommendations for combining modules into complex interventions for unique use cases.”

In the paper

“This is easier with software but can be difficult with other modalities.”

In the paper

“That said, it could very likely improve the scientific community’s ability to create efficient accumulation of knowledge about behavior change.”

In the paper

“Further, sharing contender operations and constructs and also failed tests of these contenders from early work has great value for supporting better construct specification, improved efficiency in the scientific process, supporting earlier tests of feasibility, and could plausibly support better collection of data related to external validity and, by extension, support recommender systems for”

In the paper

“Other recommendations suggest a clear distinction between testing feasibility of the operations versus research methods [54].”

What the sources say

• Not checked yetcites [54] Arain M, Campbell MJ, Cooper CL, Lancaster GA. What is a pilot or feasibility study? A review of current practice and editorial policy. BMC Med Res Methodol. 2010; 10: 67. source

In the paper

“For example, sequential n-of-1 trials could serve as a more efficient feasibility testing method by allowing for intervention adjustments after each series of trials [55].”

What the sources say

• Not checked yetcites [55] Dallery J, Raiff BR. Optimizing behavioral health interventions with single-case designs: from development to dissemination. Translat Behav Med. 2014; 4: 1–14. source

In the paper

“We see at least three use cases: (1) quality assurance of a complex intervention that will be released “as is” to a large population; (2) as an “A/B” style test that is often used in the tech industry; and (3) as used in the “trials of principles” [51].”

What the sources say

• Not checked yetcites [51] Mohr DC, Schueller SM, Riley WT, et al. Trials of intervention principles: evaluation methods for evolving behavioral intervention technologies. J Med Internet Res. 2015; 17. source

In the paper

“For example, Bond et al. conducted a 61 million person A/B-style experiment on Facebook to explore how message framing might influence voting behavior [52].”

What the sources say

• Not checked yetcites [52] Bond RM, Fariss CJ, Jones JJ, et al. A 61-million-person experiment in social influence and political mobilization. Nature. 2012; 489: 295–298. source

In the paper

“With regard to idea generation and vetting, there are important cognitive biases that likely come into play that need to be better acknowledged in the generate phase.”

In the paper

“The two versions represent different hypotheses on how the system might function and are often only subtly different.”

In the paper

“As suggested by Ioannidis, it is quite plausible that the current reward system for academics, particularly the currency (i.e., publications and grants), may be resulting in the unintended consequence of incentivizing the creation of only minimally useful products for real-world use.”

What the sources say

• Not checked yetcites [60] Ioannidis JP. How to make more published research true. PLoS Med/ Public Libr Sci. 2014; 11, e1001747.

In the paper

“In particular, behavior change modules, computational models, and personalization algorithms are the building blocks for creating complex interventions and thus are logical scientific targets.”

In the paper

“A sister project headed by coauthor Klasnja, is focused on developing modules that support physical activity maintenance [16].”

What the sources say

• Not checked yetcites [16] Klasnja P, Hekler EB, Shiffman S, et al. Micro-randomized trials: an experimental design for developing just-in-time adaptive interventions. Health Psychol. 2016. source

In the paper

“From a human-participants perspective, micro-randomization trials are very efficient and thus could also be used after more classic n-of-1 trials have provided enough confidence to warrant investment in the design of one [16].”

What the sources say

• Not checked yetcites [16] Klasnja P, Hekler EB, Shiffman S, et al. Micro-randomized trials: an experimental design for developing just-in-time adaptive interventions. Health Psychol. 2016. source

In the paper

“As such, the four-phase model ignores variations of units, treatments, outcomes, and settings until the end of the process but it is highly feasible that when, where, for whom, and in what state of the individual can all moderate when an intervention will produce an effect [9].”

What the sources say

• Not checked yetcites [9] Rothman AJ. Exploring connections between moderators and mediators: commentary on subgroup analyses in intervention research. Prev Sci. 2013; 14: 189–192. source

In the paper

“The major limitations are intervention specification to support replication, which is ideal for all sciences and effective curation as already pointed out by others [20].”

What the sources say

• Not checked yetcites [20] Chorpita BF, Rotheram-Borus MJ, Daleiden EL, et al. The old solutions are the new problem how do we better use what we already know about reducing the burden of mental illness? Perspect Psychol Sci. 2011; 6: 493–497. source

In the paper

“As with any scientific process, the essential requirement is replication with sound definitions, robust versioning control, and a process of rapidly sharing and curating that information.”

In the paper

“As such, it is more akin to the optimization methods discussed earlier.”

In the paper

“In the digital technology industry, there is a common practice of comparing different versions (often two, hence A/B) of a given system.”

What the sources say

• Not checked yetcites [36] Hekler EB, Klasnja P, Froehlich JE, Buman MP. Mind the theoretical gap: interpreting, using, and developing behavioral theory in HCI research. Proceedings of SIGCHI Conference on Human Factors in Computing Systems (CHI’13). 2013; 3307–3316. source

In the paper

“As such, it fits well with agile science within the evaluate phase.”

In the paper

“RCTs of complex interventions are the most logical method for establishing that a specific complex intervention will work for a specific target population.”

In the paper

“This stay the course observation is similar to the sunk cost bias [59], which is a well-known cognitive bias that suggests an individual will pursue a less desirable option even if a more desirable option becomes available because of the resources already invested.”

What the sources say

• Not checked yetcites [59] Arkes HR, Ayton P. The sunk cost and Concorde effects: are humans less rational than lower animals? Psychol Bull. 1999; 125: 591. source

In the paper

“More rapid and parallel development could support both better defining of a construct and minimize the impact of the sunk cost bias, thus further reinforcing the value of early-and-often sharing and developing multiple operations.”

In the paper

“The third use case for RCTs is the trials of principles, which emphasizes that in behavioral intervention technologies, it is common for a software to change over time but the general principle (e.g., using goal setting as a technique) does not.”

What the sources say

• Not checked yetcites [51] Mohr DC, Schueller SM, Riley WT, et al. Trials of intervention principles: evaluation methods for evolving behavioral intervention technologies. J Med Internet Res. 2015; 17. source

In the paper

“As demonstrated elsewhere, careful use of priors within studies of novel systems can be used to help mitigate problems with magnitude effect errors and thus could be a more appropriate statistical method for the generate phase [47].”

What the sources say

• Not checked yetcites [47] Kaye M, Nelson GL, Hekler EB. Researcher-centered design of statistics: why Bayesian statistics better fit the culture and incentives of HCI. Proceedings of SIGCHI Conference on Human Factors in Computing Systems (CHI’16). San Jose, CA: ACM; 2016. source

In the paper

“Previous work suggests early studies often test feasibility of methods (e.g., recruitment, randomization, retention, and assessment), not hypotheses [48].”

What the sources say

• Not checked yetcites [48] Leon AC, Davis LL, Kraemer HC. The role and interpretation of pilot studies in clinical research. J Psychiatr Res. 2011; 45: 626–629. source

In the paper

“Results found that individuals randomized to work on multiple ideas at once (i.e., parallel group) had greater openness to incorporating feedback and also produced better end products compared to the single iterative concept group.”

What the sources say

• Not checked yetcites [58] Dow SP, Klemmer SR. The efficacy of prototyping under time constraints. Design Thinking. Springer; 2011:111–128. source

In the paper

“For example, Dow and colleagues showed the possibility of developing better results through iteration [57] and a subsequent study examined the differences between parallel and iterative development versus serial and iterative development [58].”

What the sources say

• Not checked yetcites [57] Dow SP, Glassco A, Kass J, et al. Parallel prototyping leads to better design results, more divergence, and increased self-efficacy. Transactions of Computer-Human Interactions (ToCHI). 2010; 17: 18.
• Not checked yetcites [58] Dow SP, Klemmer SR. The efficacy of prototyping under time constraints. Design Thinking. Springer; 2011:111–128. source

In the paper

“We believe it is quite plausible that the same process can work for other intervention modalities (e.g., face-to-face, group) and other types of complex problems (e.g., precision medications, public policy initiatives).”

In the paper

“As this example illustrates, the A/B test uses the logic of an RCT but is focused on small differences.”

In the paper

“The incredible array of software developed modularly (e.g., Apache, arguably the foundation of the Internet) provides evidence for this approach, but there are also epistemological reasons.”

In the paper

“Focusing on modules, building in context, and updating fit embraces these variations and can be codified in computational models and personalization algorithms.”

In the paper

“In terms of statistics, a Bayesian approach might be more appropriate than Fisherian statistics [56] as the Fisher approach is not appropriate with limited data [48].”

What the sources say

• Not checked yetcites [56] Dienes Z. Bayesian versus orthodox statistics: which side are you on? Perspect Psychol Sci. 2011; 6: 274–290. source
• Not checked yetcites [48] Leon AC, Davis LL, Kraemer HC. The role and interpretation of pilot studies in clinical research. J Psychiatr Res. 2011; 45: 626–629. source

In the paper

“These research streams were largely the inspiration for v0.1 of the agile science process and thus are informative to help make the potential of agile science more concrete.”

In the paper

“In a trial of principles, heterogeneity between conditions is minimized with careful comparator selection and the method does not ignore technology changes.”

In the paper

“Individuals in the single concept group were less open to feedback and instead often “stayed the course.””

What the sources say

• Not checked yetcites [58] Dow SP, Klemmer SR. The efficacy of prototyping under time constraints. Design Thinking. Springer; 2011:111–128. source

In the paper

“For example, early-and-often sharing is not well incentivized in the current research enterprise, particularly the products from the generate phase.”

In the paper

“In this paper, we established three products: modules, computational models, and personalization algorithms, as logical targets for supporting more effective and efficient knowledge accumulation for evidence-based practice.”