Agile science: creating useful products for behavior change in the real world

Probably not a substantial claim: may not pass S3 (load-bearing).

The team has conducted a series of simulations to explore different ways of defining an "ambitious but doable" goal based on interactions between the variables in the model.

“For example, in the Fig. 1 dynamical model, the team has conducted a series of simulations to explore different ways of defining an “ambitious but doable” goal based on interactions between the variables in the model.”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based).

We are actively working toward testing the assumptions outlined.

“As our ongoing suite of complementary projects illustrate, we are actively working toward testing the assumptions outlined.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Multiple complementary research projects are underway that each focus on developing a different module for a complex intervention for physical activity.

“Multiple complementary research projects are underway that each focus on developing a different module for a complex intervention for physical activity.”

Probably not a substantial claim: may not pass S3 (load-bearing).

A third complementary project is focused on the development of a tool that can support the rapid creation and early-and-often sharing of behavior change modules.

“A third complementary project is focused on the development of a tool that can support the rapid creation and early-and-often sharing of behavior change modules.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

In partnership with Bob Evans at Google and his open-source Paco system, at the time of writing, we are building a system that allows researchers to rapidly build behavior change modules.

“In partnership with Bob Evans at Google and his open-source Paco system (www.pacoapp.com), at the time of writing, we are building a system that allows researchers to rapidly build behavior change modules.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

Within the ambitious but doable and context cue/implementation intention projects, we previously engaged in formative work, simulation studies, and prototype testing and thus are currently utilizing the operations created previously in the evaluate phase.

“Within the ambitious but doable and context cue/implementation intention projects, we previously engaged in formative work, simulation studies, and prototype testing and thus are currently utilizing the operations created previously in the evaluate phase.”

May combine two claims.

Evidence-based practices in behavioral interventions require both careful examination of intervention efficacy and the ability of the practice to be translated and disseminated for real-world use.

“Evidence-based practices not only require careful examination of the efficacy of an intervention but also the ability of the evidence-based practice to be translated and disseminated for real-world use [2].”

• cites [2] Rotheram-Borus MJ, Swendeman D, Chorpita BF. Disruptive innovations for designing and diffusing evidence-based interventions. Am Psychol. 2012; 67: 463. sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Chorpita et al. argued that the fundamental challenge in behavior change science may be an overemphasis on knowledge proliferation at the cost of knowledge management and suggested focusing more on algorithms to curate knowledge.

“Indeed, Chorpita et al. [20] have argued that the fundamental challenge in behavior change science may be an overemphasis on knowledge proliferation (i.e., the development of new treatments) at the cost of knowledge management (e.g., developing new ways to design, organize, and share existing resources) and thus suggest focusing more on algorithms to curate knowledge.”

• cites [20] Chorpita BF, Rotheram-Borus MJ, Daleiden EL, et al. The old solutions are the new problem how do we better use what we already know about reducing the burden of mental illness? Perspect Psychol Sci. 2011; 6: 493–497. sourceNot checked yet

  Not checked in the free run.

May combine two claims.

Sharing early and often reduces independent groups reinventing similar operations and, if information is well-curated and modules are interoperable, could facilitate increased specialization.

“This reduces independent groups reinventing similar operations and also, if the information is well-curated and modules are interoperable, could facilitate increased specialization.”

Classified as probably not a substantial claim.

Testing of assumptions via MVP modules is achieved via sprints, which involve a rapid cycle of identifying an assumption, building an MVP module to test the assumption, and releasing the module to relevant stakeholders to test the assumption.

“This testing of assumptions via MVP modules is achieved via “sprints,” which involve a rapid cycle of identifying an assumption, building an MVP module to test the assumption, and then releasing the module to relevant stakeholders to test the assumption [23].”

• cites [23] Abrahamsson P, Warsta J, Siponen MT, Ronkainen J. New directions on agile methods: a comparative analysis. Proceedings of the International Conference on Software Engineering, 2003.Not checked yet

  Not checked in the free run.

May combine two claims.

Sharing early and often enables individuals to build on one another's work at the beginning and thus makes it easier for the process to facilitate interoperable and repurposable modules across complex interventions.

“Put differently, sharing early and often enables individuals to build on one another’s work at the beginning and thus makes it easier for the process to facilitate interoperable and repurposable modules across complex interventions.”

Probably not a substantial claim: may not pass S2 (checkable).

Personalization algorithms are a pragmatic target for developing evidence-based recommendations of complex interventions because they establish the evidence base for selecting the right intervention option for a particular person in context.

“As such, the personalization algorithm is a pragmatic target for developing evidence-based recommendations of complex interventions as the personalization algorithms establish the evidence base for selecting the right intervention option for a particular person in context.”

May combine two claims.

Sharing of operations enables different research groups to use the same operations in different use cases, which enables a more robust testing of external validity.

“(3) sharing of operations enables different research groups to use the same operations in different use cases, which enables a more robust testing of external validity.”

May combine two claims.

Behavioral theories have classically been how behavioral sciences describe how interventions, individuals, and context interact but are fraught with well-documented problems such as lack of falsification, lack of information about subtle dynamics, and being so abstract that they are difficult to apply in practice.

“Behavioral theories have classically been how behavioral sciences describe how interventions, individuals, and context interact but these are fraught with well-documented problems such as lack of falsification, lack of information about subtle dynamics, and they are often so abstract that they are difficult to apply in practice [36, 37].”

• cites [36] Hekler EB, Klasnja P, Froehlich JE, Buman MP. Mind the theoretical gap: interpreting, using, and developing behavioral theory in HCI research. Proceedings of SIGCHI Conference on Human Factors in Computing Systems (CHI’13). 2013; 3307–3316. sourceNot checked yet

  Not checked in the free run.

• cites [37] Riley WT, Rivera DE, Atienza AA, et al. Health behavior models in the age of mobile interventions: are our theories up to the task? Translat Behav Med. 2011; 1: 53–71. sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

A plausible reason for the siloing of evidence-based protocols is that operations are shared at the wrong time.

“A plausible reason for this siloing of evidence-based protocols is simply that we are sharing our operations at the wrong time.”

Classified as probably not a substantial claim.

Evidence-based practices often become their own separate silos of evidence, rendering it very difficult to mix and match modules across evidence-based practices.

“It has been argued elsewhere that evidence-based practices often become their own separate silos of evidence, rendering it very difficult to mix and match modules across evidence-based practices [2].”

• cites [2] Rotheram-Borus MJ, Swendeman D, Chorpita BF. Disruptive innovations for designing and diffusing evidence-based interventions. Am Psychol. 2012; 67: 463. sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

Together, the three agile science products each uniquely contribute to supporting evidence-based decision making about which intervention to use for whom and at what time.

“Together, these three agile science products each uniquely contribute to supporting evidence-based decision making about which intervention to use for whom and at what time.”

Probably not a substantial claim: may not pass S2 (checkable).

Striving for small modules enables repurposability.

“In brief, striving for small modules enables repurposability.”

Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

It is likely that a researcher engages in formative work, simulation studies, and even prototyping testing iteratively and/or simultaneously.

“It is likely that a researcher engages in formative work, simulation studies, and even prototyping testing iteratively and/or simultaneously.”

Probably not a substantial claim: may not pass S3 (load-bearing).

MVPs are simplified modules designed to test assumptions about the module with relevant stakeholders.

“MVPs are the simplified modules that are designed to test assumptions about the module with relevant stakeholders [25].”

• cites [25] Ries E. The lean startup: how today’s entrepreneurs use continuous innovation to create radically successful businesses. Random House LLC; 2011.Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S2 (checkable).

Concrete examples, particularly multiple examples, provide richer information about a construct than a textual definition alone.

“In terms of construct specification, concrete examples, particularly multiple examples, provide richer information about a construct than a textual definition alone.”

Probably not a substantial claim: may not pass S2 (checkable).

Careful delineation of inputs, processes, and outputs makes it easier to draw connections between behavior change modules and thus the architecture on how modules interact.

“Careful delineation of inputs, processes, and outputs makes it easier to draw connections between behavior change modules and thus the architecture on how modules interact (including modules for measuring attributes of the person/population, setting, or outcome measure but,”

Probably not a substantial claim: may not pass S3 (load-bearing).

Within software development, the basic structure of modules is that they have specific inputs, a process that utilizes these inputs to transform them, and specific outputs.

“Within software development, the basic structure of modules is that they have specific inputs, a process that utilizes these inputs to transform them in some way and then this process creates specific outputs [24].”

• cites [24] Garlan D, Shaw M. An introduction to software architecture. Advances in software engineering and knowledge engineering. 1993; 1. sourceNot checked yet

  Not checked in the free run.

Classified as probably not a substantial claim.

The disruptive innovation process emphasizes providing simpler and less expensive alternatives to current practices that meet the essential need for the majority of users to make them more accessible, scalable, and replicable.

“The disruptive innovation process emphasizes the focus on providing simpler and less expensive alternatives to current practices that meet the essential need for the majority of users in order to make them more easily accessible, scalable, and replicable [2].”

• cites [2] Rotheram-Borus MJ, Swendeman D, Chorpita BF. Disruptive innovations for designing and diffusing evidence-based interventions. Am Psychol. 2012; 67: 463. sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S2 (checkable).

There are three behavior change intervention product types that are particularly important for targeting as conceptual building blocks for personalized complex interventions: behavior change intervention modules, computational models, and personalization algorithms.

“With the value of sharing operations specified, there are three behavior change intervention product types that we view as particularly important for targeting as they are the conceptual building blocks for personalized complex interventions: behavior change intervention modules, computational models, and personalization algorithms.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The micro-randomized trial is a sequential factorial design for modeling effectiveness of treatment components over time.

“The micro-randomized trial is a sequential factorial design for modeling effectiveness of treatment components over time.”

• cites [16] Klasnja P, Hekler EB, Shiffman S, et al. Micro-randomized trials: an experimental design for developing just-in-time adaptive interventions. Health Psychol. 2016. sourceNot checked yet

  Not checked in the free run.

• cites [17] Martin CA, Desphande S, Hekler EB, Rivera DE. A system identification approach for improving behavioral interventions based on social cognitive theory. American Control Conference (ACC). 2015; 5878–5883. doi: 10.1109/ACC.2015.7172261 . sourceNot checked yet

  Not checked in the free run.

• cites [18] Martin CA, Rivera DE, Hekler EB. An identification test monitoring procedure for MIMO systems based on statistical uncertainty estimation. IEEE Conference on Decision and Control (CDC); 2015. sourceNot checked yet

  Not checked in the free run.

Classified as probably not a substantial claim.

Products produced via the generate phase are not evidence-based.

“By contender, we explicitly acknowledge that the products produced via the generate phase are not evidence-based.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Riley et al. utilized simulation to generate a computational model of social cognitive theory.

“As discussed earlier, Riley et al. utilized simulation to generate a computational model of social cognitive theory [39].”

• cites [39] Riley WT, Martin CA, Rivera DE, et al. The development of a control systems model of social cognitive theory. Translat Behav Med. 2016.Not checked yet

  Not checked in the free run.

Classified as probably not a substantial claim.

Good external validity requires variations in who uses an intervention, how the intervention might be operationalized, the outcomes targeted, and the settings in which the intervention is used.

“As this example illustrates, good external validity requires variations in who uses an intervention, how the intervention might be operationalized, the outcomes the intervention is being targeted to influence, and the settings in which the intervention is used.”

• cites [28] Shadish WR, Cook TD, Campbell DT. Experimental and quasi-experimental designs for generalized causal inference. Wadsworth Cengage learning; 2002.Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S2 (checkable). May combine two claims.

Algorithms for supporting decisions about matching interventions and modules with specific individuals in context are a valuable target product and could be useful for curation.

“(3) algorithms for supporting decisions about matching interventions and modules with specific individuals in context is a second valuable target product and could be useful for curation.”

Probably not a substantial claim: may not pass S2 (checkable).

Greater value should be placed on concrete operations, even if they have not been empirically validated.

“Greater value should be placed on concrete operations, even if they have not been empirically validated because:”

May combine two claims.

Model predictive control, recommender systems, agent-based modeling, or Bayesian network analysis are plausible personalization algorithm strategies.

“Model predictive control [40–43], recommender systems [44], agent-based modeling [45], or Bayesian network analysis [46] are all plausible personalization algorithm strategies.”

• cites [40] Dong Y, Rivera DE, Downs DS, et al. Hybrid model predictive control for optimizing gestational weight gain behavioral interventions. American Control Conference (ACC). 2013; 1970–1975. sourceNot checked yet

  Not checked in the free run.

• cites [41] Nandola NN, Rivera DE. An improved formulation of hybrid model predictive control with application to production-inventory systems. IEEE Trans Control Syst Technol. 2013; 1: 121–135. sourceNot checked yet

  Not checked in the free run.

• cites [42] Deshpande S, Rivera DE, Younger JW, Nandola NN. A control systems engineering approach for adaptive behavioral interventions: illustration with a fibromyalgia intervention. Translat Behav Med. 2014; 4: 275–289. Errratum in 274(273), p439. sourceNot checked yet

  Not checked in the free run.

• cites [43] Martin CA, Rivera DE, Hekler EB. A decision framework for an adaptive behavioral intervention for physical activity using hybrid model predictive control. American Control Conference (ACC). 2016. sourceNot checked yet

  Not checked in the free run.

• cites [44] Resnick P, Varian HR. Recommender systems. Commun ACM. 1997; 40: 56–58. sourceNot checked yet

  Not checked in the free run.

• cites [45] Gilbert N. Agent-based models. Sage; 2008. sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

The Dropbox founders tested their assumption about demand for a file-syncing service with a video rather than building the technology, demonstrating a resource-efficient feasibility test.

“For example, the founders of Dropbox made the assumption that there would be demand for a service that makes it easy to sync and share files. To test this assumption, they could have built the technology but that would have required considerable resources. Instead, they tested their assumption with a video (http://techcrunch.com/2011/10/19/dropbox-minimal-viable-product/). The video was a test on”

May combine two claims.

The micro-randomized trial can be used to test proximal main effects of components over time, time-varying moderation, and, with variations in study design based on system identification, the development of idiographic computational models.

“It can be used to test proximal main effects of components over time, time-varying moderation [16], and with variations in study design based on strategies from system identification, the development of idiographic computational models [17, 18].”

• cites [16] Klasnja P, Hekler EB, Shiffman S, et al. Micro-randomized trials: an experimental design for developing just-in-time adaptive interventions. Health Psychol. 2016. sourceNot checked yet

  Not checked in the free run.

• cites [17] Martin CA, Desphande S, Hekler EB, Rivera DE. A system identification approach for improving behavioral interventions based on social cognitive theory. American Control Conference (ACC). 2015; 5878–5883. doi: 10.1109/ACC.2015.7172261 . sourceNot checked yet

  Not checked in the free run.

• cites [18] Martin CA, Rivera DE, Hekler EB. An identification test monitoring procedure for MIMO systems based on statistical uncertainty estimation. IEEE Conference on Decision and Control (CDC); 2015. sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

Within the behavioral sciences, reliable effect size estimates for constructs are more important to support evidence-based decisions.

“Within the behavioral sciences, reliable effect size estimates for constructs are more important to support evidence-based decisions [1].”

• cites [1] Kelly M, Morgan A, Ellis S, et al. Evidence based public health: a review of the experience of the National Institute of Health and Clinical Excellence (NICE) of developing public health guidance in England. Soc Sci Med. 2010; 71: 1056–1062. sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S2 (checkable), S4 (based). May combine two claims.

While complex interventions might be desired in practice, modules of complex interventions are plausibly more valuable for science.

“(2) while complex interventions might be desired in practice, modules of complex interventions are plausibly more valuable for science;”

Probably not a substantial claim: may not pass S3 (load-bearing).

Formative work methods (human-centered design, community-based participatory research, review of previous knowledge) are critical for a wide range of evidence-based practice recommendations related to the research enterprise.

“These methods are acknowledged as critical for a wide range of evidence-based practice recommendations related to the research enterprise [5].”

• cites [5] Craig P, Dieppe P, Macintyre S, et al. Developing and evaluating complex interventions: the new medical research council guidance. Int J Nurs Stud. 2013; 50: 587–592. sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S4 (based). May combine two claims.

HCI processes conform to the generate phase and behavioral science processes roughly conform to the evaluate phase, with overlap.

“we see these two as complementary with, roughly, HCI processes conforming to the generate phase and behavioral science processes roughly conforming to the evaluate phase, though obviously there is overlap.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

An advantage of software module structure is that the module is built to be self-contained and thus repurposable for any use case with the requisite inputs and desired output.

“It is essential to note that while modules are often built initially for a specific use case, an advantage of software module structure is that the module is built to be self-contained and thus repurposable for any use case with the requisite inputs (e.g., personal identification information, permission) and desires the requisite output (e.g., a secure bridge to share data across systems).”

Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

As something is learned with one action, that information is fed into the others until contender operations and constructs emerge that show promise for being useful in the real-world.

“As something is learned with one action, that information is fed into the others until contender operations and constructs emerge that show promise for being useful in the real-world.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Extreme programming (XP) emphasizes well-specified modules that are shared early-and-often with relevant stakeholders when building complex multi-component systems.

“For example, extreme programming (XP), which is one agile method [23], emphasizes well-specified modules that are shared early-and-often with relevant stakeholders when building complex multi-component systems.”

• cites [23] Abrahamsson P, Warsta J, Siponen MT, Ronkainen J. New directions on agile methods: a comparative analysis. Proceedings of the International Conference on Software Engineering, 2003.Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

Operations are highly valuable for better specifying constructs.

“(1) operations are highly valuable for better specifying constructs;”

Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

Modules, computational models, and personalization algorithms provide the evidence-based "building blocks" for generating practice recommendations for combining modules into complex interventions for unique use cases.

“Modules, computational models, and personalization algorithms provide the evidence-based “building blocks” for generating practice recommendations for combining modules into complex interventions for unique use cases.”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based). May combine two claims.

This is easier with software but can be difficult with other modalities.

“This is easier with software but can be difficult with other modalities.”

Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

Early-and-often sharing could very likely improve the scientific community's ability to create efficient accumulation of knowledge about behavior change.

“That said, it could very likely improve the scientific community’s ability to create efficient accumulation of knowledge about behavior change.”

Probably not a substantial claim: may not pass S2 (checkable), S4 (based). May combine two claims.

Sharing contender operations and constructs and also failed tests of these contenders from early work has great value for supporting better construct specification, improved efficiency in the scientific process, supporting earlier tests of feasibility, and could plausibly support better collection of data related to external validity and, by extension, support recommender systems for personal化tion

“Further, sharing contender operations and constructs and also failed tests of these contenders from early work has great value for supporting better construct specification, improved efficiency in the scientific process, supporting earlier tests of feasibility, and could plausibly support better collection of data related to external validity and, by extension, support recommender systems for”

Probably not a substantial claim: may not pass S1 (asserted), S3 (load-bearing).

Other recommendations suggest a clear distinction between testing feasibility of the operations versus research methods.

“Other recommendations suggest a clear distinction between testing feasibility of the operations versus research methods [54].”

• cites [54] Arain M, Campbell MJ, Cooper CL, Lancaster GA. What is a pilot or feasibility study? A review of current practice and editorial policy. BMC Med Res Methodol. 2010; 10: 67. sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

Sequential n-of-1 trials could serve as a more efficient feasibility testing method by allowing for intervention adjustments after each series of trials.

“For example, sequential n-of-1 trials could serve as a more efficient feasibility testing method by allowing for intervention adjustments after each series of trials [55].”

• cites [55] Dallery J, Raiff BR. Optimizing behavioral health interventions with single-case designs: from development to dissemination. Translat Behav Med. 2014; 4: 1–14. sourceNot checked yet

  Not checked in the free run.

May combine two claims.

We see at least three use cases for RCTs of complex interventions: quality assurance, A/B style test, and trials of principles.

“We see at least three use cases: (1) quality assurance of a complex intervention that will be released “as is” to a large population; (2) as an “A/B” style test that is often used in the tech industry; and (3) as used in the “trials of principles” [51].”

• cites [51] Mohr DC, Schueller SM, Riley WT, et al. Trials of intervention principles: evaluation methods for evolving behavioral intervention technologies. J Med Internet Res. 2015; 17. sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

Bond et al. conducted a 61 million person A/B-style experiment on Facebook to explore how message framing might influence voting behavior.

“For example, Bond et al. conducted a 61 million person A/B-style experiment on Facebook to explore how message framing might influence voting behavior [52].”

• cites [52] Bond RM, Fariss CJ, Jones JJ, et al. A 61-million-person experiment in social influence and political mobilization. Nature. 2012; 489: 295–298. sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

There are important cognitive biases that likely come into play that need to be better acknowledged in the generate phase.

“With regard to idea generation and vetting, there are important cognitive biases that likely come into play that need to be better acknowledged in the generate phase.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The two versions in A/B testing represent different hypotheses on how the system might function and are often only subtly different.

“The two versions represent different hypotheses on how the system might function and are often only subtly different.”

Probably not a substantial claim: may not pass S3 (load-bearing).

It is quite plausible that the current reward system for academics, particularly the currency (i.e., publications and grants), may be resulting in the unintended consequence of incentivizing the creation of only minimally useful products for real-world use.

“As suggested by Ioannidis, it is quite plausible that the current reward system for academics, particularly the currency (i.e., publications and grants), may be resulting in the unintended consequence of incentivizing the creation of only minimally useful products for real-world use.”

• cites [60] Ioannidis JP. How to make more published research true. PLoS Med/ Public Libr Sci. 2014; 11, e1001747.Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

Behavior change modules, computational models, and personalization algorithms are the building blocks for creating complex interventions and thus are logical scientific targets.

“In particular, behavior change modules, computational models, and personalization algorithms are the building blocks for creating complex interventions and thus are logical scientific targets.”

Probably not a substantial claim: may not pass S3 (load-bearing).

A sister project headed by coauthor Klasnja is focused on developing modules that support physical activity maintenance.

“A sister project headed by coauthor Klasnja, is focused on developing modules that support physical activity maintenance [16].”

• cites [16] Klasnja P, Hekler EB, Shiffman S, et al. Micro-randomized trials: an experimental design for developing just-in-time adaptive interventions. Health Psychol. 2016. sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

Micro-randomization trials are very efficient and thus could also be used after more classic n-of-1 trials have provided enough confidence to warrant investment in the design of one.

“From a human-participants perspective, micro-randomization trials are very efficient and thus could also be used after more classic n-of-1 trials have provided enough confidence to warrant investment in the design of one [16].”

• cites [16] Klasnja P, Hekler EB, Shiffman S, et al. Micro-randomized trials: an experimental design for developing just-in-time adaptive interventions. Health Psychol. 2016. sourceNot checked yet

  Not checked in the free run.

May combine two claims.

The four-phase model ignores variations of units, treatments, outcomes, and settings until the end of the process, but it is highly feasible that when, where, for whom, and in what state of the individual can all moderate when an intervention will produce an effect.

“As such, the four-phase model ignores variations of units, treatments, outcomes, and settings until the end of the process but it is highly feasible that when, where, for whom, and in what state of the individual can all moderate when an intervention will produce an effect [9].”

• cites [9] Rothman AJ. Exploring connections between moderators and mediators: commentary on subgroup analyses in intervention research. Prev Sci. 2013; 14: 189–192. sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The major limitations are intervention specification to support replication, which is ideal for all sciences and effective curation as already pointed out by others.

“The major limitations are intervention specification to support replication, which is ideal for all sciences and effective curation as already pointed out by others [20].”

• cites [20] Chorpita BF, Rotheram-Borus MJ, Daleiden EL, et al. The old solutions are the new problem how do we better use what we already know about reducing the burden of mental illness? Perspect Psychol Sci. 2011; 6: 493–497. sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S2 (checkable), S4 (based). May combine two claims.

As with any scientific process, the essential requirement is replication with sound definitions, robust versioning control, and a process of rapidly sharing and curating that information.

“As with any scientific process, the essential requirement is replication with sound definitions, robust versioning control, and a process of rapidly sharing and curating that information.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The A/B test is more akin to the optimization methods discussed earlier.

“As such, it is more akin to the optimization methods discussed earlier.”

Probably not a substantial claim: may not pass S3 (load-bearing).

In the digital technology industry, there is a common practice of comparing different versions (often two, hence A/B) of a given system.

“In the digital technology industry, there is a common practice of comparing different versions (often two, hence A/B) of a given system.”

• cites [36] Hekler EB, Klasnja P, Froehlich JE, Buman MP. Mind the theoretical gap: interpreting, using, and developing behavioral theory in HCI research. Proceedings of SIGCHI Conference on Human Factors in Computing Systems (CHI’13). 2013; 3307–3316. sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

The trial of principles fits well with agile science within the evaluate phase.

“As such, it fits well with agile science within the evaluate phase.”

Probably not a substantial claim: may not pass S2 (checkable).

RCTs of complex interventions are the most logical method for establishing that a specific complex intervention will work for a specific target population.

“RCTs of complex interventions are the most logical method for establishing that a specific complex intervention will work for a specific target population.”

May combine two claims.

The stay the course observation is similar to the sunk cost bias, which suggests an individual will pursue a less desirable option even if a more desirable option becomes available because of the resources already invested.

“This stay the course observation is similar to the sunk cost bias [59], which is a well-known cognitive bias that suggests an individual will pursue a less desirable option even if a more desirable option becomes available because of the resources already invested.”

• cites [59] Arkes HR, Ayton P. The sunk cost and Concorde effects: are humans less rational than lower animals? Psychol Bull. 1999; 125: 591. sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S2 (checkable). May combine two claims.

More rapid and parallel development could support both better defining of a construct and minimize the impact of the sunk cost bias, thus further reinforcing the value of early-and-often sharing and developing multiple operations.

“More rapid and parallel development could support both better defining of a construct and minimize the impact of the sunk cost bias, thus further reinforcing the value of early-and-often sharing and developing multiple operations.”

May combine two claims.

In behavioral intervention technologies, it is common for software to change over time but the general principle (e.g., using goal setting as a technique) does not.

“The third use case for RCTs is the trials of principles, which emphasizes that in behavioral intervention technologies, it is common for a software to change over time but the general principle (e.g., using goal setting as a technique) does not.”

• cites [51] Mohr DC, Schueller SM, Riley WT, et al. Trials of intervention principles: evaluation methods for evolving behavioral intervention technologies. J Med Internet Res. 2015; 17. sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Careful use of priors within studies of novel systems can be used to help mitigate problems with magnitude effect errors and thus could be a more appropriate statistical method for the generate phase.

“As demonstrated elsewhere, careful use of priors within studies of novel systems can be used to help mitigate problems with magnitude effect errors and thus could be a more appropriate statistical method for the generate phase [47].”

• cites [47] Kaye M, Nelson GL, Hekler EB. Researcher-centered design of statistics: why Bayesian statistics better fit the culture and incentives of HCI. Proceedings of SIGCHI Conference on Human Factors in Computing Systems (CHI’16). San Jose, CA: ACM; 2016. sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

Previous work suggests early studies often test feasibility of methods (e.g., recruitment, randomization, retention, and assessment), not hypotheses.

“Previous work suggests early studies often test feasibility of methods (e.g., recruitment, randomization, retention, and assessment), not hypotheses [48].”

• cites [48] Leon AC, Davis LL, Kraemer HC. The role and interpretation of pilot studies in clinical research. J Psychiatr Res. 2011; 45: 626–629. sourceNot checked yet

  Not checked in the free run.

May combine two claims.

Individuals randomized to work on multiple ideas at once (parallel group) had greater openness to incorporating feedback and also produced better end products compared to the single iterative concept group.

“Results found that individuals randomized to work on multiple ideas at once (i.e., parallel group) had greater openness to incorporating feedback and also produced better end products compared to the single iterative concept group.”

• cites [58] Dow SP, Klemmer SR. The efficacy of prototyping under time constraints. Design Thinking. Springer; 2011:111–128. sourceNot checked yet

  Not checked in the free run.

May combine two claims.

Dow and colleagues showed the possibility of developing better results through iteration, and a subsequent study examined the differences between parallel and iterative development versus serial and iterative development.

“For example, Dow and colleagues showed the possibility of developing better results through iteration [57] and a subsequent study examined the differences between parallel and iterative development versus serial and iterative development [58].”

• cites [57] Dow SP, Glassco A, Kass J, et al. Parallel prototyping leads to better design results, more divergence, and increased self-efficacy. Transactions of Computer-Human Interactions (ToCHI). 2010; 17: 18.Not checked yet

  Not checked in the free run.

• cites [58] Dow SP, Klemmer SR. The efficacy of prototyping under time constraints. Design Thinking. Springer; 2011:111–128. sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing). May combine two claims.

It is quite plausible that the same agile science process can work for other intervention modalities (e.g., face-to-face, group) and other types of complex problems (e.g., precision medications, public policy initiatives).

“We believe it is quite plausible that the same process can work for other intervention modalities (e.g., face-to-face, group) and other types of complex problems (e.g., precision medications, public policy initiatives).”

May combine two claims.

The A/B test uses the logic of an RCT but is focused on small differences.

“As this example illustrates, the A/B test uses the logic of an RCT but is focused on small differences.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

The incredible array of software developed modularly (e.g., Apache) provides evidence for the modular approach to behavior change.

“The incredible array of software developed modularly (e.g., Apache, arguably the foundation of the Internet) provides evidence for this approach, but there are also epistemological reasons.”

May combine two claims.

Focusing on modules, building in context, and updating fit embraces these variations and can be codified in computational models and personalization algorithms.

“Focusing on modules, building in context, and updating fit embraces these variations and can be codified in computational models and personalization algorithms.”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing). May combine two claims.

A Bayesian approach might be more appropriate than Fisherian statistics as the Fisher approach is not appropriate with limited data.

“In terms of statistics, a Bayesian approach might be more appropriate than Fisherian statistics [56] as the Fisher approach is not appropriate with limited data [48].”

• cites [56] Dienes Z. Bayesian versus orthodox statistics: which side are you on? Perspect Psychol Sci. 2011; 6: 274–290. sourceNot checked yet

  Not checked in the free run.

• cites [48] Leon AC, Davis LL, Kraemer HC. The role and interpretation of pilot studies in clinical research. J Psychiatr Res. 2011; 45: 626–629. sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based). May combine two claims.

These research streams were largely the inspiration for v0.1 of the agile science process and thus are informative to help make the potential of agile science more concrete.

“These research streams were largely the inspiration for v0.1 of the agile science process and thus are informative to help make the potential of agile science more concrete.”

Probably not a substantial claim: may not pass S4 (based). May combine two claims.

In a trial of principles, heterogeneity between conditions is minimized with careful comparator selection and the method does not ignore technology changes.

“In a trial of principles, heterogeneity between conditions is minimized with careful comparator selection and the method does not ignore technology changes.”

May combine two claims.

Individuals in the single concept group were less open to feedback and instead often "stayed the course."

“Individuals in the single concept group were less open to feedback and instead often “stayed the course.””

• cites [58] Dow SP, Klemmer SR. The efficacy of prototyping under time constraints. Design Thinking. Springer; 2011:111–128. sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based).

Early-and-often sharing is not well incentivized in the current research enterprise, particularly the products from the generate phase.

“For example, early-and-often sharing is not well incentivized in the current research enterprise, particularly the products from the generate phase.”

Probably not a substantial claim: may not pass S4 (based).

In this paper, we established three products: modules, computational models, and personalization algorithms, as logical targets for supporting more effective and efficient knowledge accumulation for evidence-based practice.

“In this paper, we established three products: modules, computational models, and personalization algorithms, as logical targets for supporting more effective and efficient knowledge accumulation for evidence-based practice.”