Appropriate controls for digital therapeutic clinical trials: A narrative review of control conditions in clinical trials of digital therapeutics (DTx) deploying psychosocial, cognitive, or behavioral content

Classified as probably not a substantial claim.

The review of RCTs reveals room for improvement in describing and naming control conditions more consistently.

“Our review of RCTs reveals room for improvement in describing and naming control conditions more consistently.”

• AssumptionRests on an assumption: The 14 reviewed RCTs are representative of DTx RCTs deploying psychosocial, cognitive, or behavioral content.

Probably not a substantial claim: may not pass S3 (load-bearing).

Fourteen RCTs were reviewed.

“Fourteen RCTs were reviewed (Table 1 in the Supplementary Material).”

• AssumptionRests on an assumption: The 14 reviewed RCTs are representative of DTx RCTs deploying psychosocial, cognitive, or behavioral content.

May combine two claims.

The Endeavor trial found that the treatment separated from the control on the primary, objective measure of attention, but not on clinical measures related to ADHD subjective symptoms.

“The study found that the treatment separated from the control on the primary, objective measure of attention, but not on clinical measures related to ADHD subjective symptoms (27).”

• cites [27] Kollins, 2020, A novel digital intervention for actively reducing severity of paediatric ADHD (STARS-ADHD): a randomised controlled trial, Lancet Digit Heal sourceNot checked yet

  Not checked in the free run.

• AssumptionRests on an assumption: The 14 reviewed RCTs are representative of DTx RCTs deploying psychosocial, cognitive, or behavioral content.

May combine two claims.

Sham controls either replaced core treatment activities with "fake, but plausible therapies" or delivered general, disease agnostic wellbeing tips.

“Sham controls either replaced core treatment activities with “fake, but plausible therapies” (33) or they delivered general, disease agnostic wellbeing tips (32, 56).”

• cites [33] Espie, 2012, A randomized, placebo-controlled trial of online cognitive behavioral therapy for chronic insomnia disorder delivered via an automated Media-rich web application, Sleep sourceNot checked yet

  Not checked in the free run.

• cites [32] Boucher, 2020, Immediate and long-term effects of an 8-week digital mental health intervention on adults with poorly managed type 2 diabetes: protocol for a randomized controlled trial, JMIR Res Protoc sourceNot checked yet

  Not checked in the free run.

• cites [56] Christensen, 2016, Effectiveness of an online insomnia program (SHUTi) for prevention of depressive episodes (the GoodNight Study): a randomised controlled trial, The Lancet Psychiatry sourceNot checked yet

  Not checked in the free run.

• AssumptionRests on an assumption: The 14 reviewed RCTs are representative of DTx RCTs deploying psychosocial, cognitive, or behavioral content.

May combine two claims.

All reviewed DTx containing gamified cognitive training or remediation interventions chose a form of sham control, with some sham controls looking similar to consumer grade apps or video games.

“It is noteworthy that all reviewed DTx containing gamified cognitive training or remediation interventions chose a form of sham control, with some sham controls looking similar to consumer grade apps or video games.”

• AssumptionRests on an assumption: The 14 reviewed RCTs are representative of DTx RCTs deploying psychosocial, cognitive, or behavioral content.

Probably not a substantial claim: may not pass S3 (load-bearing).

Another strategy was to "disarm" certain content, which means removing key aspects hypothesized to drive efficacy.

“Another strategy was to “disarm” certain content, which means removing key aspects hypothesized to drive efficacy.”

• AssumptionRests on an assumption: The 14 reviewed RCTs are representative of DTx RCTs deploying psychosocial, cognitive, or behavioral content.

May combine two claims.

Only one trial deployed an active comparator, and only as part of a 3-arm RCT including a TAU control arm for the main comparison.

“Only one trial deployed an active comparator and only as part of a 3-arm RCT including a TAU control arm for the main comparison.”

• AssumptionRests on an assumption: The 14 reviewed RCTs are representative of DTx RCTs deploying psychosocial, cognitive, or behavioral content.

May combine two claims.

TAU was the main comparator arm in a 3-arm RCT of iCBT for Irritable Bowel Syndrome, the data from which was used to support the FDA clearance of Parallel.

“TAU was the main comparator arm in a 3-arm RCT of iCBT for Irritable Bowel Syndrome (IBS), the data from which was used to support the FDA clearance of Parallel (10).”

• cites [10] Everitt, 2015, Assessing cognitive behavioural therapy in irritable bowel (ACTIB): protocol for a randomised controlled trial of clinical-effectiveness and cost-effectiveness of therapist delivered cognitive behavioural therapy and web-based self-management in irritable, BMJ Open sourceNot checked yet

  Not checked in the free run.

• AssumptionRests on an assumption: The 14 reviewed RCTs are representative of DTx RCTs deploying psychosocial, cognitive, or behavioral content.

Classified as probably not a substantial claim.

Authors did not regularly discuss control design choices such as why certain features were removed or disarmed.

“Authors did not regularly discuss control design choices such as why certain features were removed or disarmed.”

May combine two claims.

A wellbeing program (The Health and Well-Being Program) included as a control condition for obsessive compulsive disorder was described as an "active intervention", while similar "general health and well-being educational content" was called a placebo control for a RCT for diabetes mellitus, type 2.

“For example, a wellbeing program (The Health and Well-Being Program) included as a control condition for obsessive compulsive disorder was described as an “active intervention” (61), while similar “general health and well-being educational content” was called a placebo control for a RCT for diabetes mellitus, type 2 (32).”

• cites [61]Not checked yet

  Not checked in the free run.

• cites [32] Boucher, 2020, Immediate and long-term effects of an 8-week digital mental health intervention on adults with poorly managed type 2 diabetes: protocol for a randomized controlled trial, JMIR Res Protoc sourceNot checked yet

  Not checked in the free run.

May combine two claims.

None of the reviewed trials report blinding checks, which are recommended by trial reporting standards (CONSORT).

“Surprisingly, none of the trials report blinding checks, which are recommended by trial reporting standards (Consolidated Standards Of Reporting Trials, CONSORT (57)).”

• cites [57] Altman, 2001, The revised CONSORT statement for reporting randomized trials: explanation and elaboration, Artic Ann Intern Med sourceNot checked yet

  Not checked in the free run.

• AssumptionRests on an assumption: The 14 reviewed RCTs are representative of DTx RCTs deploying psychosocial, cognitive, or behavioral content.

May combine two claims.

In the schizophrenia DTx trial, the digital control improved some of the secondary outcomes and did not perform significantly worse than the investigational treatment.

“it is notable that the digital control improved some of the secondary outcomes and did not perform significantly worse than the investigational treatment (59).”

• cites [59]Not checked yet

  Not checked in the free run.

• AssumptionRests on an assumption: The 14 reviewed RCTs are representative of DTx RCTs deploying psychosocial, cognitive, or behavioral content.

May combine two claims.

A unified approach to designing appropriate control conditions for DTx RCTs has been a challenge because establishing control conditions for psychosocial interventions that build the core of many DTx is challenging.

“and (2) challenges in establishing control conditions for psychosocial interventions that build the core of many DTx.”

May combine two claims.

The above examples highlight the complexity of designing control conditions for DTx and the challenge of designing inactive control conditions while keeping enough credible content for participant blinding.

“The above examples highlight the complexity of designing control conditions for DTx as well as the challenge of designing inactive control conditions while keeping enough credible content for participants blinding.”

May combine two claims.

A control condition that establishes a structured and supportive therapeutic alliance already contains major efficacy factors of psychotherapy and will therefore not be inactive.

“Thus, a control condition which establishes structured and supportive therapeutic alliance already contains major efficacy factors of psychotherapy and will therefore not be inactive.”

May combine two claims.

In the SHUTi trial, the investigational treatment and control condition differ on at least core CBT content and sleep time suggestions, and the difference between them might be based on core CBT content, the concrete sleep time suggestions, or both.

“Therefore the investigational treatment and control condition differ on at least these two aspects, and thus the difference between them might be based on core CBT content, the concrete sleep time suggestions, or both (56).”

• cites [56] Christensen, 2016, Effectiveness of an online insomnia program (SHUTi) for prevention of depressive episodes (the GoodNight Study): a randomised controlled trial, The Lancet Psychiatry sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

The use of the term "active control" for off-the-shelf games in the Posit Science trial is likely applying a more academic nomenclature of active controls that is not in line with the FDA definition where active controls are interventions with known therapeutic effects.

“The use of active control here is likely applying a more academic nomenclature of active controls which is not in line with the FDA definition where active controls are interventions with known therapeutic effects (42).”

• cites [42] 2013Not checked yet

  Not checked in the free run.

May combine two claims.

Trying to design placebo therapies with low or no active ingredients often results in therapies with low face-validity and the risk of unblinding participants to study assignment.

“On the flip side, trying to design placebo therapies with low or no active ingredients often result in therapies with low face-validity and the risk to unblind all or at least some participants to study assignment (22).”

• cites [22] Enck, 2019, Placebo effects in psychotherapy: a framework, Front Psychiatry sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

By analyzing which aspects of digital interventions were changed or removed in a digital control, one can draw conclusions about which features were considered to have the highest likelihood for being efficacious in the DTx.

“It is interesting to note that by analyzing which aspects of digital interventions were changed or removed in a digital control, we can draw conclusions about which features were considered to have the highest likelihood for being efficacious in the DTx.”

May combine two claims.

Studies employing waitlist or no intervention control groups are not blinded, and thus do not control for the expectation of benefit.

“However, studies employing these control groups are not blinded, and thus do not control for the expectation of benefit.”

Probably not a substantial claim: may not pass S3 (load-bearing).

RCTs are considered the "gold standard" to evaluate therapeutic interventions.

“RCTs are considered the “gold standard” to evaluate therapeutic interventions (13).”

• cites [13] D’Agostino, 1995, Measuring effectiveness: what to expect without a randomized control group, Med CareNot checked yet

  Not checked in the free run.

May combine two claims.

Challenges of selecting appropriate control conditions for DTx stem from the breadth of underlying treatment principles and mechanisms, known challenges in designing placebo controls for psychological treatments, and the fact that no physical device can aid in blinding participants.

“In summary, challenges of selecting appropriate control conditions for DTx, stem from the breadth of underlying treatment principles and mechanisms, sometimes interacting in a complex nature, known challenges in designing placebo controls for psychological treatments, and the fact that no physical device can aid in the blinding of participants.”

• AssumptionRests on an assumption: In software-only SaMD interventions, no physical device can aid in blinding participants.

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

FDA guidelines applicable for SaMD follow typical medical device settings, which usually contain a hardware component, without concrete recommendations for control conditions in software-only SaMD interventions.

“Taken together, the FDA guidelines applicable for SaMD follow typical medical device settings, which usually contain a hardware component, without concrete recommendations for control conditions in software only SaMD interventions.”

Probably not a substantial claim: may not pass S2 (checkable).

It is unlikely that there will ever be a one-size-fits-all approach to control condition design for DTx.

“Given the breadth of DTx content and applications, it is unlikely that there will ever be a one-size-fits-all approach to control condition design.”

Classified as probably not a substantial claim. May combine two claims.

The effect in the schizophrenia DTx trial may be due to at least partially ineffective blinding in the group receiving the minimal control condition or to regular engagement with a structured digital experience resembling behavioral activation.

“This effect may be due to at least partially ineffective blinding in the group receiving the minimal control condition or they could be related to regular engagement with a structured digital experience, which resembles specific components discussed in psychotherapy, such as behavioral activation, discussed above.”

May combine two claims.

Real-world and pragmatic study approaches are gaining popularity, and have been recommended to support regulatory decisions (e.g. The 21st Century Cures Act).

“Indeed, real-world and pragmatic study approaches are gaining popularity, and have been recommended to support regulatory decisions (e.g. The 21st Century Cures Act) (65).”

• cites [65]Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

The minimal control level applies to drug trials as much as to DTx.

“This applies to drug trials as much as to DTx.”

May combine two claims.

Based on results from research into the placebo effect, such trial specifications may bias RCTs in different ways and affect comparability between trials.

“Based on results from research into the placebo effect, such trial specifications may bias RCTs in different ways (62) and affect comparability between trials.”

• cites [62] Kaptchuk, 2001, The double-blind, randomized, placebo-controlled trial: gold standard or golden calf?, J Clin Epidemiol sourceNot checked yet

  Not checked in the free run.

May combine two claims.

The names used for control conditions may not accurately describe the actual control designs, which limits the comparability of results across studies and even across fields.

“Thus, the names used for control conditions may not accurately describe the actual control designs, which limits the comparability of results across studies and even across fields.”

Probably not a substantial claim: may not pass S3 (load-bearing).

In light of the iterative nature of software development, there has been a call for more innovative real-world data approaches to evidence generation for DTx beyond classical RCTs.

“Indeed, in light of the iterative nature of software development, there has been a call for more innovative real-world data approaches to evidence generation for DTx beyond classical RCTs (64).”

• cites [64] Guo, 2020, Challenges for the evaluation of digital health solutions—a call for innovative evidence generation approaches, npj Digit Med sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S2 (checkable). May combine two claims.

The differing descriptions of similar control conditions may exemplify the "known unknowns" of mechanisms and efficacy factors in DTx and how individual authors judge the efficacy of different features differently or at the very least point to inconsistencies of describing control conditions.

“This may exemplify the “known unknowns” of mechanisms and efficacy factors in DTx and how individual authors judge the efficacy of different features differently or at the very least point to inconsistencies of describing control conditions.”

Classified as probably not a substantial claim. May combine two claims.

The review does not constitute a systematic review and may not contain all control condition choices in the field.

“The review therefore does not constitute a systematic review and may not contain all control condition choices in the field.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The regulatory considerations provided and majority of trials in this review are US-centered, and therefore our assessment does not represent or discuss potential differences in regulatory practices beyond the US.

“Further, the regulatory considerations provided and majority of trials in this review are US-centered, and therefore our assessment does not represent or discuss potential differences in regulatory practices beyond the US.”

Classified as probably not a substantial claim. May combine two claims.

This review is based on DTx deploying cognitive, behavioral and psychosocial interventions that fulfill the DTx definition, may have attempted to or are attempting FDA clearance, and are presented by the DTx alliance and similar trials based on the authors' awareness of the field.

“This review is based on DTx deploying cognitive, behavioral and psychosocial interventions that that fulfill the DTx definition, may have attempted to or are attempting FDA clearance, and are presented by the DTx alliance and similar trials based on the authors' awareness of the field.”

Classified as probably not a substantial claim.

The inconsistency in control condition nomenclature may relate to different definitions of these terms in the literature, for example in the behavioral research literature versus FDA guidance.

“This may relate to different definitions of these terms in the literature, for example in the behavioral research literature versus FDA guidance (see Definitions Table 1).”

Probably not a substantial claim: may not pass S2 (checkable).

Relatively more novel interventions may require more stringent controls compared to digital translations of known psychosocial behavioral interventions, such as CBT.

“In addition, relatively more novel interventions may require more stringent controls compared to digital translations of known psychosocial behavioral interventions, such as CBT (12).”

• cites [12] Mohr, 2021, Banbury forum consensus statement on the path forward for digital mental health treatment, Psychiatr Serv sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S4 (based). May combine two claims.

Whether participants expected two potentially active treatments versus a treatment and placebo control can influence the size of nonspecific effects of the two arms and may affect the number needed to enroll to show a significant difference between the conditions.

“Whether participants expected two potentially active treatments versus a treatment and placebo control can influence the size of nonspecific effects of the two arms and may affect the number needed to enroll to show a significant difference between the conditions.”