CoChat
Sign Up
Cited by 305Digital Mental Health InterventionsMental Health Treatment and AccessMobile Health and mHealth Applications

Effect of Engagement With Digital Interventions on Mental Health Outcomes: A Systematic Review and Meta-Analysis

Open access
Gan, Daniel Z. Q., McGillivray, Lauren, Han, Jin, Christensen, Helen, Torok, Michelle·2021·Frontiers in Digital Health
DOI 10.3389/fdgth.2021.764079PMC PMC8599127
Open this paper in CoChat — free accountOpen PDF

Abstract

Digital mental health interventions (DMHIs) present a promising way to address gaps in mental health service provision. However, the relationship between user engagement and outcomes in the context of these interventions has not been established. This study addressed the current state of evidence on the relationship between engagement with DMHIs and mental health outcomes. MEDLINE, PsycINFO, and EmBASE databases were searched from inception to August 1, 2021. Original or secondary analyses of randomized controlled trials (RCTs) were included if they examined the relationship between DMHI engagement and post-intervention outcome(s). Thirty-five studies were eligible for inclusion in the narrative review and 25 studies had sufficient data for meta-analysis. Random-effects meta-analyses indicated that greater engagement was significantly associated with post-intervention mental health improvements, regardless of whether this relationship was explored using correlational [ r = 0.24, 95% CI (0.17, 0.32), Z = 6.29, p < 0.001] or between-groups designs [Hedges' g = 0.40, 95% CI (0.097, 0.705), p = 0.010]. This association was also consistent regardless of intervention type (unguided/guided), diagnostic status, or mental health condition targeted. This is the first review providing empirical evidence that engagement with DMHIs is associated with therapeutic gains. Implications and future directions are discussed. Systematic Review Registration: PROSPERO, identifier: CRD 42020184706.

AI snapshot

Written from the paper’s full text.

TL;DR

This systematic review and meta-analysis found that greater engagement with digital mental health interventions was modestly but significantly associated with post-intervention mental health improvements. It is reported as the first meta-analytic evidence quantifying this engagement–outcome relationship across randomized controlled trial analyses.

Key findings

  • In the main correlational meta-analysis, greater engagement was associated with better post-intervention mental health outcomes: 20 studies, N = 1,808, r = 0.25, 95% CI (0.17, 0.32), Z = 6.29, p < 0.001; the abstract reports r = 0.24 for this association.
  • In between-groups designs, higher content access was associated with moderate improvements: 5 studies, Hedges’ g = 0.40, SE = 0.16, 95% CI (0.097, 0.705), Z = 2.59, p = 0.010.
  • Subgroup meta-analyses found significant positive associations for unguided interventions (r = 0.23), guided interventions (r = 0.30), self-report symptom screening (r = 0.24), diagnosed samples (r = 0.28), anxiety outcomes (r = 0.33), and depressive outcomes (r = 0.33).
  • Narrative synthesis of 35 studies was mixed: 14 studies (40%) reported a significant positive association, 5 (17.1%) reported mixed findings, and 16 (45.7%) found no significant association.

Methods

  • Systematic search of MEDLINE, PsycINFO, and EmBASE from inception to August 1, 2021; eligible studies were original or secondary analyses of RCTs of digital mental health interventions that quantitatively examined engagement and post-intervention mental health outcomes.
  • Thirty-five studies were included in the narrative review; 25 had sufficient data for meta-analysis. Engagement was defined as objective intervention use, with number of modules/activities/sessions completed used as the primary engagement measure.
  • Random-effects meta-analyses were conducted separately for correlational and between-groups designs, with planned subgroup analyses by guided/unguided intervention, diagnostic status, and anxiety/depression target; heterogeneity was assessed with I², publication bias with funnel plots and Egger’s test, and risk of bias with Cochrane RoB 2.

Limitations

  • The authors state that included studies differed in target population, digital intervention, and mental health condition, preventing specific analyses of a particular engagement measure on a particular outcome.
  • The authors state that studies used different statistical approaches, so repeated-measures and between-subject designs had to be analyzed separately; Pearson’s r also had to be estimated for some correlational studies.
  • The authors note that bivariate correlations do not account for other variables that may influence the engagement–outcome association; the text also makes evident risk-of-bias concerns, including selective reporting in 28 studies (80.0%), outcome measurement bias in 19 studies (54.3%), and lack of participant blinding in 16 studies (45.7%).

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.3389/fdgth.2021.764079

Cites

74 of 74 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

0 of 27 cited sources checked · 0 supported · 0 flagged

  • Findingown data

    Among 14 studies of guided digital interventions, 6 reported a positive association between engagement and mental health outcomes.

    “Fourteen studies examined DMHIs that were guided (20, 50–62). Among these studies, six reported a positive association between engagement and mental health outcomes”
    Quote in the paperNot a substantial claim
  • Findingown data

    Of 35 included studies, 16 (45.7%) did not find any significant association between engagement and post-intervention mental health outcomes.

    “Sixteen studies (45.7%) did not find any significant association between engagement and post-intervention mental health outcomes”
    Quote in the paperNot a substantial claim
  • Findingown data

    In 23 studies where participants were not formally diagnosed but screened for symptoms, 11 did not find a significant association between engagement and the primary mental health outcome.

    “while the other 11 did not find a significant association”
    Quote in the paperNot a substantial claim
  • Findingown data

    The association between greater engagement and post-intervention mental health improvements was consistent regardless of intervention type (unguided/guided), diagnostic status, or mental health condition targeted.

    “This association was also consistent regardless of intervention type (unguided/guided), diagnostic status, or mental health condition targeted.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The included studies' engagement measures and mental health outcomes are sufficiently comparable to pool in meta-analysis.
  • Findingown data

    In 23 studies where participants were not formally diagnosed but screened for symptoms, 12 reported a positive association between engagement and the primary mental health outcome.

    “In the remaining 23 studies, participants were not formally diagnosed with a mental health condition but were screened for symptoms indicative of mental disorders. Of these studies, 12 reported a positive association between engagement and the primary mental health outcome”
    Quote in the paperNot a substantial claim
  • May combine two claims. 20
  • Classified as probably not a substantial claim. 10
  • The quote only partly states this claim. 1

  • May combine two claims.

    Findingown data

    Among 4 studies including both unguided and guided versions, 3 reported a positive association between engagement and mental health outcomes.

    “The remaining four studies included participants who used both unguided and guided versions of the same digital intervention (63–66). Of these studies, three reported a positive association between engagement and mental health outcomes”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Among 4 studies including both unguided and guided versions, 1 did not find a significant association between engagement and mental health outcomes.

    “while one did not find a significant association”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    In the nine included studies examining psychological distress or psychological recovery, three reported a positive association between engagement and post-intervention symptoms, while six found no significant association.

    “Nine studies (35, 36, 42, 44, 46, 50, 52, 59, 64) investigated the relationship between engagement and psychological distress or psychological recovery. Of these, three reported a positive association between engagement and post-intervention symptoms (36, 50, 52) while six did not find any significant association (35, 42, 44, 46, 59, 64).”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Omission of Oromendia et al. (66) had the largest effect reduction in the between-group meta-analysis, increasing the Hedges' g value from 0.40 to 0.25.

    “Omission of Oromendia et al. (66) had the largest effect reduction, increasing the Hedges' g value from 0.40 to 0.25.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    In the nine included studies examining depression-related symptoms, six reported a positive association between engagement and post-intervention symptoms, while three found no significant association.

    “Nine studies (33, 38, 40, 48, 49, 57, 58, 63, 65) investigated the relationship between engagement and depression-related symptoms. Of these, six reported a positive association between engagement and post-intervention symptoms (33, 40, 49, 57, 63, 65) while three did not find any significant association (38, 48, 58).”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Of the 25 studies included in the meta-analysis, 20 employed correlational designs to investigate the association between engagement and mental health outcomes, and 5 used between-group mixed model comparisons to identify differences in post-intervention outcomes between users with higher vs. lower levels of engagement.

    “Of the 25 studies included in the meta-analysis, 20 studies (20, 33–35, 37, 40, 41, 43, 47, 48, 51, 54–57, 59, 61, 63–65) employed correlational designs to investigate the association between engagement and mental health outcomes. The other five studies (36, 38, 50, 52, 66) used between-group mixed model comparisons to identify any differences in post-intervention outcomes between users who”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    25 of the 35 included studies were included in the meta-analysis.

    “All studies were included in the narrative synthesis, and 25 of the 35 were included in the meta-analysis.”
    Quote in the paperNot a substantial claim

  • The quote only partly states this claim.

    Findingown data

    There was no heterogeneity in the distribution of individual study effect sizes for the anxiety-related symptoms meta-analysis (I2 = 0.0%).

    “There was no heterogeneity in the distribution of individual study effect sizes (I2 = 0.0%).”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductown data

    The search strategy achieved 100% sensitivity against the initial test set in MEDLINE.

    “This search strategy was tested in MEDLINE, where it achieved 100% sensitivity against the initial test set”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The search strategy adapted from MEDLINE to other databases retains adequate sensitivity.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductown data

    Inter-rater consensus was acceptable for full-text screening (κ = 0.80, p < 0.001).

    “Inter-rater consensus was acceptable for both title/abstract (κ = 0.86, p < 0.001) and full-text screening (κ = 0.80, p < 0.001).”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductcitation

    This study is registered with PROSPERO (CRD 42020184706) and adheres to PRISMA reporting guidelines.

    “This study is registered with PROSPERO (CRD 42020184706) and adheres to the Preferred Reporting Items for Systematic Reviews and Meta-analyses (PRISMA) reporting guidelines (23).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [23] Liberati, 2009, The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate health care interventions: explanation and elaboration, J Clin Epidemiol. source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductown data

    Risk of bias ratings were independently conducted by two pairs of coders, with an overall agreement rate of 93.3%.

    “Ratings were independently conducted by two pairs of coders (40% by DZQG and LM, and 60% by DZQG and JH). Discrepancies were resolved through mutual discussion. An overall agreement rate of 93.3% was reached.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Study design and conductown data

    Selective reporting was the largest source of bias, with 28 studies (80.0%) not reporting sufficient information to rule out bias in this domain.

    “Selective reporting was identified to be the largest source of bias, with 28 studies (80.0%) not reporting sufficient information—such as a prospectively published trial protocol—to rule out bias in this domain.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductown data

    Inter-rater consensus was acceptable for title/abstract screening (κ = 0.86, p < 0.001).

    “Inter-rater consensus was acceptable for both title/abstract (κ = 0.86, p < 0.001) and full-text screening (κ = 0.80, p < 0.001).”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Premisecitation

    Only one prior systematic review has explored the effect of engagement on the effectiveness of digital interventions, and it was limited to a narrative synthesis due to heterogeneity in how engagement is operationalized.

    “To date, only one prior systematic review (18) has explored the effect of engagement on the effectiveness of digital interventions. This review was limited to a narrative synthesis, citing heterogeneity in how engagement is operationalized”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [18] Donkin, 2011, Systematic review of the impact of adherence on the effectiveness of e-therapies, J Med Internet Res. source

      Not checked in the free run.

  • May combine two claims.

    Study design and conductown data

    Risk of outcome measurement bias was the second most common source of potential bias, with 19 studies (54.3%) reporting that outcome assessors were aware of the intervention received.

    “Risk of outcome measurement bias was the second most common source of potential bias, with 19 studies (54.3%) reporting that outcome assessors—usually study participants themselves—were aware of the intervention received by study participants.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductcitation

    Risk of bias was assessed using Version 2 of the Cochrane risk-of-bias tool for randomized trials (RoB 2).

    “Risk of bias was assessed using Version 2 of the Cochrane risk-of-bias tool for randomized trials (RoB 2) (32).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [32] Sterne, 2019, RoB 2: a revised tool for assessing risk of bias in randomised trials, BMJ. source

      Not checked in the free run.

  • May combine two claims.

    Interpretationown data

    Greater engagement with digital interventions is modestly but significantly associated with improvements in mental health (effect size range: r = 0.23 to Hedges' g = −0.40) regardless of the level of guidance provided, mental health symptom severity of users, or type of mental health condition(s) targeted by the intervention.

    “That is, the results unequivocally support that greater engagement with digital interventions is modestly but significantly associated with improvements in mental health (effect size range: r = 0.23 to Hedges' g = −0.40) regardless of the level of guidance provided, mental health symptom severity of users, or type of mental health condition(s) targeted by the intervention.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Study design and conductcitation

    Pearson's r had to be estimated for some studies employing correlational designs; this was preferred to omitting studies without this data to maximize objectivity and minimize selection bias.

    “Third, Pearson's r had to be estimated for some of the studies which employed correlational designs. While estimating r from other indices may not be ideal, it is preferred to omitting studies without this data so as to maximize objectivity and minimize selection bias (26, 29).”
    Quote in the paperSubstantial claim0 of 2 sources checked
    • cites [26] Wilson, 2001

      Not checked in the free run.

    • cites [29] Borenstein, 2009, Effect sizes for continuous data

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S2 (checkable).

    Interpretationreasoning

    Controlling for factors linked with engagement may be necessary for verifying the robustness of its relationship with outcomes.

    “Controlling for factors linked with engagement may be necessary for verifying the robustness of its relationship with outcomes.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Study design and conductcitation

    Effect sizes from repeated measures and between-subject designs are not comparable, so data from both types of studies had to be analyzed separately.

    “As effect sizes from repeated measures and between-subject designs are not comparable (26), data provided by both types of studies had to be analyzed separately.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [26] Wilson, 2001

      Not checked in the free run.

  • May combine two claims.

    Interpretationown data

    The meta-analyses (main and subgroup) consistently supported the a priori hypothesis of a positive engagement-outcome relationship, while the narrative synthesis showed mixed support.

    “While the narrative synthesis showed mixed support for a positive engagement—outcome relationship, the meta-analyses (main and subgroup) consistently supported our main a priori hypothesis.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Premisecitation

    Over 70% of users fail to complete all treatment modules, and more than 50% disengage before completing half of all treatment modules.

    “A review of empirical studies has found collectively low rates of DMHI completion, with over 70% of users failing to complete all treatment modules, and more than 50% disengaging before completing half of all treatment modules (15).”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [15] Karyotaki, 2015, Predictors of treatment dropout in self-guided web-based interventions for depression: an “individual patient data” meta-analysis, Psychol Med. source

      Not checked in the free run.

  • Classified as probably not a substantial claim.

    Premisecitation

    Apps which incorporated more elements aimed at promoting user engagement had larger effect sizes in a review of efficacy studies of smartphone apps for depression and anxiety.

    “a review of efficacy studies of smartphone apps for depression and anxiety found that apps which incorporated more elements aimed at promoting user engagement had larger effect sizes (17).”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [17] Wu, 2021, Smartphone apps for depression and anxiety: a systematic review and meta-analysis of techniques to increase engagement, NPJ Digit Med. source

      Not checked in the free run.

  • May combine two claims.

    Premisecitation

    Low engagement is a barrier to both optimal efficacy and adoption of digital mental health interventions into health settings and other translational pathways.

    “Low engagement—defined as suboptimal levels of user access and/or adherence to an intervention (12)—is touted as one of the main reasons why the potential benefits of these interventions remain unrealized in the real world (13, 14).”
    Quote in the paperSubstantial claim0 of 3 sources checked
    • cites [12] Torous, 2018, Clinical review of user engagement with mental health smartphone apps: evidence, theory and improvements, Evid Based Mental Health. source

      Not checked in the free run.

    • cites [13] Bidargaddi, 2018, To prompt or not to prompt? A microrandomized trial of time-varying push notifications to increase proximal engagement with a mobile health app, JMIR mHealth uHealth. source

      Not checked in the free run.

    • cites [14] Yardley, 2016, Understanding and promoting effective engagement with digital behavior change interventions, Am J Prev Med. source

      Not checked in the free run.

  • May combine two claims.

    Study design and conductreasoning

    The included studies differed in target population, DMHIs employed, and types of mental health conditions examined, so specific analyses of the effect of a specific measure of engagement on a particular mental health outcome could not be conducted.

    “First, the included studies differed in many ways, such as by target population, DMHIs employed, and the types of mental health conditions examined. Thus, specific analyses of the effect of a specific measure of engagement on a particular mental health outcome could not be conducted.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Premisecitation

    In one included study, sessions completed and time spent were correlated with reduced depression scores at post-intervention, but these associations were non-significant after email support was accounted for.

    “For example, one of the included studies reported that sessions completed and time spent were correlated with reduced depression scores at post-intervention; however, these associations were non-significant after email support was accounted for (63).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [63] Fuhr, 2018, The association between adherence and outcome in an Internet intervention for depression, J Affect Disord. source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S2 (checkable).

    Interpretationown data

    The present findings suggest that module completion may be one of the more acceptable measures of engagement to evaluate.

    “Secondly, our findings suggest that module completion may be one of the more acceptable measures of engagement to evaluate.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationcitation

    The present findings validate the qualitative findings of Donkin et al. (18) that improvements in mental health-related outcomes appeared to be associated with the number of modules accessed, but not with other engagement indicators (e.g., time spent, logins, and online interactions).

    “Our findings validate the qualitative findings reported in the systematic review by Donkin and colleagues (18), who reported that improvements in mental health-related outcomes appeared to be associated with the number of modules accessed, but not with other engagement indicators (e.g., time spent, logins, and online interactions).”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Donkin and colleagues (18)

      Not checked in the free run.

29 of 29

  • The paper relies on this without support.

    The included studies' engagement measures and mental health outcomes are sufficiently comparable to pool in meta-analysis.

    “Studies which utilized similar measures of engagement and which employed similar analytical techniques to test the association between engagement and outcome(s) were pooled together for meta-analysis.”

  • The paper relies on this without support.

    The association between engagement and mental health outcomes can be interpreted as therapeutic gains from engagement, rather than reverse causation or confounding.

    “This is the first review providing empirical evidence that engagement with DMHIs is associated with therapeutic gains.”

  • The paper relies on this without support.

    The search strategy adapted from MEDLINE to other databases retains adequate sensitivity.

    “This search strategy was tested in MEDLINE, where it achieved 100% sensitivity against the initial test set, and was subsequently adapted for use in the other databases.”

  • The paper relies on this without support.

    The increasing number of individual studies examining the engagement–mental health outcome association now enables a review and quantification of the literature.

    “The increasing number of individual studies examining the engagement—mental health outcome association published in recent years, now enables a review and quantification of the literature.”

  • The paper relies on this without support.

    The relationship between user engagement and outcomes in digital mental health interventions has not been established.

    “However, the relationship between user engagement and outcomes in the context of these interventions has not been established.”

  • The paper relies on this without support.

    This is the first systematic review and meta-analysis to quantitatively examine whether the level of user engagement with a digital intervention is associated with change in mental health outcomes after the intervention period.

    “To our knowledge, this is the first systematic review and meta-analysis to quantitatively examine whether the level of user engagement with a digital intervention was associated with change in mental health outcomes after the intervention period.”

  • The paper relies on this without support.

    Robust empirical evidence to support or validate the hypothesis that engagement with digital interventions is positively associated with improvements in mental health is scant.

    “Although it is widely accepted that the extent of engagement with digital interventions will be positively associated with improvements in mental health, robust empirical evidence to support or validate this hypothesis is scant.”
7 of 7

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Statistics recomputed

Every reported test result, recomputed from its test statistic and degrees of freedom. A result shows as not adding up only once an independent check has confirmed it.

  • Consistentz = 6.29reported p < .001 · recomputed p < .001

    “Z = 6.29, p < .001”

  • Consistentz = 2.59reported p = .01 · recomputed p = .010

    “Z = 2.59, p = .010”

  • Consistentz = 6.07reported p < .001 · recomputed p < .001

    “Z = 6.07, p < .001”

  • Consistentz = 6.13reported p < .001 · recomputed p < .001

    “Z = 6.13, p < .001”

  • Consistentz = 6.03reported p < .001 · recomputed p < .001

    “Z = 6.03, p < .001”

  • Consistentz = 6.76reported p < .001 · recomputed p < .001

    “Z = 6.76, p < .001”

  • Consistentz = 3.12reported p = .002 · recomputed p = .002

    “Z = 3.12, p = .002”

7 of 7

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • Mixed-findings percentage does not match the study count

    Minor

    Five of 35 studies is 14.3%, not 17.1%.

    Reported 17.1 · recomputed 14.285714

  • Reported maximum mean age excludes a listed study

    Minor

    The stated maximum mean age is 58.4 years, but Table 1 lists a study with a mean age of 63.6 years.

  • A decrease in Hedges' g is called an increase

    Minor

    The leave-one-out description calls the change from 0.40 to 0.25 an increase, although it also describes the omission as producing a reduction.

  • Discussion reverses the sign of the pooled Hedges' g

    Minor

    The discussion reports the conclusion-bearing between-groups effect as −0.40, whereas the main analysis reports +0.40.

  • Study 50 is classified as not formally diagnosed

    Minor

    The diagnostic-status narrative places study 50 among studies whose participants were not formally diagnosed, whereas Table 1 says its participants fulfilled diagnostic criteria.

  • Self-report subgroup has conflicting starting correlations

    Minor

    The subgroup's pooled correlation is stated as 0.24, but its leave-one-out description says removal reduced the overall correlation from 0.27.

  • Study 47 is described as mixed despite its sole listed positive measure

    Minor

    Study 47 is included among studies with positive results for some but not all engagement measures. Table 1 lists only one engagement measure for it, marked positive.

  • Study 52 did not show an association across all engagement measures

    Minor

    The narrative includes study 52 among studies with significant improvements consistently across all measures, but Table 1 marks four of its five engagement measures as having no association.

  • Outcome subgroup is introduced as a diagnostic-status subgroup

    Minor

    The specific-outcomes section says its second analysis combines six studies involving participants who met diagnostic criteria, but describes the resulting analysis as six studies with depressive symptoms as the primary outcome.

  • Study 41 is counted as null despite a positive table result

    Minor

    The narrative places study 41 among those with no significant association, while Table 1 marks its engagement measure as positively associated with improvement.

  • CBT study percentage does not match the study count

    Minor

    Twenty-seven of 35 studies is 77.1%, not 80%.

    Reported 80 · recomputed 77.142857

  • Abstract and results give different pooled correlations

    Minor

    The abstract reports r = 0.24 for the correlational analysis, while the main results report r = 0.25 for that analysis.

12 of 12

Cites

Flagged first, then by citations

Research works printed but not linked: 1

  1. Borenstein, 2013, Comprehensive Meta-Analysis (version 3.0)

Cited by

Sorted by citations

Related

Authors
Gan, D. Z. Q., McGillivray, L., Han, J., Christensen, H., & Torok, M.
Affiliations
Black Dog Institute (AU)
Corresponding author
Gan, Daniel Z. Q.
ORCID
Venue
Frontiers in Digital Health
Publisher
Frontiers Media SA
Published
2021-11-04
Volume
3
Pages
764079-764079
ISSN
2673-253X (ISSN-L)
DOI
10.3389/fdgth.2021.764079
PMC
PMC8599127
OpenAlex
W3213740945
Semantic Scholar
e547b3c221a962dd1b4e073942362ba3e9bc08fa
Cited by
305
References
74
Citations by year
2026: 82 · 2025: 95 · 2024: 71 · 2023: 42 · 2022: 14 · 2021: 1
FWCI
5.14
Type
journal article
Language
en
Open access
Licence
Locations
  • Frontiers in Digital Health (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
Funders
  • Australian Government
  • National Health and Medical Research Council
Topics
  • Social Sciences › Psychology › Applied Psychology › Digital Mental Health Interventions
  • Social Sciences › Psychology › Social Psychology › Mental Health Treatment and Access
  • Health Sciences › Health Professions › General Health Professions › Mobile Health and mHealth Applications
Keywords
digital mental health interventions, user engagement, mental health outcomes, mental health treatment gap, therapeutic gain

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-09.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-09.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review