CoChat
Sign Up
Cited by 305Digital Mental Health InterventionsMental Health Treatment and AccessMobile Health and mHealth Applications

The DOI matches its Crossref record; 74 of the 74 references it cites were found in a registry or an index; none of the 27 sources its claims cite is checked for support yet.

Effect of Engagement With Digital Interventions on Mental Health Outcomes: A Systematic Review and Meta-Analysis

Open access
Gan, Daniel Z. Q., McGillivray, Lauren, Han, Jin, Christensen, Helen, Torok, Michelle·2021·Frontiers in Digital Health
DOI 10.3389/fdgth.2021.764079PMC PMC8599127
Open this paper in CoChat — free accountOpen PDF

Abstract

Digital mental health interventions (DMHIs) present a promising way to address gaps in mental health service provision. However, the relationship between user engagement and outcomes in the context of these interventions has not been established. This study addressed the current state of evidence on the relationship between engagement with DMHIs and mental health outcomes. MEDLINE, PsycINFO, and EmBASE databases were searched from inception to August 1, 2021. Original or secondary analyses of randomized controlled trials (RCTs) were included if they examined the relationship between DMHI engagement and post-intervention outcome(s). Thirty-five studies were eligible for inclusion in the narrative review and 25 studies had sufficient data for meta-analysis. Random-effects meta-analyses indicated that greater engagement was significantly associated with post-intervention mental health improvements, regardless of whether this relationship was explored using correlational [ r = 0.24, 95% CI (0.17, 0.32), Z = 6.29, p < 0.001] or between-groups designs [Hedges' g = 0.40, 95% CI (0.097, 0.705), p = 0.010]. This association was also consistent regardless of intervention type (unguided/guided), diagnostic status, or mental health condition targeted. This is the first review providing empirical evidence that engagement with DMHIs is associated with therapeutic gains. Implications and future directions are discussed. Systematic Review Registration: PROSPERO, identifier: CRD 42020184706.

Overview

AI snapshot

Written from the paper’s full text.

TL;DR

This systematic review and meta-analysis found that greater engagement with digital mental health interventions was modestly but significantly associated with post-intervention mental health improvements. It is reported as the first meta-analytic evidence quantifying this engagement–outcome relationship across randomized controlled trial analyses.

Key findings

  • In the main correlational meta-analysis, greater engagement was associated with better post-intervention mental health outcomes: 20 studies, N = 1,808, r = 0.25, 95% CI (0.17, 0.32), Z = 6.29, p < 0.001; the abstract reports r = 0.24 for this association.
  • In between-groups designs, higher content access was associated with moderate improvements: 5 studies, Hedges’ g = 0.40, SE = 0.16, 95% CI (0.097, 0.705), Z = 2.59, p = 0.010.
  • Subgroup meta-analyses found significant positive associations for unguided interventions (r = 0.23), guided interventions (r = 0.30), self-report symptom screening (r = 0.24), diagnosed samples (r = 0.28), anxiety outcomes (r = 0.33), and depressive outcomes (r = 0.33).
  • Narrative synthesis of 35 studies was mixed: 14 studies (40%) reported a significant positive association, 5 (17.1%) reported mixed findings, and 16 (45.7%) found no significant association.

Methods

  • Systematic search of MEDLINE, PsycINFO, and EmBASE from inception to August 1, 2021; eligible studies were original or secondary analyses of RCTs of digital mental health interventions that quantitatively examined engagement and post-intervention mental health outcomes.
  • Thirty-five studies were included in the narrative review; 25 had sufficient data for meta-analysis. Engagement was defined as objective intervention use, with number of modules/activities/sessions completed used as the primary engagement measure.
  • Random-effects meta-analyses were conducted separately for correlational and between-groups designs, with planned subgroup analyses by guided/unguided intervention, diagnostic status, and anxiety/depression target; heterogeneity was assessed with I², publication bias with funnel plots and Egger’s test, and risk of bias with Cochrane RoB 2.

Limitations

  • The authors state that included studies differed in target population, digital intervention, and mental health condition, preventing specific analyses of a particular engagement measure on a particular outcome.
  • The authors state that studies used different statistical approaches, so repeated-measures and between-subject designs had to be analyzed separately; Pearson’s r also had to be estimated for some correlational studies.
  • The authors note that bivariate correlations do not account for other variables that may influence the engagement–outcome association; the text also makes evident risk-of-bias concerns, including selective reporting in 28 studies (80.0%), outcome measurement bias in 19 studies (54.3%), and lack of participant blinding in 16 studies (45.7%).

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.3389/fdgth.2021.764079

Cites

74 of 74 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

0 of 27 cited sources checked · 0 supported · 0 flagged

Show all 65 claims

  • It may state two separate claims as one. 20
  • Probably not a substantial claim. 10
  • The quote only partly states this claim. 1
29 of 29

  • The paper relies on this without support.

    The included studies' engagement measures and mental health outcomes are sufficiently comparable to pool in meta-analysis.

    “Studies which utilized similar measures of engagement and which employed similar analytical techniques to test the association between engagement and outcome(s) were pooled together for meta-analysis.”

  • The paper relies on this without support.

    The association between engagement and mental health outcomes can be interpreted as therapeutic gains from engagement, rather than reverse causation or confounding.

    “This is the first review providing empirical evidence that engagement with DMHIs is associated with therapeutic gains.”

  • The paper relies on this without support.

    The search strategy adapted from MEDLINE to other databases retains adequate sensitivity.

    “This search strategy was tested in MEDLINE, where it achieved 100% sensitivity against the initial test set, and was subsequently adapted for use in the other databases.”

  • The paper relies on this without support.

    The increasing number of individual studies examining the engagement–mental health outcome association now enables a review and quantification of the literature.

    “The increasing number of individual studies examining the engagement—mental health outcome association published in recent years, now enables a review and quantification of the literature.”

  • The paper relies on this without support.

    The relationship between user engagement and outcomes in digital mental health interventions has not been established.

    “However, the relationship between user engagement and outcomes in the context of these interventions has not been established.”

  • The paper relies on this without support.

    This is the first systematic review and meta-analysis to quantitatively examine whether the level of user engagement with a digital intervention is associated with change in mental health outcomes after the intervention period.

    “To our knowledge, this is the first systematic review and meta-analysis to quantitatively examine whether the level of user engagement with a digital intervention was associated with change in mental health outcomes after the intervention period.”

  • The paper relies on this without support.

    Robust empirical evidence to support or validate the hypothesis that engagement with digital interventions is positively associated with improvements in mental health is scant.

    “Although it is widely accepted that the extent of engagement with digital interventions will be positively associated with improvements in mental health, robust empirical evidence to support or validate this hypothesis is scant.”
7 of 7

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Statistics recomputed

Every reported test result, recomputed from its test statistic and degrees of freedom. A result shows as not adding up only once an independent check has confirmed it.

  • Consistentz = 6.29reported p < .001 · recomputed p < .001

    “Z = 6.29, p < .001”

  • Consistentz = 2.59reported p = .01 · recomputed p = .010

    “Z = 2.59, p = .010”

  • Consistentz = 6.07reported p < .001 · recomputed p < .001

    “Z = 6.07, p < .001”

  • Consistentz = 6.13reported p < .001 · recomputed p < .001

    “Z = 6.13, p < .001”

  • Consistentz = 6.03reported p < .001 · recomputed p < .001

    “Z = 6.03, p < .001”

  • Consistentz = 6.76reported p < .001 · recomputed p < .001

    “Z = 6.76, p < .001”

  • Consistentz = 3.12reported p = .002 · recomputed p = .002

    “Z = 3.12, p = .002”

7 of 7

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • Mixed-findings percentage does not match the study count

    Minor

    Five of 35 studies is 14.3%, not 17.1%.

    Reported 17.1 · recomputed 14.285714

  • Reported maximum mean age excludes a listed study

    Minor

    The stated maximum mean age is 58.4 years, but Table 1 lists a study with a mean age of 63.6 years.

  • A decrease in Hedges' g is called an increase

    Minor

    The leave-one-out description calls the change from 0.40 to 0.25 an increase, although it also describes the omission as producing a reduction.

  • Discussion reverses the sign of the pooled Hedges' g

    Minor

    The discussion reports the conclusion-bearing between-groups effect as −0.40, whereas the main analysis reports +0.40.

  • Study 50 is classified as not formally diagnosed

    Minor

    The diagnostic-status narrative places study 50 among studies whose participants were not formally diagnosed, whereas Table 1 says its participants fulfilled diagnostic criteria.

  • Self-report subgroup has conflicting starting correlations

    Minor

    The subgroup's pooled correlation is stated as 0.24, but its leave-one-out description says removal reduced the overall correlation from 0.27.

  • Study 47 is described as mixed despite its sole listed positive measure

    Minor

    Study 47 is included among studies with positive results for some but not all engagement measures. Table 1 lists only one engagement measure for it, marked positive.

  • Study 52 did not show an association across all engagement measures

    Minor

    The narrative includes study 52 among studies with significant improvements consistently across all measures, but Table 1 marks four of its five engagement measures as having no association.

  • Outcome subgroup is introduced as a diagnostic-status subgroup

    Minor

    The specific-outcomes section says its second analysis combines six studies involving participants who met diagnostic criteria, but describes the resulting analysis as six studies with depressive symptoms as the primary outcome.

  • Study 41 is counted as null despite a positive table result

    Minor

    The narrative places study 41 among those with no significant association, while Table 1 marks its engagement measure as positively associated with improvement.

  • CBT study percentage does not match the study count

    Minor

    Twenty-seven of 35 studies is 77.1%, not 80%.

    Reported 80 · recomputed 77.142857

  • Abstract and results give different pooled correlations

    Minor

    The abstract reports r = 0.24 for the correlational analysis, while the main results report r = 0.25 for that analysis.

12 of 12

Citations

Cites

Flagged first, then by citations

Show all 74 references

Research works printed but not linked: 1

  1. Borenstein, 2013, Comprehensive Meta-Analysis (version 3.0)

Cited by

Sorted by citations

Related

See all

Metadata

Authors
Gan, D. Z. Q., McGillivray, L., Han, J., Christensen, H., & Torok, M.
Affiliations
Black Dog Institute (AU)
Corresponding author
Gan, Daniel Z. Q.
ORCID
Venue
Frontiers in Digital Health
Publisher
Frontiers Media SA
Published
2021-11-04
Volume
3
Pages
764079-764079
ISSN
2673-253X (ISSN-L)
DOI
10.3389/fdgth.2021.764079
PMC
PMC8599127
OpenAlex
W3213740945
Semantic Scholar
e547b3c221a962dd1b4e073942362ba3e9bc08fa
Cited by
305
References
74
Citations by year
2026: 82 · 2025: 95 · 2024: 71 · 2023: 42 · 2022: 14 · 2021: 1
FWCI
5.14
Type
journal article
Language
en
Open access
Licence
Locations
  • Frontiers in Digital Health (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
Funders
  • Australian Government
  • National Health and Medical Research Council
Topics
  • Social Sciences › Psychology › Applied Psychology › Digital Mental Health Interventions
  • Health Sciences › Medicine › Psychiatry and Mental health › Mental Health Treatment and Access
  • Health Sciences › Health Professions › General Health Professions › Mobile Health and mHealth Applications
Keywords
digital mental health interventions, user engagement, mental health outcomes, mental health treatment gap, therapeutic gain

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

A language model drew the claims from the paper on 2026-10-09, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-09 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review