Exploring the Relationship Between Usage and Outcomes of an Internet-Based Intervention for Individuals With Depressive Symptoms: Secondary Analysis of Data From a Randomized Controlled Trial
Open accessPublished version of a preprint
The DOI matches its Crossref record; 43 of the 43 references it cites were found in a registry or an index.
Abstract
Background: Internet interventions can easily generate objective data about program usage. Increasingly, more studies explore the relationship between usage and outcomes, but they often report different metrics of use, and the findings are mixed. Thus, current evaluations fail to demonstrate which metrics should be considered and how these metrics are related to clinically meaningful change. Objective: This study aimed to explore the relationship between several usage metrics and outcomes of an internet-based intervention for depression. Methods: This is a secondary analysis of data from a randomized controlled trial that examined the efficacy of an internet-based cognitive behavioral therapy for depression (Space from Depression) in an adult community sample. All participants who enrolled in the intervention, regardless of meeting the inclusion criteria, were included in this study. Space from Depression is a 7-module supported intervention, delivered over a period of 8 weeks. Different usage metrics (ie, time spent, modules and activities completed, and percentage of program completion) were automatically collected by the platform, and composite variables from these (eg, activities per session) were computed. A breakdown of the usage metrics was obtained by weeks. For the analysis, the sample was divided into those who obtained a reliable change (RC)—and those who did not. Results: Data from 216 users who completed pre- and posttreatment outcomes were included in the analyses. A total of 89 participants obtained an RC, and 127 participants did not obtain an RC. Those in the RC group significantly spent more time, had more log-ins, used more tools, viewed a higher percentage of the program, and got more reviews from their supporter compared with those who did not obtain an RC. Differences between groups in usage were observed from the first week in advance across the different metrics, although they vanished over time. In the RC group, the usage was higher during the first 4 weeks, and then a significant decrease was observed. Our results showed that specific levels of platform usage, 7 hours total time spent, 15 sessions, 30 tools used, and 50% of program completion, were associated with RC. Conclusions: Overall, the results showed that those individuals who obtained an RC after the intervention had higher levels of exposure to the platform. The usage during the first half of the intervention was higher, and differences between groups were observed from the first week. This study also showed specific usage levels associated with outcomes that could be tested in controlled studies to inform the minimal usage to establish adherence. These results will help to better understand how to use internet-based interventions and what optimal level of engagement can most affect outcomes. Trial Registration: ISRCTN Registry ISRCTN03704676; http://www.isrctn.com/ISRCTN03704676 International Registered Report Identifier (IRRID): RR2-10.1186/1471-244X-14-147
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures





Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Associations with adherence have been demonstrated for a range of outcomes, including depression [105,106] and anxiety [107,108].”
From The therapeutic alliance in digital mental health interventions for serious mental illnesses: narrative review (2020) · cites this paper as [105]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Eysenbach, G. (2005) · Journal of Medical Internet Research · doi:10.2196/jmir.7.1.e11 · Cited by 2,524 · cited 2×
“Although treatment dropout and spontaneous remission might explain some of these trends, as they could in other psychological interventions, understanding more completely how internet-delivered interventions are used and whether their usage differs depending on user characteristics is an important area of investigatio…”
Greiner, M., Pfeiffer, D., & Smith, R. D. (2000) · Preventive Veterinary Medicine · doi:10.1016/s0167-5877(00)00115-x · Cited by 1,923 · cited 1×
“To obtain the optimal cutoff for achieving RC, a receiver operating characteristic (ROC) curve analysis [34] was performed using each of the 4 individual usage metrics (total time spent on the platform, number of sessions, number of activities, and percentage of the program viewed) as test variables against the RC sta…”
Computer-based psychological treatments for depression: A systematic review and meta-analysis
Richards, D., & Richardson, T. (2012) · Clinical Psychology Review · doi:10.1016/j.cpr.2012.02.004 · Cited by 1,442 · cited 2×
“Different meta-analytic reviews have shown that supported interventions have higher rates of adherence and better outcomes than self-guided ones [1,3,4,25].”
Self-citationInternet-Based and Other Computerized Psychological Treatments for Adult Depression: A Meta-Analysis
Andersson, G., & Cuijpers, P. (2009) · Cognitive Behaviour Therapy · doi:10.1080/16506070903318960 · Cited by 1,368 · cited 2×
“Different meta-analytic reviews have shown that supported interventions have higher rates of adherence and better outcomes than self-guided ones [1,3,4,25].”
Persuasive System Design Does Matter: a Systematic Review of Adherence to Web-based Interventions
Kelders, S. M., Kok, R. N., Ossebaard, H. C., & Van Gemert-Pijnen, J. E. (2012) · Journal of Medical Internet Research · doi:10.2196/jmir.2104 · Cited by 1,308 · cited 2×
“This paper contributes to the concept of adherence by providing an empirical justification of intended use, that is, “the extent to which individuals should experience the content to derive maximum benefit from the intervention, as defined or implied by its creators” [22].”
Perski, O., Blandford, A., West, R., & Michie, S. (2017) · Translational Behavioral Medicine · doi:10.1007/s13142-016-0453-1 · Cited by 1,193 · cited 2×
“Our results confirm the largely used statement the more usage, the better [5,16], although it is important to note that our variables account for active (ie, activities completed) and passive (ie, percentage viewed) engagement, and both elements are important to get the most out of these interventions [20].”
Adherence in Internet Interventions for Anxiety and Depression
Christensen, H., Griffiths, K. M., & Farrer, L. (2009) · Journal of Medical Internet Research · doi:10.2196/jmir.1194 · Cited by 1,133 · cited 3×
“In this study, on average, the users utilized 57% of the program, which is similar to completion rates in other internet-based interventions for depression [36,39].”
Mohr, D. C., Cuijpers, P., & Lehman, K. (2011) · Journal of Medical Internet Research · doi:10.2196/jmir.1602 · Cited by 1,097 · cited 1×
“In this sense, no studies have explored the role of metrics related to the support, such as the number of reviews and the number of user replies to these reviews, which could shed more light on the role of support in the usage of the program and their outcomes [28].”
Bower, P., & Gilbody, S. (2005) · British Journal of Psychiatry · doi:10.1192/bjp.186.1.11 · Cited by 1,084
Andersson, G., Cuijpers, P., Carlbring, P., Riper, H., & Hedman, E. (2014) · World Psychiatry · doi:10.1002/wps.20151 · Cited by 1,077 · cited 1×
“Internet- and computer-based interventions for depression have been shown to be effective in several meta-analytic reviews, reporting comparable effects to face-to-face treatments [1-5].”
Research works printed but not linked: 1
- Beck, AT, 1996, BDI-II, Beck Depression Inventory: Manual
Cited by
Digital interventions for the treatment of depression: a meta-analytic review
Moshe, I., Terhorst, Y., Philippi, P., Domhardt, M., Cuijpers, P., Cristea, I., Pulkki-Råback, L., Baumeister, H., & Sander, L. B. (2021) · Psychological Bulletin · doi:10.1037/bul0000334 · Cited by 390
Tremain, H., McEnery, C., Fletcher, K., & Murray, G. (2020) · JMIR Mental Health · doi:10.2196/17204 · Cited by 193
Richards, D., Enrique, A., Eilert, N., Franklin, M., Palacios, J., Duffy, D., Earley, C., Chapman, J., Jell, G., Sollesse, S., et al. (2020) · npj Digital Medicine · doi:10.1038/s41746-020-0293-8 · Cited by 154
Self-citationChien, I., Enrique, A., Palacios, J., Regan, T., Keegan, D., Carter, D., Tschiatschek, S., Nori, A., Thieme, A., Richards, D., et al. (2020) · JAMA Network Open · doi:10.1001/jamanetworkopen.2020.10791 · Cited by 111
Deisenhofer, A., Barkham, M., Beierl, E. T., Schwartz, B., Doorn, K. A., Beevers, C. G., Berwian, I. M., Blackwell, S. E., Bockting, C., Brakemeier, E., et al. (2023) · Behaviour Research and Therapy · doi:10.1016/j.brat.2023.104443 · Cited by 138
“In addition to active data collection, these tools may also be used to gather passive data on granular and objective treatment engagement, such as the tools and pages viewed during a digital therapy session (Enrique, Palacios, Ryan, & Richards, 2019).”
Method
Metadata
- Authors
- Enrique, A., Palacios, J. E., Ryan, H., & Richards, D.
- Affiliations
- SilverCloud (Ireland) (IE)
- Venue
- Journal of Medical Internet Research
- Publisher
- JMIR Publications Inc.
- Published
- 2019-08-01
- Volume
- 21
- Issue
- 8
- Pages
- e12775
- ISSN
- 1438-8871 (ISSN-L)
- PMC
- PMC6694731
- OpenAlex
- W2950346625
- Cited by
- 81
- References
- 43
- Citations by year
- 2026: 3 · 2025: 12 · 2024: 10 · 2023: 8 · 2022: 24 · 2021: 11 · 2020: 13
- FWCI
- 13.50
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- Locations
- Journal of Medical Internet Research (publishedVersion, cc-by)
- Europe PMC (PubMed Central) (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Arrow@dit (Dublin Institute of Technology) (submittedVersion, cc-by-nc-sa)
- Trinity's Access to Research Output (TARA) (Trinity College Dublin) (submittedVersion, cc-by)
- Topics
- Social Sciences › Psychology › Applied Psychology › Digital Mental Health Interventions
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Anxiety, Depression, Psychometrics, Treatment, Cognitive Processes
- Health Sciences › Medicine › Psychiatry and Mental health › Mental Health Treatment and Access
- Keywords
- online intervention, usage metrics, program completion, reliable change, medication adherence, engagement, digital mental health interventions, depressive symptoms, community-dwelling adults
- MeSH terms
- Internet-Based Intervention, Adult, Depression, Depressive Disorder, Female, Humans, Male, Time Factors, Cognitive Behavioral Therapy, Randomized Controlled Trials as Topic, Treatment Outcome
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.