Exploring the Relationship Between Usage and Outcomes of an Internet-Based Intervention for Individuals With Depressive Symptoms: Secondary Analysis of Data From a Randomized Controlled Trial
Open accessPublished version of a preprint
The DOI matches its Crossref record; 43 of the 43 references it cites were found in a registry or an index.
Abstract
Background: Internet interventions can easily generate objective data about program usage. Increasingly, more studies explore the relationship between usage and outcomes, but they often report different metrics of use, and the findings are mixed. Thus, current evaluations fail to demonstrate which metrics should be considered and how these metrics are related to clinically meaningful change. Objective: This study aimed to explore the relationship between several usage metrics and outcomes of an internet-based intervention for depression. Methods: This is a secondary analysis of data from a randomized controlled trial that examined the efficacy of an internet-based cognitive behavioral therapy for depression (Space from Depression) in an adult community sample. All participants who enrolled in the intervention, regardless of meeting the inclusion criteria, were included in this study. Space from Depression is a 7-module supported intervention, delivered over a period of 8 weeks. Different usage metrics (ie, time spent, modules and activities completed, and percentage of program completion) were automatically collected by the platform, and composite variables from these (eg, activities per session) were computed. A breakdown of the usage metrics was obtained by weeks. For the analysis, the sample was divided into those who obtained a reliable change (RC)—and those who did not. Results: Data from 216 users who completed pre- and posttreatment outcomes were included in the analyses. A total of 89 participants obtained an RC, and 127 participants did not obtain an RC. Those in the RC group significantly spent more time, had more log-ins, used more tools, viewed a higher percentage of the program, and got more reviews from their supporter compared with those who did not obtain an RC. Differences between groups in usage were observed from the first week in advance across the different metrics, although they vanished over time. In the RC group, the usage was higher during the first 4 weeks, and then a significant decrease was observed. Our results showed that specific levels of platform usage, 7 hours total time spent, 15 sessions, 30 tools used, and 50% of program completion, were associated with RC. Conclusions: Overall, the results showed that those individuals who obtained an RC after the intervention had higher levels of exposure to the platform. The usage during the first half of the intervention was higher, and differences between groups were observed from the first week. This study also showed specific usage levels associated with outcomes that could be tested in controlled studies to inform the minimal usage to establish adherence. These results will help to better understand how to use internet-based interventions and what optimal level of engagement can most affect outcomes. Trial Registration: ISRCTN Registry ISRCTN03704676; http://www.isrctn.com/ISRCTN03704676 International Registered Report Identifier (IRRID): RR2-10.1186/1471-244X-14-147
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures





Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Associations with adherence have been demonstrated for a range of outcomes, including depression [105,106] and anxiety [107,108].”
From The therapeutic alliance in digital mental health interventions for serious mental illnesses: narrative review (2020) · cites this paper as [105]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Eysenbach, G. (2005) · Journal of Medical Internet Research · doi:10.2196/jmir.7.1.e11 · Cited by 2,524 · cited 2×
“Although treatment dropout and spontaneous remission might explain some of these trends, as they could in other psychological interventions, understanding more completely how internet-delivered interventions are used and whether their usage differs depending on user characteristics is an important area of investigatio…”
Greiner, M., Pfeiffer, D., & Smith, R. D. (2000) · Preventive Veterinary Medicine · doi:10.1016/s0167-5877(00)00115-x · Cited by 1,923 · cited 1×
“To obtain the optimal cutoff for achieving RC, a receiver operating characteristic (ROC) curve analysis [34] was performed using each of the 4 individual usage metrics (total time spent on the platform, number of sessions, number of activities, and percentage of the program viewed) as test variables against the RC sta…”
Computer-based psychological treatments for depression: A systematic review and meta-analysis
Richards, D., & Richardson, T. (2012) · Clinical Psychology Review · doi:10.1016/j.cpr.2012.02.004 · Cited by 1,442 · cited 2×
“Different meta-analytic reviews have shown that supported interventions have higher rates of adherence and better outcomes than self-guided ones [1,3,4,25].”
Self-citationInternet-Based and Other Computerized Psychological Treatments for Adult Depression: A Meta-Analysis
Andersson, G., & Cuijpers, P. (2009) · Cognitive Behaviour Therapy · doi:10.1080/16506070903318960 · Cited by 1,368 · cited 2×
“Different meta-analytic reviews have shown that supported interventions have higher rates of adherence and better outcomes than self-guided ones [1,3,4,25].”
Persuasive System Design Does Matter: a Systematic Review of Adherence to Web-based Interventions
Kelders, S. M., Kok, R. N., Ossebaard, H. C., & Van Gemert-Pijnen, J. E. (2012) · Journal of Medical Internet Research · doi:10.2196/jmir.2104 · Cited by 1,308 · cited 2×
“This paper contributes to the concept of adherence by providing an empirical justification of intended use, that is, “the extent to which individuals should experience the content to derive maximum benefit from the intervention, as defined or implied by its creators” [22].”
Perski, O., Blandford, A., West, R., & Michie, S. (2017) · Translational Behavioral Medicine · doi:10.1007/s13142-016-0453-1 · Cited by 1,193 · cited 2×
“Our results confirm the largely used statement the more usage, the better [5,16], although it is important to note that our variables account for active (ie, activities completed) and passive (ie, percentage viewed) engagement, and both elements are important to get the most out of these interventions [20].”
Adherence in Internet Interventions for Anxiety and Depression
Christensen, H., Griffiths, K. M., & Farrer, L. (2009) · Journal of Medical Internet Research · doi:10.2196/jmir.1194 · Cited by 1,133 · cited 3×
“In this study, on average, the users utilized 57% of the program, which is similar to completion rates in other internet-based interventions for depression [36,39].”
Mohr, D. C., Cuijpers, P., & Lehman, K. (2011) · Journal of Medical Internet Research · doi:10.2196/jmir.1602 · Cited by 1,097 · cited 1×
“In this sense, no studies have explored the role of metrics related to the support, such as the number of reviews and the number of user replies to these reviews, which could shed more light on the role of support in the usage of the program and their outcomes [28].”
Bower, P., & Gilbody, S. (2005) · British Journal of Psychiatry · doi:10.1192/bjp.186.1.11 · Cited by 1,084
Andersson, G., Cuijpers, P., Carlbring, P., Riper, H., & Hedman, E. (2014) · World Psychiatry · doi:10.1002/wps.20151 · Cited by 1,077 · cited 1×
“Internet- and computer-based interventions for depression have been shown to be effective in several meta-analytic reviews, reporting comparable effects to face-to-face treatments [1-5].”
The impact of guidance on Internet-based mental health interventions — A systematic review
Baumeister, H., Reichler, L., Munzinger, M., & Lin, J. (2014) · Internet Interventions · doi:10.1016/j.invent.2014.08.003 · Cited by 1,037 · cited 2×
“improvements are barely understood and results so far do not show differences among different doses of support [24,26,27].”
Andrews, G., Basu, A., Cuijpers, P., Craske, M. G., McEvoy, P., English, C. L., & Newby, J. M. (2018) · Journal of Anxiety Disorders · doi:10.1016/j.janxdis.2018.01.001 · Cited by 922 · cited 1×
“Different meta-analytic reviews have shown that supported interventions have higher rates of adherence and better outcomes than self-guided ones [1,3,4,25].”
A Systematic Review of the Impact of Adherence on the Effectiveness of e-Therapies
Donkin, L., Christensen, H., Naismith, S. L., Neal, B., Hickie, I. B., & Glozier, N. (2011) · Journal of Medical Internet Research · doi:10.2196/jmir.1772 · Cited by 876 · cited 6×
“Previous research has revealed a relationship between how much users were actually exposed to the program and outcomes; nevertheless, as different metrics of usage have been used across studies, and results are somewhat mixed, the specific contributions of these variables to outcomes remain unclear [15,16].”
InfluentialReceiver operating characteristic (ROC) curve for medical researchers
Kumar, R., & Indrayan, A. (2011) · Indian Pediatrics · doi:10.1007/s13312-011-0055-4 · Cited by 872 · cited 1×
“The optimal cutoff was determined using the point of curve closest to the (0,1) criteria, which uses the formula d2=[(1–Sn)2+(1–Sp)2], where Sn = sensitivity and Sp = specificity, to calculate the distance of each point to the (0,1) point representing maximal sensitivity and specificity [35].”
Shimokawa, K., Lambert, M. J., & Smart, D. W. (2010) · Journal of Consulting and Clinical Psychology · doi:10.1037/a0019247 · Cited by 657 · cited 1×
“One of these strategies could be outcome and engagement monitoring, inbuilt in the feedback system, so that participants could be flagged if they were deviating from the expected results of the intervention, and causes for this could be explored and addressed [40].”
Titov, N., Andrews, G., Davies, M., McIntyre, K., Robinson, E., & Solley, K. (2010) · PLoS ONE · doi:10.1371/journal.pone.0010939 · Cited by 406 · cited 1×
“Similar criteria have been used in other studies of internet-delivered interventions for depression [19,32,33]. t tests were performed to explore differences in the usage metrics between those who obtained an RC and those who did not.”
Waller, R., & Gilbody, S. (2009) · Psychological Medicine · doi:10.1017/s0033291708004224 · Cited by 393 · cited 1×
“In this sense, some research has shown that the users’ uptake and long-term use of these technologies are lower than might be expected, and a median of 56% of the participants complete the whole program [11].”
Meyer, B., Berger, T., Caspar, F., Beevers, C. G., Andersson, G., & Weiss, M. (2009) · Journal of Medical Internet Research · doi:10.2196/jmir.1151 · Cited by 363 · cited 1×
“Similar criteria have been used in other studies of internet-delivered interventions for depression [19,32,33].”
Sieverink, F., Kelders, S. M., & van Gemert-Pijnen, J. E. (2017) · Journal of Medical Internet Research · doi:10.2196/jmir.8578 · Cited by 331 · cited 5×
“Previous research has revealed a relationship between how much users were actually exposed to the program and outcomes; nevertheless, as different metrics of usage have been used across studies, and results are somewhat mixed, the specific contributions of these variables to outcomes remain unclear [15,16].”
InfluentialComputer-Assisted Cognitive-Behavior Therapy for Depression: A Systematic Review and Meta-Analysis
Wright, J. H., Owen, J. J., Richards, D., Eells, T. D., Richardson, T., Brown, G. K., Barrett, M., Rasku, M. A., Polser, G., & Thase, M. E. (2019) · The Journal of Clinical Psychiatry · doi:10.4088/jcp.18r12188 · Cited by 200 · cited 2×
“Our results confirm the largely used statement the more usage, the better [5,16], although it is important to note that our variables account for active (ie, activities completed) and passive (ie, percentage viewed) engagement, and both elements are important to get the most out of these interventions [20].”
Self-citationDonkin, L., Hickie, I. B., Christensen, H., Naismith, S. L., Neal, B., Cockayne, N. L., & Glozier, N. (2013) · Journal of Medical Internet Research · doi:10.2196/jmir.2771 · Cited by 193 · cited 5×
“It should be noted that the time spent on the platform by those who achieved an RC was around 7 hours, which is slightly higher but still similar to the 6 hours found in previous studies, and the time spent by those who did not achieve an RC is similar with previous results reporting between 4 and 5 hours [21,37].”
InfluentialBendelin, N., Hesser, H., Dahl, J., Carlbring, P., Nelson, K. Z., & Andersson, G. (2011) · BMC Psychiatry · doi:10.1186/1471-244x-11-107 · Cited by 173 · cited 1×
“In this regard, it is possible to distinguish between active and passive engagement in internet-based interventions, where the former involves users interacting with the program and completing the activities, and the latter speaks about users who go through the program only superficially and therefore barely interacti…”
The Law of Attrition Revisited
Christensen, H., & Mackinnon, A. (2006) · Journal of Medical Internet Research · doi:10.2196/jmir.8.3.e20 · Cited by 161 · cited 1×
“However, other research has shown that the users do not necessarily need to go through the entire program to benefit clinically [12].”
Delgadillo, J., de Jong, K., Lucock, M., Lutz, W., Rubel, J., Gilbody, S., Ali, S., Aguirre, E., Appleton, M., Nevin, J., et al. (2018) · The Lancet Psychiatry · doi:10.1016/s2215-0366(18)30162-7 · Cited by 152 · cited 1×
“the intervention period, these patients had significantly less severe symptoms after treatment compared with similar patients assigned to therapists who were not receiving this feedback [41].”
Nordgren, L. B., Hedman, E., Etienne, J., Bodin, J., Kadowaki, Å., Eriksson, S., Lindkvist, E., Andersson, G., & Carlbring, P. (2014) · Behaviour Research and Therapy · doi:10.1016/j.brat.2014.05.007 · Cited by 145
Attrition Revisited: Adherence and Retention in a Web-Based Alcohol Trial
Murray, E., White, I. R., Varagunam, M., Godfrey, C., Khadjesari, Z., & McCambridge, J. (2013) · Journal of Medical Internet Research · doi:10.2196/jmir.2336 · Cited by 128 · cited 1×
“Moreover, the inclusion of only those participants who reported posttreatment outcomes might be limiting the generalizability of the results, as it has been found that participants who engage more are also more likely to respond to follow-up assessments [43].”
Internet‐Based Treatment for Panic Disorder: Does Frequency of Therapist Contact Make a Difference?
Klein, B., Austin, D., Pier, C., Kiropoulos, L., Shandley, K., Mitchell, J., Gilson, K., & Ciechomski, L. (2009) · Cognitive Behaviour Therapy · doi:10.1080/16506070802561132 · Cited by 123 · cited 1×
“improvements are barely understood and results so far do not show differences among different doses of support [24,26,27].”
Rollman, B. L., Herbeck Belnap, B., Abebe, K. Z., Spring, M. B., Rotondi, A. J., Rothenberger, S. D., & Karp, J. F. (2018) · JAMA Psychiatry · doi:10.1001/jamapsychiatry.2017.3379 · Cited by 118
Approaches for Informing Optimal Dose of Behavioral Interventions
Voils, C. I., King, H. A., Maciejewski, M. L., Allen, K. D., Yancy, W. S., & Shaffer, J. A. (2014) · Annals of Behavioral Medicine · doi:10.1007/s12160-014-9618-7 · Cited by 117 · cited 3×
“Therefore, finding the most relevant metrics and determining thresholds of usage for these metrics are key to determining the optimal dosage, which in turn may be different depending on the target population, setting, or the type of intervention that is evaluated [23].”
InfluentialRichards, D., Timulak, L., O'Brien, E., Hayes, C., Vigano, N., Sharry, J., & Doherty, G. (2015) · Behaviour Research and Therapy · doi:10.1016/j.brat.2015.10.005 · Cited by 109
Self-citationHan, J. Y., Kim, J.-H., Yoon, H. J., Shim, M., McTavish, F. M., & Gustafson, D. H. (2012) · Journal of Health Communication · doi:10.1080/10810730.2011.585696 · Cited by 109 · cited 1×
“In this regard, it is possible to distinguish between active and passive engagement in internet-based interventions, where the former involves users interacting with the program and completing the activities, and the latter speaks about users who go through the program only superficially and therefore barely interacti…”
The association between adherence and outcome in an Internet intervention for depression
Fuhr, K., Schröder, J., Berger, T., Moritz, S., Meyer, B., Lutz, W., Hohagen, F., Hautzinger, M., & Klein, J. P. (2018) · Journal of Affective Disorders · doi:10.1016/j.jad.2017.12.028 · Cited by 97 · cited 1×
“In recent years, several studies have turned their attention to the usage of internet-delivered intervention and how usage is related to outcomes [10,14].”
Aardoom, J. J., Dingemans, A. E., Spinhoven, P., van Ginkel, J. R., de Rooij, M., & van Furth, E. F. (2016) · Journal of Medical Internet Research · doi:10.2196/jmir.5709 · Cited by 93 · cited 1×
“improvements are barely understood and results so far do not show differences among different doses of support [24,26,27].”
Kelders, S. M., Bohlmeijer, E. T., & Van Gemert-Pijnen, J. E. (2013) · Journal of Medical Internet Research · doi:10.2196/jmir.2258 · Cited by 88 · cited 2×
“In this study, on average, the users utilized 57% of the program, which is similar to completion rates in other internet-based interventions for depression [36,39].”
Opening the Black Box of Electronic Health: Collecting, Analyzing, and Interpreting Log Data
Sieverink, F., Kelders, S., Poel, M., & van Gemert-Pijnen, L. (2017) · JMIR Research Protocols · doi:10.2196/resprot.6452 · Cited by 88 · cited 2×
“In recent years, several studies have turned their attention to the usage of internet-delivered intervention and how usage is related to outcomes [10,14].”
de Graaf, L. E., Huibers, M. J. H., Riper, H., Gerhards, S. A. H., & Arntz, A. (2009) · Journal of Affective Disorders · doi:10.1016/j.jad.2008.12.009 · Cited by 86
Mattila, E., Lappalainen, R., Välkkynen, P., Sairanen, E., Lappalainen, P., Karhunen, L., Peuhkuri, K., Korpela, R., Kolehmainen, M., & Ermes, M. (2016) · JMIR mhealth and uhealth · doi:10.2196/mhealth.5241 · Cited by 67 · cited 4×
“Overall, these results are in line with another study, where the program was mostly used during the first half of the intervention period [37].”
InfluentialKolovos, S., van Dongen, J. M., Riper, H., Buntrock, C., Cuijpers, P., Ebert, D. D., Geraedts, A. S., Kenter, R. M., Nobis, S., Smith, A., et al. (2018) · Depression and Anxiety · doi:10.1002/da.22714 · Cited by 63 · cited 1×
“The inclusion of these interventions into mental health services may lead to benefits, such as reduced costs associated with treatment, reduced waiting-list burden, and increased accessibility to services [7,8]; however, results are mixed, and more research is needed to draw firmer conclusions [9].”
Analyzing Engagement in a Web-Based Intervention Platform Through Visualizing Log-Data
Morrison, C., & Doherty, G. (2014) · Journal of Medical Internet Research · doi:10.2196/jmir.3575 · Cited by 62 · cited 1×
“Although our results will help identify optimal levels of exposure to maximize the benefits from these interventions, future studies should be conducted to go deeper into the usage patterns, as they can take many different pathways, and different types of usage can lead to benefits [42].”
Wells, M. J., Owen, J. J., McCray, L. W., Bishop, L. B., Eells, T. D., Brown, G. K., Richards, D., Thase, M. E., & Wright, J. H. (2018) · The Primary Care Companion For CNS Disorders · doi:10.4088/pcc.17r02196 · Cited by 46 · cited 2×
“Different meta-analytic reviews have shown that supported interventions have higher rates of adherence and better outcomes than self-guided ones [1,3,4,25].”
Self-citationRichards, D., Timulak, L., Doherty, G., Sharry, J., Colla, A., Joyce, C., & Hayes, C. (2014) · BMC Psychiatry · doi:10.1186/1471-244x-14-147 · Cited by 35 · cited 3×
“Space from Depression has been described in detail elsewhere [29,30].”
Self-citationEarley, C., Joyce, C., McElvaney, J., Richards, D., & Timulak, L. (2017) · Internet Interventions · doi:10.1016/j.invent.2017.07.003 · Cited by 19 · cited 1×
“Still, it is worth mentioning that although the literature in this regard shows mixed results [36], a related study identified important benefits of the Space from Depression program for those with subclinical symptoms [38].”
Self-citationInternet Delivered Treatment for symptoms of Depression in the community
Richards, D. (2013) · doi:10.1186/isrctn03704676 · Cited by 2
Self-citation
Research works printed but not linked: 1
- Beck, AT, 1996, BDI-II, Beck Depression Inventory: Manual
Cited by
Digital interventions for the treatment of depression: a meta-analytic review
Moshe, I., Terhorst, Y., Philippi, P., Domhardt, M., Cuijpers, P., Cristea, I., Pulkki-Råback, L., Baumeister, H., & Sander, L. B. (2021) · Psychological Bulletin · doi:10.1037/bul0000334 · Cited by 390
Tremain, H., McEnery, C., Fletcher, K., & Murray, G. (2020) · JMIR Mental Health · doi:10.2196/17204 · Cited by 193
Richards, D., Enrique, A., Eilert, N., Franklin, M., Palacios, J., Duffy, D., Earley, C., Chapman, J., Jell, G., Sollesse, S., et al. (2020) · npj Digital Medicine · doi:10.1038/s41746-020-0293-8 · Cited by 154
Self-citationChien, I., Enrique, A., Palacios, J., Regan, T., Keegan, D., Carter, D., Tschiatschek, S., Nori, A., Thieme, A., Richards, D., et al. (2020) · JAMA Network Open · doi:10.1001/jamanetworkopen.2020.10791 · Cited by 111
Deisenhofer, A., Barkham, M., Beierl, E. T., Schwartz, B., Doorn, K. A., Beevers, C. G., Berwian, I. M., Blackwell, S. E., Bockting, C., Brakemeier, E., et al. (2023) · Behaviour Research and Therapy · doi:10.1016/j.brat.2023.104443 · Cited by 138
“In addition to active data collection, these tools may also be used to gather passive data on granular and objective treatment engagement, such as the tools and pages viewed during a digital therapy session (Enrique, Palacios, Ryan, & Richards, 2019).”
Method
Metadata
- Authors
- Enrique, A., Palacios, J. E., Ryan, H., & Richards, D.
- Affiliations
- SilverCloud (Ireland) (IE)
- Venue
- Journal of Medical Internet Research
- Publisher
- JMIR Publications Inc.
- Published
- 2019-08-01
- Volume
- 21
- Issue
- 8
- Pages
- e12775
- ISSN
- 1438-8871 (ISSN-L)
- PMC
- PMC6694731
- OpenAlex
- W2950346625
- Cited by
- 81
- References
- 43
- Citations by year
- 2026: 3 · 2025: 12 · 2024: 10 · 2023: 8 · 2022: 24 · 2021: 11 · 2020: 13
- FWCI
- 13.50
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- Locations
- Journal of Medical Internet Research (publishedVersion, cc-by)
- Europe PMC (PubMed Central) (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Arrow@dit (Dublin Institute of Technology) (submittedVersion, cc-by-nc-sa)
- Trinity's Access to Research Output (TARA) (Trinity College Dublin) (submittedVersion, cc-by)
- Topics
- Social Sciences › Psychology › Applied Psychology › Digital Mental Health Interventions
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Anxiety, Depression, Psychometrics, Treatment, Cognitive Processes
- Health Sciences › Medicine › Psychiatry and Mental health › Mental Health Treatment and Access
- Keywords
- online intervention, usage metrics, program completion, reliable change, medication adherence, engagement, digital mental health interventions, depressive symptoms, community-dwelling adults
- MeSH terms
- Internet-Based Intervention, Adult, Depression, Depressive Disorder, Female, Humans, Male, Time Factors, Cognitive Behavioral Therapy, Randomized Controlled Trials as Topic, Treatment Outcome
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.