The DOI matches its Crossref record; 60 of the 60 references it cites were found in a registry or an index.
Is there a trial bias impacting user engagement with unguided e-mental health interventions? A systematic comparison of published reports and real-world usage of the same programs
Abstract
Trial settings that include proactive recruitment, human contact, and assessment procedures may substantially impact the way users engage with unguided e-mental health programs and the generalizability of reported findings. This study examined the impact of trial setting on user behavior by directly comparing reported user engagement in trial-based research and objective measures of real-world usage of the same unguided mental health programs. The authors conducted a systematic search for papers reporting user engagement with off-the-shelf unguided e-mental health programs. Real-world usage was obtained from a panel that presents aggregated nonpersonal information on user engagement with digital programs across the world. A total of 13 papers yielding 14 comparable usage metrics met all inclusion criteria. In three papers reporting the use of programs by lay users without any proactive trial procedures, the ratios calculated by dividing the usage reported in the paper by the usage documented within the objective dataset were 0.84, 1.05, and 1.27—suggesting a sufficient criterion validity for our examination. In studies that proactively recruited users and included pre- to post-assessment procedures (11 comparisons), the median program usage rate reported was 4.06 times higher (IQR = 4.49) than the real-world usage of the same program. Severity of clinical symptoms, in-person versus remote assessment procedures, study design, and program cost had no impact on these differences. The results suggest that trial settings have a large impact on user engagement with unguided interventions and, therefore, on the generalizability of the findings to the real world.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“real-world engagement in these W-MHIs has remained low compared with engagement reported in research studies.”
From Young People’s Barriers and Facilitators of Engagement with Web-Based Mental Health Interventions for Anxiety and Depression: A Qualitative Study (2024) · cites this paper as [12]
In the citing paper
“It should also be taken into consideration that by way of including only RCTs, results may reflect online intervention use under more ideal conditions as compared to natural use (Baumel et al., 2019).”
From A systematic review and meta-analysis of self-guided online acceptance and commitment therapy as a transdiagnostic self-help intervention (2023) · cites this paper as Baumel et al., 2019
In the citing paper
“Similarly, Baumel et al. (2019) calculated that there is a research bias regarding engagement estimates with intervention usage being four times higher in trials compared with the same interventions in real-world.”
From Special issue on digital health interventions in chronic medical conditions: Editorial (2022) · cites this paper as Baumel et al., 2019
In the citing paper
“Additionally, the presence of reminders from the research team could encourage app uptake beyond what might be expected in a naturalistic setting (Baumel et al. 2019).”
From Evaluating the quality, safety, and functionality of commonly used smartphone apps for bipolar disorder mood and sleep self-management (2022) · cites this paper as Baumel et al. 2019
In the citing paper
“engagement with these technologies remains to be an ongoing issue, varies from study to study, and is typically lower in real-world use than research studies [9]”
From Barriers to and facilitators of user engagement with digital mental health interventions: Systematic review (2021) · cites this paper as [9]
In the citing paper
“However, these analyses did not report on factors related to this variability in engagement.”
From Barriers to and facilitators of user engagement with digital mental health interventions: Systematic review (2021) · cites this paper as [9]
In the citing paper
“systematic comparisons in 2018 and 2019 on self-help DMHIs found that real-world uptake varies widely [9,11]”
From Barriers to and facilitators of user engagement with digital mental health interventions: Systematic review (2021) · cites this paper as [9]
In the citing paper
“Previous reviews have highlighted the variability in engagement and uptake, analyzing both DMHIs published in the academic literature [9-11] and publicly available mental health apps in app stores [16].”
From Barriers to and facilitators of user engagement with digital mental health interventions: Systematic review (2021) · cites this paper as [9]
In the citing paper
“In practice, many existing DMHIs have high attrition rates and low user engagement, especially when accessed in stand-alone self-guided formats [25,26] and in “real-world” naturalistic settings outside of controlled research studies [27].”
Citations
Cites
The world health report 2001 - Mental health: new understanding, new hope
Sayers, J. (2001) · Europe PMC (PubMed Central) · doi:10.1590/s0042-96862001001100014 · Cited by 3,307
A behavior model for persuasive design
Fogg, B. (2009) · Proceedings of the 4th International Conference on Persuasive Technology · doi:10.1145/1541948.1541999 · Cited by 2,328
Adherence in Internet Interventions for Anxiety and Depression
Christensen, H., Griffiths, K. M., & Farrer, L. (2009) · Journal of Medical Internet Research · doi:10.2196/jmir.1194 · Cited by 1,133
Mohr, D. C., Cuijpers, P., & Lehman, K. (2011) · Journal of Medical Internet Research · doi:10.2196/jmir.1602 · Cited by 1,097
The impact of guidance on Internet-based mental health interventions — A systematic review
Baumeister, H., Reichler, L., Munzinger, M., & Lin, J. (2014) · Internet Interventions · doi:10.1016/j.invent.2014.08.003 · Cited by 1,037
Delivering interventions for depression by using the internet: randomised controlled trial
Christensen, H. (2004) · BMJ · doi:10.1136/bmj.37945.566632.ee · Cited by 928
Healthcare via Cell Phones: A Systematic Review
Krishna, S., Boren, S. A., & Balas, E. A. (2009) · Telemedicine and e-Health · doi:10.1089/tmj.2008.0099 · Cited by 882
Personal Sensing: Understanding Mental Health Using Ubiquitous Sensors and Machine Learning
Mohr, D. C., Zhang, M., & Schueller, S. M. (2017) · Annual Review of Clinical Psychology · doi:10.1146/annurev-clinpsy-032816-044949 · Cited by 802
Fleming, T., Bavin, L., Lucassen, M., Stasiak, K., Hopkins, S., & Merry, S. (2018) · Journal of Medical Internet Research · doi:10.2196/jmir.9275 · Cited by 670
Research works printed but not linked: 4
- Rainie, 2010, Internet, Broadband, and Cell Phone Statistics, 5, 3
- Bezerra, 2015, The Mobile Revolution: How Mobile Technologies Drive a Trillion-dollar Impact
- Christensen, 2009, Adherence in internet interventions for anxiety and depression, J Med Internet Res., 11, e13, 10.2196/jmir.1194 · DOI · Paper page
- iProspect, 2009, Search Engine User Behavior Study
Cited by
Sorted by citations
Borghouts, J., Eikey, E., Mark, G., De Leon, C., Schueller, S. M., Schneider, M., Stadnick, N., Zheng, K., Mukamel, D., & Sorkin, D. H. (2021) · Journal of Medical Internet Research · doi:10.2196/24387 · Cited by 1,069
“Similarly, systematic comparisons in 2018 and 2019 on self-help DMHIs found that real-world uptake varies widely [9,11], and acceptability can be lower than traditional treatment [12].”
BackgroundJakob, R., Harperink, S., Rudolf, A. M., Fleisch, E., Haug, S., Mair, J. L., Salamanca-Sanabria, A., & Kowatsch, T. (2022) · Journal of Medical Internet Research · doi:10.2196/35371 · Cited by 446
Gál, É., Ștefan, S., & Cristea, I. A. (2021) · Journal of Affective Disorders · doi:10.1016/j.jad.2020.09.134 · Cited by 289
Lipschitz, J. M., Pike, C. K., Hogan, T. P., Murphy, S. A., & Burdick, K. E. (2023) · Current Treatment Options in Psychiatry · doi:10.1007/s40501-023-00297-3 · Cited by 241
Engagement and retention in digital mental health interventions: a narrative review
Boucher, E. M., & Raiker, J. S. (2024) · BMC Digital Health · doi:10.1186/s44247-024-00105-9 · Cited by 218
Metadata
- Authors
- Baumel, A., Edan, S., & Kane, J. M.
- Affiliations
- University of Haifa (IL)
- Northwell Health (US)
- Corresponding author
- Baumel, Amit
- Venue
- Translational Behavioral Medicine
- Publisher
- Oxford University Press (OUP)
- Published
- 2019-11-05
- Volume
- 9
- Issue
- 6
- Pages
- 1020-1033
- ISSN
- 1613-9860 (ISSN-L)
- 1869-6716
- OpenAlex
- W2989270862
- Cited by
- 143
- References
- 60
- Citations by year
- 2026: 17 · 2025: 38 · 2024: 26 · 2023: 17 · 2022: 21 · 2021: 14 · 2020: 10
- FWCI
- 12.74
- Type
- journal article
- Language
- en
- Open access
- closed
- Licence
- academic.oup.com/journals/pages/open_access/funder_policies/chorus/standard_publication_model (vor)
- Locations
- Translational Behavioral Medicine (publishedVersion)
- PubMed (publishedVersion)
- Topics
- Social Sciences › Psychology › Applied Psychology › Digital Mental Health Interventions
- Health Sciences › Health Professions › General Health Professions › Mobile Health and mHealth Applications
- Health Sciences › Medicine › Public Health, Environmental and Occupational Health › Telemedicine and Telehealth Implementation
- Keywords
- trial bias, user engagement
- MeSH terms
- Clinical Trials as Topic, Humans, Mental Health, Program Development, Telemedicine
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
