The DOI matches its Crossref record; 60 of the 60 references it cites were found in a registry or an index.
Is there a trial bias impacting user engagement with unguided e-mental health interventions? A systematic comparison of published reports and real-world usage of the same programs
Abstract
Trial settings that include proactive recruitment, human contact, and assessment procedures may substantially impact the way users engage with unguided e-mental health programs and the generalizability of reported findings. This study examined the impact of trial setting on user behavior by directly comparing reported user engagement in trial-based research and objective measures of real-world usage of the same unguided mental health programs. The authors conducted a systematic search for papers reporting user engagement with off-the-shelf unguided e-mental health programs. Real-world usage was obtained from a panel that presents aggregated nonpersonal information on user engagement with digital programs across the world. A total of 13 papers yielding 14 comparable usage metrics met all inclusion criteria. In three papers reporting the use of programs by lay users without any proactive trial procedures, the ratios calculated by dividing the usage reported in the paper by the usage documented within the objective dataset were 0.84, 1.05, and 1.27—suggesting a sufficient criterion validity for our examination. In studies that proactively recruited users and included pre- to post-assessment procedures (11 comparisons), the median program usage rate reported was 4.06 times higher (IQR = 4.49) than the real-world usage of the same program. Severity of clinical symptoms, in-person versus remote assessment procedures, study design, and program cost had no impact on these differences. The results suggest that trial settings have a large impact on user engagement with unguided interventions and, therefore, on the generalizability of the findings to the real world.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“real-world engagement in these W-MHIs has remained low compared with engagement reported in research studies.”
From Young People’s Barriers and Facilitators of Engagement with Web-Based Mental Health Interventions for Anxiety and Depression: A Qualitative Study (2024) · cites this paper as [12]
In the citing paper
“It should also be taken into consideration that by way of including only RCTs, results may reflect online intervention use under more ideal conditions as compared to natural use (Baumel et al., 2019).”
From A systematic review and meta-analysis of self-guided online acceptance and commitment therapy as a transdiagnostic self-help intervention (2023) · cites this paper as Baumel et al., 2019
In the citing paper
“Similarly, Baumel et al. (2019) calculated that there is a research bias regarding engagement estimates with intervention usage being four times higher in trials compared with the same interventions in real-world.”
From Special issue on digital health interventions in chronic medical conditions: Editorial (2022) · cites this paper as Baumel et al., 2019
In the citing paper
“Additionally, the presence of reminders from the research team could encourage app uptake beyond what might be expected in a naturalistic setting (Baumel et al. 2019).”
From Evaluating the quality, safety, and functionality of commonly used smartphone apps for bipolar disorder mood and sleep self-management (2022) · cites this paper as Baumel et al. 2019
In the citing paper
“engagement with these technologies remains to be an ongoing issue, varies from study to study, and is typically lower in real-world use than research studies [9]”
From Barriers to and facilitators of user engagement with digital mental health interventions: Systematic review (2021) · cites this paper as [9]
In the citing paper
“However, these analyses did not report on factors related to this variability in engagement.”
From Barriers to and facilitators of user engagement with digital mental health interventions: Systematic review (2021) · cites this paper as [9]
In the citing paper
“systematic comparisons in 2018 and 2019 on self-help DMHIs found that real-world uptake varies widely [9,11]”
From Barriers to and facilitators of user engagement with digital mental health interventions: Systematic review (2021) · cites this paper as [9]
In the citing paper
“Previous reviews have highlighted the variability in engagement and uptake, analyzing both DMHIs published in the academic literature [9-11] and publicly available mental health apps in app stores [16].”
From Barriers to and facilitators of user engagement with digital mental health interventions: Systematic review (2021) · cites this paper as [9]
In the citing paper
“In practice, many existing DMHIs have high attrition rates and low user engagement, especially when accessed in stand-alone self-guided formats [25,26] and in “real-world” naturalistic settings outside of controlled research studies [27].”
Citations
Cites
The world health report 2001 - Mental health: new understanding, new hope
Sayers, J. (2001) · Europe PMC (PubMed Central) · doi:10.1590/s0042-96862001001100014 · Cited by 3,307
A behavior model for persuasive design
Fogg, B. (2009) · Proceedings of the 4th International Conference on Persuasive Technology · doi:10.1145/1541948.1541999 · Cited by 2,328
Adherence in Internet Interventions for Anxiety and Depression
Christensen, H., Griffiths, K. M., & Farrer, L. (2009) · Journal of Medical Internet Research · doi:10.2196/jmir.1194 · Cited by 1,133
Mohr, D. C., Cuijpers, P., & Lehman, K. (2011) · Journal of Medical Internet Research · doi:10.2196/jmir.1602 · Cited by 1,097
The impact of guidance on Internet-based mental health interventions — A systematic review
Baumeister, H., Reichler, L., Munzinger, M., & Lin, J. (2014) · Internet Interventions · doi:10.1016/j.invent.2014.08.003 · Cited by 1,037
Delivering interventions for depression by using the internet: randomised controlled trial
Christensen, H. (2004) · BMJ · doi:10.1136/bmj.37945.566632.ee · Cited by 928
Healthcare via Cell Phones: A Systematic Review
Krishna, S., Boren, S. A., & Balas, E. A. (2009) · Telemedicine and e-Health · doi:10.1089/tmj.2008.0099 · Cited by 882
Personal Sensing: Understanding Mental Health Using Ubiquitous Sensors and Machine Learning
Mohr, D. C., Zhang, M., & Schueller, S. M. (2017) · Annual Review of Clinical Psychology · doi:10.1146/annurev-clinpsy-032816-044949 · Cited by 802
Fleming, T., Bavin, L., Lucassen, M., Stasiak, K., Hopkins, S., & Merry, S. (2018) · Journal of Medical Internet Research · doi:10.2196/jmir.9275 · Cited by 670
Flagged first, then by citations
Research works printed but not linked: 4
- Rainie, 2010, Internet, Broadband, and Cell Phone Statistics, 5, 3
- Bezerra, 2015, The Mobile Revolution: How Mobile Technologies Drive a Trillion-dollar Impact
- Christensen, 2009, Adherence in internet interventions for anxiety and depression, J Med Internet Res., 11, e13, 10.2196/jmir.1194 · DOI · Paper page
- iProspect, 2009, Search Engine User Behavior Study
Cited by
Sorted by citations
Items 101–143 of 143
Etzelmueller, A., Heber, E., Horvath, H., Radkovsky, A., Lehr, D., & Ebert, D. D. (2024) · Journal of Medical Internet Research · doi:10.2196/42976 · Cited by 3
Hawkes, R. E., Marsden, A., Cotterill, S., Benton, J. S., & French, D. (2025) · Social Science & Medicine · doi:10.1016/j.socscimed.2025.117858 · Cited by 2
Piccolo, M. C. D., Hammoud, R., Khan, M., D'Costa, V. S., Almuqrin, A., Georgiades, A., Tognin, S., & Mechelli, A. (2025) · Journal of Medical Internet Research · doi:10.2196/71989 · Cited by 2
Baumel, A., & Tamir, S. (2022) · Journal of Technology in Human Services · doi:10.1080/15228835.2022.2036300 · Cited by 2
Self-citationYoung adolescents’ interest in a mental health casual video game
Pine, R., Sutcliffe, K., McCallum, S., & Fleming, T. M. (2020) · doi:10.26686/wgtn.12904523.v1 · Cited by 2
Weiner, L. S., Crowley, R. N., Sheeber, L., Koegler, F. H., Davis, J. F., Wells, M., Funkhouser, C. J., Auerbach, R. P., & Allen, N. B. (2024) · doi:10.31234/osf.io/5tjrw · Cited by 2
Digital interventions for mental health care
Naslund, J. A., & Babalola, D. (2025) · Elsevier eBooks · doi:10.1016/b978-0-443-15728-8.00023-9 · Cited by 1
Klimczak, K. S., Twohig, M. P., Peacock, G. G., & Levin, M. E. (2025) · Journal of Contextual Behavioral Science · doi:10.1016/j.jcbs.2025.100922 · Cited by 2
Levin, M. E., Woolley, M. G., Bowers, E. M., & Klimczak, K. S. (2025) · Journal of Contextual Behavioral Science · doi:10.1016/j.jcbs.2025.100929 · Cited by 1
A Scoping Review of Digital Interventions Targeting Anxiety in Young Children
Coverdale, G., Perry, Y., Magiati, I., Badcock, N. A., Bollig, B., & Alvares, G. A. (2025) · Child & Youth Care Forum · doi:10.1007/s10566-025-09899-9 · Cited by 2
A literature review of remote mental health screening: barriers, potential solutions, and tools
Quiram, N., Salam, T., Sadjadpour, F., Hosseinichimeh, N., Jarvis, L. R., & Soghier, L. M. (2025) · Frontiers in Digital Health · doi:10.3389/fdgth.2025.1670691 · Cited by 2
Alsaqqa, H. H. (2025) · Journal of Multidisciplinary Healthcare · doi:10.2147/jmdh.s544368 · Cited by 1
Ramia, J. A., Khoury, S., Armani, N., Cuijpers, P. J., Carswell, K., Hof, E. V., Zoghbi, E., Sijbrandij, M., & Chammay, R. E. (2025) · Frontiers in Public Health · doi:10.3389/fpubh.2025.1665093 · Cited by 1
Seeking Late Night Life Lines: Experiences of Conversational AI Use in Mental Health Crisis
Ajmani, L., Ghosh, A., Kaveladze, B. T., Kim, E., Namuduri, K., Nguyen, T., Okoli, E., Schleider, J. L., Ford, D., & Suh, J. (2026) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3805689.3812256 · Cited by 6
Gugel, J., Rauschenberg, C., Paetzold, I., Schick, A., Wierzba, E., Hirjak, D., Boehnke, J. R., Boecking, B., & Reininghaus, U. (2026) · Journal of Medical Internet Research · doi:10.2196/77150 · Cited by 2
Kulke, J. K., Nacke, B., & Jacobi, C. (2023) · Mental Health Science · doi:10.1002/mhs2.27 · Cited by 1
Hoeppner, B. B., Siegel, K. R., Carlon, H. A., Kahler, C. W., Park, E. R., Taylor, S. T., Simpson, H. V., & Hoeppner, S. S. (2022) · doi:10.2196/preprints.38234 · Cited by 1
Single-session interventions embedded within Tumblr: Acceptability, feasibility, and utility studyPreprint
Dobias, M., Morris, R. R., & Schleider, J. L. (2022) · doi:10.31234/osf.io/r93bn · Cited by 1
Gugel, J., Paetzold, I., Rauschenberg, C., Schick, A., Hirjak, D., Boehnke, J. R., Boecking, B., & Reininghaus, U. (2025) · doi:10.2196/preprints.77150 · Cited by 1
Michalak, E. E., Barnes, S. J., Morton, E. E., O'Brien, H. L., Murray, G., Hole, R., & Meyer, D. (2022) · doi:10.2196/preprints.36213
Reich, H., Hegerl, U., Rosenthal, A., & Oehler, C. (2023) · doi:10.31234/osf.io/fsxbd
An App-based CBT Program tailored for College Students: A Randomized Controlled Trial (Preprint)Preprint
Kim, M. H., & Chung, K. (2024) · doi:10.2196/preprints.50006
Pierce, B. G., Twohig, M. P., & Levin, M. E. (2025) · Journal of College Student Mental Health · doi:10.1080/28367138.2024.2409244
García-Batista, Z. E., Vindel, A. C., Guerra-Peña, K., Álvarez-Hernández, A., Cruz, I., Amparo, Ó. R., & Medrano, L. A. (2025) · Suma Psicológica · doi:10.14349/sumapsi.2025.v32.n2.2
Guebey, J., Gosetto, L., Rehberg, B., Lovis, C., Ehrler, F., & Molinard‐Chenu, A. (2025) · medRxiv · doi:10.1101/2025.10.28.25338623
Parker, R., Padfield, P. L., Hanley, J., Pinnock, H., Kennedy, J., Stoddart, A., Hammersley, V., Sheikh, A., & McKinstry, B. (2021) · Figshare · doi:10.6084/m9.figshare.13889173
Taylor, K. M., Ward, T., Cella, M., Garety, P. A., Grant, A. R., Philps, K., Wojdecka, A. B., Flory, M., & Hardy, A. (2026) · doi:10.2196/preprints.91188
A Chronic Pain Self-Management Mobile App (Dolodoc): Cross-Sectional Acceptability Study
Guebey, J., Gosetto, L., Rehberg, B., Lovis, C., Ehrler, F., & Molinard‐Chenu, A. (2026) · JMIR Human Factors · doi:10.2196/77163
Levin, M. E., Knudsen, F. M., Klimczak, K. S., Nettleton, S., Donahue, M. L., & Aller, T. B. (2026) · Cognitive Behaviour Therapy · doi:10.1080/16506073.2026.2642732
Trueman, C., Viskovich, S., & Fowler, J. A. (2026) · Australian psychologist · doi:10.1080/00050067.2026.2671115
Marsden, A., Hawkes, R. E., Benton, J. S., French, D., & Cotterill, S. (2026) · Journal of Medical Internet Research · doi:10.2196/89690
Fowler, J. A., Buckley, L., Viskovich, S., Muir, M., & Dean, J. A. (2026) · Australian Psychologist · doi:10.1080/00050067.2026.2683191
Madrid‐Cagigal, A., Ryan, L., Kealy, C., McNulty, J., Mulvenna, M. D., Byrne, M., Barry, M. M., & Donohoe, G. J. (2026) · Behaviour and Information Technology · doi:10.1080/0144929x.2026.2675626 · Cited by 1
Kuhlmeier, F. O., Bernecker, S. L., Bauch, L., Reason, R., Torres‐Latorre, C., Akın, R. I., & Großmann, I. (2026) · doi:10.2196/preprints.104491
Romanelli, M., Nguyen, T., Rushton, K., & Marion, J. L. (2026) · JMIR Research Protocols · doi:10.2196/97126
Taylor, K. M., Ward, T., Cella, M., Garety, P. A., Grant, A. R., Philps, K., Wojdecka, A. B., Flory, M., & Hardy, A. (2026) · JMIR Human Factors · doi:10.2196/91188
Bolpagni, M., Piffer, S. A., Pozza, M., Maminimini, M., & Gabrielli, S. (2026) · JMIR Human Factors · doi:10.2196/89876
Kempe, L., Iakimova, G., & Graziani, P. (2026) · Advances in Mental Health · doi:10.1080/18387357.2026.2733116
Barešová, Z., Lekárová, M., Světlák, M., Malatincová, T., Vigašová, D., Šedo, J., Halámková, J., Slezáčková, A., Šumec, R., Humpolíček, P., et al. (2026) · doi:10.2196/preprints.112800
Determinants of Successful Mobile Smoking Cessation Interventions: Cross-Sectional Study
Pradhan, B. K., Lukavská, K., Gabrhelík, R., & Novák, D. (2026) · JMIR Formative Research · doi:10.2196/80428
Metadata
- Authors
- Baumel, A., Edan, S., & Kane, J. M.
- Affiliations
- University of Haifa (IL)
- Northwell Health (US)
- Corresponding author
- Baumel, Amit
- Venue
- Translational Behavioral Medicine
- Publisher
- Oxford University Press (OUP)
- Published
- 2019-11-05
- Volume
- 9
- Issue
- 6
- Pages
- 1020-1033
- ISSN
- 1613-9860 (ISSN-L)
- 1869-6716
- OpenAlex
- W2989270862
- Cited by
- 143
- References
- 60
- Citations by year
- 2026: 17 · 2025: 38 · 2024: 26 · 2023: 17 · 2022: 21 · 2021: 14 · 2020: 10
- FWCI
- 12.74
- Type
- journal article
- Language
- en
- Open access
- closed
- Licence
- academic.oup.com/journals/pages/open_access/funder_policies/chorus/standard_publication_model (vor)
- Locations
- Translational Behavioral Medicine (publishedVersion)
- PubMed (publishedVersion)
- Topics
- Social Sciences › Psychology › Applied Psychology › Digital Mental Health Interventions
- Health Sciences › Health Professions › General Health Professions › Mobile Health and mHealth Applications
- Health Sciences › Medicine › Public Health, Environmental and Occupational Health › Telemedicine and Telehealth Implementation
- Keywords
- trial bias, user engagement
- MeSH terms
- Clinical Trials as Topic, Humans, Mental Health, Program Development, Telemedicine
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
