Is there a trial bias impacting user engagement with unguided e-mental health interventions? A systematic comparison of published reports and real-world usage of the same programs
Abstract
Trial settings that include proactive recruitment, human contact, and assessment procedures may substantially impact the way users engage with unguided e-mental health programs and the generalizability of reported findings. This study examined the impact of trial setting on user behavior by directly comparing reported user engagement in trial-based research and objective measures of real-world usage of the same unguided mental health programs. The authors conducted a systematic search for papers reporting user engagement with off-the-shelf unguided e-mental health programs. Real-world usage was obtained from a panel that presents aggregated nonpersonal information on user engagement with digital programs across the world. A total of 13 papers yielding 14 comparable usage metrics met all inclusion criteria. In three papers reporting the use of programs by lay users without any proactive trial procedures, the ratios calculated by dividing the usage reported in the paper by the usage documented within the objective dataset were 0.84, 1.05, and 1.27—suggesting a sufficient criterion validity for our examination. In studies that proactively recruited users and included pre- to post-assessment procedures (11 comparisons), the median program usage rate reported was 4.06 times higher (IQR = 4.49) than the real-world usage of the same program. Severity of clinical symptoms, in-person versus remote assessment procedures, study design, and program cost had no impact on these differences. The results suggest that trial settings have a large impact on user engagement with unguided interventions and, therefore, on the generalizability of the findings to the real world.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Real-world engagement in W-MHIs has remained low compared with engagement reported in research studies.
“real-world engagement in these W-MHIs has remained low compared with engagement reported in research studies.”
From Young People’s Barriers and Facilitators of Engagement with Web-Based Mental Health Interventions for Anxiety and Depression: A Qualitative Study (2024) · cites this paper as [12]
By including only RCTs, results may reflect online intervention use under more ideal conditions as compared to natural use.
“It should also be taken into consideration that by way of including only RCTs, results may reflect online intervention use under more ideal conditions as compared to natural use (Baumel et al., 2019).”
From A systematic review and meta-analysis of self-guided online acceptance and commitment therapy as a transdiagnostic self-help intervention (2023) · cites this paper as Baumel et al., 2019
There is a research bias regarding engagement estimates, with intervention usage being four times higher in trials compared with the same interventions in real-world.
“Similarly, Baumel et al. (2019) calculated that there is a research bias regarding engagement estimates with intervention usage being four times higher in trials compared with the same interventions in real-world.”
From Special issue on digital health interventions in chronic medical conditions: Editorial (2022) · cites this paper as Baumel et al., 2019
The presence of reminders from the research team could encourage app uptake beyond what might be expected in a naturalistic setting.
“Additionally, the presence of reminders from the research team could encourage app uptake beyond what might be expected in a naturalistic setting (Baumel et al. 2019).”
From Evaluating the quality, safety, and functionality of commonly used smartphone apps for bipolar disorder mood and sleep self-management (2022) · cites this paper as Baumel et al. 2019
Engagement with DMHIs remains an ongoing issue, varies from study to study, and is typically lower in real-world use than in research studies.
“engagement with these technologies remains to be an ongoing issue, varies from study to study, and is typically lower in real-world use than research studies [9]”
From Barriers to and facilitators of user engagement with digital mental health interventions: Systematic review (2021) · cites this paper as [9]
Previous analyses did not report on factors related to variability in engagement.
“However, these analyses did not report on factors related to this variability in engagement.”
From Barriers to and facilitators of user engagement with digital mental health interventions: Systematic review (2021) · cites this paper as [9]
Real-world uptake of self-help DMHIs varies widely.
“systematic comparisons in 2018 and 2019 on self-help DMHIs found that real-world uptake varies widely [9,11]”
From Barriers to and facilitators of user engagement with digital mental health interventions: Systematic review (2021) · cites this paper as [9]
Previous reviews have highlighted variability in engagement and uptake, analyzing both DMHIs published in the academic literature and publicly available mental health apps in app stores.
“Previous reviews have highlighted the variability in engagement and uptake, analyzing both DMHIs published in the academic literature [9-11] and publicly available mental health apps in app stores [16].”
From Barriers to and facilitators of user engagement with digital mental health interventions: Systematic review (2021) · cites this paper as [9]
Many existing DMHIs have high attrition rates and low user engagement, especially when accessed in stand-alone self-guided formats and in real-world naturalistic settings.
“In practice, many existing DMHIs have high attrition rates and low user engagement, especially when accessed in stand-alone self-guided formats [25,26] and in “real-world” naturalistic settings outside of controlled research studies [27].”
Cites
The world health report 2001 - Mental health: new understanding, new hope
Sayers, J. (2001) · Europe PMC (PubMed Central) · doi:10.1590/s0042-96862001001100014 · Cited by 3,307 · see citing works
A behavior model for persuasive design
Fogg, B. (2009) · Proceedings of the 4th International Conference on Persuasive Technology · doi:10.1145/1541948.1541999 · Cited by 2,328 · see citing works
Adherence in Internet Interventions for Anxiety and Depression
Christensen, H., Griffiths, K. M., & Farrer, L. (2009) · Journal of Medical Internet Research · doi:10.2196/jmir.1194 · Cited by 1,133 · see citing works
Mohr, D. C., Cuijpers, P., & Lehman, K. (2011) · Journal of Medical Internet Research · doi:10.2196/jmir.1602 · Cited by 1,097 · see citing works
The impact of guidance on Internet-based mental health interventions — A systematic review
Baumeister, H., Reichler, L., Munzinger, M., & Lin, J. (2014) · Internet Interventions · doi:10.1016/j.invent.2014.08.003 · Cited by 1,037 · see citing works
Delivering interventions for depression by using the internet: randomised controlled trial
Christensen, H. (2004) · BMJ · doi:10.1136/bmj.37945.566632.ee · Cited by 928 · see citing works
Healthcare via Cell Phones: A Systematic Review
Krishna, S., Boren, S. A., & Balas, E. A. (2009) · Telemedicine and e-Health · doi:10.1089/tmj.2008.0099 · Cited by 882 · see citing works
Personal Sensing: Understanding Mental Health Using Ubiquitous Sensors and Machine Learning
Mohr, D. C., Zhang, M., & Schueller, S. M. (2017) · Annual Review of Clinical Psychology · doi:10.1146/annurev-clinpsy-032816-044949 · Cited by 802 · see citing works
Fleming, T., Bavin, L., Lucassen, M., Stasiak, K., Hopkins, S., & Merry, S. (2018) · Journal of Medical Internet Research · doi:10.2196/jmir.9275 · Cited by 670 · see citing works
Understanding the Impact of Convergent Validity on Research Results
Carlson, K. D., & Herdman, A. O. (2012) · Organizational Research Methods · doi:10.1177/1094428110392383 · Cited by 642 · see citing works
Kazdin, A. E. (2017) · Behaviour Research and Therapy · doi:10.1016/j.brat.2016.06.004 · Cited by 554 · see citing works
Godwin, M. S., Rühland, L., Casson, I., MacDonald, S., Delva, D., Birtwhistle, R., Lam, M., & Seguin, R. (2003) · BMC Medical Research Methodology · doi:10.1186/1471-2288-3-28 · Cited by 507 · see citing works
E-Mental Health: A Rapid Review of the Literature
Lal, S., & Adair, C. E. (2014) · Psychiatric Services · doi:10.1176/appi.ps.201300009 · Cited by 417 · see citing works
Current and Future Trends in Internet-Supported Mental Health Interventions
Barak, A., & Grohol, J. M. (2011) · Journal of Technology in Human Services · doi:10.1080/15228835.2011.616939 · Cited by 373 · see citing works
Mohr, D. C., Weingardt, K. R., Reddy, M., & Schueller, S. M. (2017) · Psychiatric Services · doi:10.1176/appi.ps.201600541 · Cited by 365 · see citing works
Using the internet: Skill related problems in users’ online behavior
van Deursen, A. J. A. M., & van Dijk, J. A. G. M. (2009) · Interacting with Computers · doi:10.1016/j.intcom.2009.06.005 · Cited by 361 · see citing works
Mental health: there’s an app for that
Anthes, E. (2016) · Nature · doi:10.1038/532020a · Cited by 333 · see citing works
Emerging mHealth and eHealth interventions for serious mental illness: a review of the literature
Naslund, J. A., Marsch, L. A., McHugo, G. J., & Bartels, S. J. (2015) · Journal of Mental Health · doi:10.3109/09638237.2015.1019054 · Cited by 327 · see citing works
Ritterband, L. M., Thorndike, F. P., Ingersoll, K. S., Lord, H. R., Gonder-Frederick, L., Frederick, C., Quigg, M. S., Cohn, W. F., & Morin, C. M. (2017) · JAMA Psychiatry · doi:10.1001/jamapsychiatry.2016.3249 · Cited by 325 · see citing works
Economic evaluations of Internet interventions for mental health: a systematic review
Donker, T., Blankers, M., Hedman, E., Ljótsson, B., Petrie, K., & Christensen, H. (2015) · Psychological Medicine · doi:10.1017/s0033291715001427 · Cited by 316 · see citing works
Neuroscience of Psychoactive Substance Use and Dependence
Mann, K. F. (2004) · Addiction · doi:10.1111/j.1360-0443.2004.00906.x · Cited by 300 · see citing works
Self-guided psychological treatment for depressive symptoms: a meta-analysis
Cuijpers, P., Donker, T., Johansson, R., Mohr, D. C., van Straten, A., & Andersson, G. (2011) · PLoS ONE · doi:10.1371/journal.pone.0021274 · Cited by 250 · see citing works
Christensen, H., Griffiths, K. M., Korten, A. E., Brittliffe, K., & Groves, C. (2004) · Journal of Medical Internet Research · doi:10.2196/jmir.6.4.e46 · Cited by 240 · see citing works
A randomized controlled trial of a smartphone app for posttraumatic stress disorder symptoms.
Kuhn, E., Kanuri, N., Hoffman, J. E., Garvert, D. W., Ruzek, J. I., & Taylor, C. B. (2017) · Journal of Consulting and Clinical Psychology · doi:10.1037/ccp0000163 · Cited by 225 · see citing works
Champion, L., Economides, M., & Chandler, C. (2018) · PLOS ONE · doi:10.1371/journal.pone.0209482 · Cited by 225 · see citing works
mHealth in the Wild: Using Novel Data to Examine the Reach, Use, and Impact of PTSD Coach
Owen, J. E., Jaworski, B. K., Kuhn, E., Makin-Byrd, K. N., Ramsey, K. M., & Hoffman, J. E. (2015) · JMIR Mental Health · doi:10.2196/mental.3935 · Cited by 216 · see citing works
Baumel, A., Faber, K., Mathur, N., Kane, J. M., & Muench, F. (2017) · Journal of Medical Internet Research · doi:10.2196/jmir.7270 · Cited by 206 · see citing works
Self-citationMapping the evolving definitions of translational research
Fort, D. G., Herr, T. M., Shaw, P. L., Gutzman, K. E., & Starren, J. B. (2017) · Journal of Clinical and Translational Science · doi:10.1017/cts.2016.10 · Cited by 195 · see citing works
Lewis, C., Pearce, J., & Bisson, J. I. (2012) · British Journal of Psychiatry · doi:10.1192/bjp.bp.110.084756 · Cited by 189 · see citing works
Versluis, A., Verkuil, B., Spinhoven, P., van der Ploeg, M. M., & Brosschot, J. F. (2016) · Journal of Medical Internet Research · doi:10.2196/jmir.5642 · Cited by 186 · see citing works
Miner, A., Kuhn, E., Hoffman, J. E., Owen, J. E., Ruzek, J. I., & Taylor, C. B. (2016) · Psychological Trauma: Theory, Research, Practice, and Policy · doi:10.1037/tra0000092 · Cited by 185 · see citing works
Titov, N., Dear, B. F., Staples, L. G., Bennett-Levy, J., Klein, B., Rapee, R. M., Andersson, G., Purtell, C., Bezuidenhout, G., & Nielssen, O. B. (2017) · Australian & New Zealand Journal of Psychiatry · doi:10.1177/0004867416671598 · Cited by 183 · see citing works
A meta-analytic study of self-help interventions for anxiety problems.
Hirai, M., & Clum, G. (2006) · The Behavior Therapist · doi:10.1016/j.beth.2005.05.002 · Cited by 181 · see citing works
Lintvedt, O. K., Griffiths, K. M., Sørensen, K., Østvik, A. R., Wang, C. E. A., Eisemann, M., & Waterloo, K. (2013) · Clinical Psychology & Psychotherapy · doi:10.1002/cpp.770 · Cited by 157 · see citing works
Moller, A. C., Merchant, G., Conroy, D. E., West, R., Hekler, E., Kugler, K. C., & Michie, S. (2017) · Journal of Behavioral Medicine · doi:10.1007/s10865-016-9818-7 · Cited by 151 · see citing works
Day, J. J., & Sanders, M. R. (2018) · Behavior Therapy · doi:10.1016/j.beth.2018.03.002 · Cited by 146 · see citing works
Baumel, A., Pawar, A. K. S., Kane, J. M., & Correll, C. U. (2016) · Journal of Child and Adolescent Psychopharmacology · doi:10.1089/cap.2016.0048 · Cited by 141 · see citing works
Self-citationBaumel, A., & Kane, J. M. (2018) · Journal of Medical Internet Research · doi:10.2196/11491 · Cited by 131 · see citing works
Self-citationBaumel, A., & Yom-Tov, E. (2018) · Translational Behavioral Medicine · doi:10.1093/tbm/ibx037 · Cited by 128 · see citing works
Self-citationUptake and usage of IntelliCare: A publicly available suite of mental health and well-being apps
Lattie, E. G., Schueller, S. M., Sargent, E., Stiles-Shields, C., Tomasino, K. N., Corden, M. E., Begale, M., Karr, C. J., & Mohr, D. C. (2016) · Internet Interventions · doi:10.1016/j.invent.2016.06.003 · Cited by 113 · see citing works
Baumel, A., Tinkelman, A., Mathur, N., & Kane, J. M. (2018) · JMIR mHealth and uHealth · doi:10.2196/mhealth.9482 · Cited by 101 · see citing works
Self-citationEncouraging Mindfulness in Medical House Staff via Smartphone App: A Pilot Study
Wen, L., Sweeney, T. E., Welton, L., Trockel, M., & Katznelson, L. (2017) · Academic Psychiatry · doi:10.1007/s40596-017-0768-3 · Cited by 94 · see citing works
E‐mental health: a new era in delivery of mental health services
Christensen, H., & Hickie, I. B. (2010) · Medical Journal of Australia · doi:10.5694/j.1326-5377.2010.tb03684.x · Cited by 77 · see citing works
Torous, J. B., & Nebeker, C. (2017) · Journal of Medical Internet Research · doi:10.2196/jmir.6793 · Cited by 74 · see citing works
Technology-Assisted Parent Training Programs for Children and Adolescents With Disruptive Behaviors
Baumel, A., Pawar, A. K. S., Mathur, N., Kane, J. M., & Correll, C. U. (2017) · The Journal of Clinical Psychiatry · doi:10.4088/jcp.16r11063 · Cited by 68 · see citing works
Self-citationCarpenter, J. M., Crutchley, P., Zilca, R., Schwartz, H. A., Smith, L. K., Cobb, A. M., & Parks, A. C. (2016) · Journal of Medical Internet Research · doi:10.2196/jmir.5725 · Cited by 58 · see citing works
Baumel, A. (2015) · Journal of Mental Health · doi:10.3109/09638237.2015.1079308 · Cited by 44 · see citing works
Self-citationUtility of the T2 Mood Tracker Mobile Application Among Army Warrior Transition Unit Service Members
Bush, N. E., Ouellette, G., & Kinn, J. (2014) · Military Medicine · doi:10.7205/milmed-d-14-00271 · Cited by 39 · see citing works
Ristkari, T., Kurki, M., Suominen, A., Gilbert, S., Sinokki, A., Kinnunen, M., Huttunen, J., McGrath, P. J., & Sourander, A. (2018) · Journal of Medical Internet Research · doi:10.2196/11446 · Cited by 34 · see citing works
Grasping a Global View of Smartphone Diffusion: An Analysis from a Global Smartphone Study
Kakihara, M. (2014) · Journal of the Association for Information Systems · Cited by 29 · see citing works
Baumel, A., Correll, C. U., & Birnbaum, M. L. (2016) · Internet Interventions · doi:10.1016/j.invent.2016.03.003 · Cited by 29 · see citing works
Self-citationInternet-Based Self-help Interventions for Depression in Routine Care
Ebert, D. D., & Baumeister, H. (2017) · JAMA Psychiatry · doi:10.1001/jamapsychiatry.2017.1394 · Cited by 29 · see citing works
Development of an Internet Intervention to Promote Wellbeing in College Students
Stallman, H. M., & Kavanagh, D. J. (2018) · Australian Psychologist · doi:10.1111/ap.12246 · Cited by 27 · see citing works
Making the Case for a Feasible Evaluation Method of Available E-Mental Health Products
Baumel, A. (2016) · Administration and Policy in Mental Health and Mental Health Services Research · doi:10.1007/s10488-016-0764-z · Cited by 19 · see citing works
Self-citationBaumel, A., Baker, J. T., Birnbaum, M. L., Christensen, H. M., Choudhury, M. D., Mohr, D. C., Muench, F. J., Schlosser, D. A., Titov, N., & Kane, J. M. (2018) · Psychiatric Services · doi:10.1176/appi.ps.201700270 · Cited by 17 · see citing works
Self-citationYu, J. S., Szigethy, E., Wallace, M., Solano, F., & Oser, M. (2018) · Telemedicine and e-Health · doi:10.1089/tmj.2017.0280 · Cited by 15 · see citing works
Falkenstein, M. J., Rogers, K., Malloy, E., & Haaga, D. (2015) · Journal of Clinical Psychology · doi:10.1002/jclp.22171 · Cited by 12 · see citing works
Improving Athlete Mental Training Engagement Using Smartphone Phone Technology
Rist, B., & Pearce, A. J. (2016) · International Journal of Social Science and Humanity · doi:10.18178/ijssh.2017.v7.809 · Cited by 9 · see citing works
Li, F. (2018) · Research Commons (University of Waikato) · Cited by 3 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Borghouts, J., Eikey, E., Mark, G., De Leon, C., Schueller, S. M., Schneider, M., Stadnick, N., Zheng, K., Mukamel, D., & Sorkin, D. H. (2021) · Journal of Medical Internet Research · doi:10.2196/24387 · Cited by 1,069 · see citing works
“Similarly, systematic comparisons in 2018 and 2019 on self-help DMHIs found that real-world uptake varies widely [9,11], and acceptability can be lower than traditional treatment [12].”
BackgroundJakob, R., Harperink, S., Rudolf, A. M., Fleisch, E., Haug, S., Mair, J. L., Salamanca-Sanabria, A., & Kowatsch, T. (2022) · Journal of Medical Internet Research · doi:10.2196/35371 · Cited by 446 · see citing works
Gál, É., Ștefan, S., & Cristea, I. A. (2021) · Journal of Affective Disorders · doi:10.1016/j.jad.2020.09.134 · Cited by 289 · see citing works
Lipschitz, J. M., Pike, C. K., Hogan, T. P., Murphy, S. A., & Burdick, K. E. (2023) · Current Treatment Options in Psychiatry · doi:10.1007/s40501-023-00297-3 · Cited by 230 · see citing works
Engagement and retention in digital mental health interventions: a narrative review
Boucher, E. M., & Raiker, J. S. (2024) · BMC Digital Health · doi:10.1186/s44247-024-00105-9 · Cited by 201 · see citing works
Scaling evidence-based treatments through digital mental health.
Schueller, S. M., & Torous, J. B. (2020) · American Psychologist · doi:10.1037/amp0000654 · Cited by 174 · see citing works
Simon, L., Steinmetz, L., Feige, B., Benz, F., Spiegelhalder, K., & Baumeister, H. (2023) · Scientific Reports · doi:10.1038/s41598-023-28853-0 · Cited by 135 · see citing works
Baumel, A., Fleming, T. M., & Schueller, S. M. (2020) · Journal of Medical Internet Research · doi:10.2196/20631 · Cited by 116 · see citing works
Self-citationKlimczak, K. S., San Miguel, G. G., Mukasa, M. N., Twohig, M. P., & Levin, M. E. (2023) · Cognitive Behaviour Therapy · doi:10.1080/16506073.2023.2178498 · Cited by 88 · see citing works
“It should also be taken into consideration that by way of including only RCTs, results may reflect online intervention use under more ideal conditions as compared to natural use (Baumel et al., 2019).”
BackgroundRiadi, I., Kervin, L. M., Dhillon, S., Teo, K., Churchill, R., Card, K. G., Sixsmith, A. J., Moreno, S., Fortuna, K. L., Torous, J. B., et al. (2022) · The Lancet Healthy Longevity · doi:10.1016/s2666-7568(22)00121-0 · Cited by 87 · see citing works
Engagement and attrition in digital mental health: current challenges and potential solutions
Smith, K., Ward, T., Lambe, S., Ostinelli, E. G., Blease, C., Gant, T., Gold, S. M., Holmes, E. A., Paccoud, I., Vinnikova, A., et al. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-01778-w · Cited by 71 · see citing works
Ramos, G., Hernandez-Ramos, R., Taylor, M. E., & Schueller, S. M. (2024) · Behavior Therapy · doi:10.1016/j.beth.2024.04.004 · Cited by 64 · see citing works
Is There an App for That?: Ethical Issues in the Digital Mental Health Response to COVID-19
Skorburg, J. A., & Yam, J. (2021) · AJOB Neuroscience · doi:10.1080/21507740.2021.1918284 · Cited by 62 · see citing works
Hull, T. D., Malgaroli, M., Gazzaley, A., Akiki, T. J., Madan, A., Vando, L., Arden, K., Swain, J., Klotz, M., & Paleos, C. (2022) · Journal of Affective Disorders · doi:10.1016/j.jad.2022.07.004 · Cited by 58 · see citing works
Single-session interventions embedded within Tumblr: acceptability, feasibility, and utility study
Dobias, M. L., Morris, R. R., & Schleider, J. L. (2022) · JMIR Formative Research · doi:10.2196/39004 · Cited by 56 · see citing works
Baumel, A., Torous, J. B., Edan, S., & Kane, J. M. (2020) · Journal of Affective Disorders · doi:10.1016/j.jad.2020.05.011 · Cited by 40 · see citing works
Self-citationLucassen, M. F. G., Stasiak, K., Fleming, T. M., FRAMPTON, C. M. A., Perry, Y., Shepherd, M., & Merry, S. (2020) · Australian & New Zealand Journal of Psychiatry · doi:10.1177/0004867420976846 · Cited by 38 · see citing works
Ho, T. Q. A., Le, L. K.-D., Engel, L., Le, N., Melvin, G., Le, H. N. D., & Mihalopoulos, C. (2025) · European Child & Adolescent Psychiatry · doi:10.1007/s00787-024-02386-x · Cited by 38 · see citing works
Potts, S., Krafft, J., & Levin, M. E. (2020) · Behavior Modification · doi:10.1177/0145445520975112 · Cited by 36 · see citing works
Psihogios, A. M., King‐Dowling, S., O’Hagan, B., Darabos, K., Maurer, L. A., Young, J., Fleisher, L. G., Barakat, L. P., Szalda, D. E., Hill‐Kayser, C. E., et al. (2021) · Annals of Behavioral Medicine · doi:10.1093/abm/kaab008 · Cited by 34 · see citing works
Electronic bridge to mental health for college students: A randomized controlled intervention trial.
King, C. A., Eisenberg, D., Pistorello, J., Coryell, W. H., Albucher, R. C., Favorite, T., Horwitz, A. G., Bonar, E. E., Epstein, D. J., & Zheng, K. (2022) · Journal of Consulting and Clinical Psychology · doi:10.1037/ccp0000709 · Cited by 34 · see citing works
Baumel, A., & Muench, F. J. (2021) · Journal of Medical Internet Research · doi:10.2196/24905 · Cited by 30 · see citing works
Self-citationMeyerhoff, J., Kornfield, R., Mohr, D. C., & Reddy, M. C. (2022) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3555203 · Cited by 32 · see citing works
Kulikov, V. N., Crosthwaite, P. C., Hall, S. A., Flannery, J., Strauss, G., Vierra, E., Koepsell, X., Lake, J. I., & Padmanabhan, A. (2023) · Frontiers in Digital Health · doi:10.3389/fdgth.2023.1062471 · Cited by 33 · see citing works
Barakat, S., Burton, A. L., Cunich, M. M., Hay, P., Hazelton, J. L., Kim, M., Lymer, S. J., Madden, S., Maloney, D., Miskovic‐Wheatley, J., et al. (2023) · Psychiatry Research · doi:10.1016/j.psychres.2023.115534 · Cited by 29 · see citing works
Hoffman, B. D., Oppert, M. L., & Owen, M. S. (2024) · Computers in Human Behavior Artificial Humans · doi:10.1016/j.chbah.2024.100086 · Cited by 34 · see citing works
Chen, C., Huang, N., Hu, B., Zhang, M., Yuan, J., & Guo, J. (2024) · GeroScience · doi:10.1007/s11357-024-01446-z · Cited by 29 · see citing works
Wanniarachchi, V. U., Greenhalgh, C., Choi, A., & Warren, J. (2025) · Frontiers in Digital Health · doi:10.3389/fdgth.2025.1500220 · Cited by 28 · see citing works
Klimczak, K. S., Twohig, M. P., Peacock, G. G., & Levin, M. E. (2022) · Behaviour Research and Therapy · doi:10.1016/j.brat.2022.104228 · Cited by 28 · see citing works
Boucher, E. M., Ward, H. E., Miles, C. J., Vlisides‐Henry, R. D., & Stoeckl, S. E. (2024) · Journal of Medical Internet Research · doi:10.2196/54282 · Cited by 28 · see citing works
Predictors of adherence to a publicly available self-guided digital mental health intervention
Woolley, M. G., Klimczak, K. S., Davis, C. H., & Levin, M. E. (2024) · Cognitive Behaviour Therapy · doi:10.1080/16506073.2024.2341807 · Cited by 25 · see citing works
Baumel, A. (2022) · Journal of Technology in Behavioral Science · doi:10.1007/s41347-022-00245-7 · Cited by 22 · see citing works
Self-citationThe Guardians: Designing a Game for Long-term Engagement with Mental Health Therapy
Ferguson, C., Lewis, R., Wilks, C. R., & Picard, R. W. (2021) · IEEE Conference on Games (CoG) · doi:10.1109/cog52621.2021.9619026 · Cited by 22 · see citing works
Morton, E., Nicholas, J., Yang, L., Lapadat, L., Barnes, S. J., Provencher, M. D., Depp, C., Chan, M., Kulur, R., & Michalak, E. E. (2022) · International Journal of Bipolar Disorders · doi:10.1186/s40345-022-00256-6 · Cited by 20 · see citing works
Kluge, M. G., Maltby, S., Kühne, C., Walker, N., Bennett, N., Aidman, E., Nalivaiko, E., & Walker, F. R. (2023) · Journal of Medical Internet Research · doi:10.2196/46368 · Cited by 21 · see citing works
May, C. N., Cox‐Martin, M., Ho, A. S., McCallum, M., Chan, C., Blessing, K., Behr, H., Blanco, P., Mitchell, E. S., & Michaelides, A. P. (2023) · Obesity Science & Practice · doi:10.1002/osp4.666 · Cited by 20 · see citing works
Alagarajah, J., Ceccolini, D., & Butler, S. (2024) · Cambridge Prisms Global Mental Health · doi:10.1017/gmh.2024.71 · Cited by 21 · see citing works
Young adolescents’ interest in a mental health casual video game
Pine, R., Sutcliffe, K., McCallum, S., & Fleming, T. M. (2020) · Digital Health · doi:10.1177/2055207620949391 · Cited by 20 · see citing works
Wu, M. S., Wickham, R. E., Chen, S.-Y., Chen, C., & Lungu, A. (2021) · JMIR Formative Research · doi:10.2196/33452 · Cited by 15 · see citing works
Baumel, A., Brandes, O., Brendryen, H., Muench, F. J., Kane, J. M., & Saar, C. R. (2023) · Internet Interventions · doi:10.1016/j.invent.2023.100680 · Cited by 17 · see citing works
Self-citationMorriss, R. K., Kaylor‐Hughes, C., Rawsthorne, M., Coulson, N., Simpson, S., Guo, B., James, M., Lathe, J., Moran, P., Tata, L. J., et al. (2020) · Journal of Medical Internet Research · doi:10.2196/23487 · Cited by 17 · see citing works
Special issue on digital health interventions in chronic medical conditions: Editorial
Baumeister, H., Ebert, D. D., & Snoek, F. (2022) · Internet Interventions · doi:10.1016/j.invent.2021.100457 · Cited by 13 · see citing works
Ho, T. Q. A., Engel, L., Melvin, G., Le, L. K.-D., Le, H. N. D., & Mihalopoulos, C. (2024) · The Patient - Patient-Centered Outcomes Research · doi:10.1007/s40271-024-00707-5 · Cited by 15 · see citing works
“Engagement with W-MHIs has been shown to be typically lower than that in research studies [12].”
ResultHoepper, B. B., Siegel, K. R., Carlon, H. A., Kahler, C. W., Park, E. R., Taylor, S. T., Simpson, H. V., & Hoeppner, S. S. (2022) · JMIR Formative Research · doi:10.2196/38234 · Cited by 15 · see citing works
Meyerhoff, J., Haldar, S., & Mohr, D. C. (2021) · Internet Interventions · doi:10.1016/j.invent.2021.100399 · Cited by 14 · see citing works
Weiner, L. S., Crowley, R. N., Sheeber, L., Koegler, F. H., Davis, J. F., Wells, M., Funkhouser, C. J., Auerbach, R. P., & Allen, N. B. (2024) · JMIR Mental Health · doi:10.2196/51366 · Cited by 18 · see citing works
Ross, J., Hawkes, R. E., Miles, L. M., Cotterill, S., Bower, P., & Murray, E. (2023) · Journal of Medical Internet Research · doi:10.2196/47436 · Cited by 13 · see citing works
Löbner, M., Stein, J., Luppa, M., Bleckwenn, M., Mehnert, A., & Riedel-Heller, S. G. (2022) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph19106203 · Cited by 14 · see citing works
Wu, M. S., Chen, S.-Y., Wickham, R. E., Leykin, Y., Varra, A. A., Chen, C., & Lungu, A. (2022) · Digital Health · doi:10.1177/20552076221133760 · Cited by 13 · see citing works
Hennemann, S., Böhme, K., Kleinstäuber, M., Ruckes, C., Baumeister, H., Daniel Ebert, D., Küchler, A.-M., & Witthöft, M. (2022) · Behavior Therapy · doi:10.1016/j.beth.2022.06.006 · Cited by 11 · see citing works
Expanding, improving, and understanding behaviour research and therapy through digital mental health
Cohen, Z. D., & Schueller, S. M. (2023) · Behaviour Research and Therapy · doi:10.1016/j.brat.2023.104358 · Cited by 10 · see citing works
Lungu, A., Wickham, R. E., Chen, S.-Y., Jun, J. J., Leykin, Y., & Chen, C. E.-J. (2022) · Internet Interventions · doi:10.1016/j.invent.2022.100536 · Cited by 11 · see citing works
Jaegere, E. D., van Heeringen, K., Emmery, P., Mommerency, G., & Portzky, G. (2024) · JMIR Serious Games · doi:10.2196/47513 · Cited by 13 · see citing works
Kranenburg, L. W., Gillis, J., Mayer, B., & Hoogendijk, W. J. G. (2021) · JMIR Mental Health · doi:10.2196/32123 · Cited by 12 · see citing works
Acceptance of smart sensing: a barrier to implementation—results from a randomized controlled trial
Terhorst, Y., Weilbacher, N., Suda, C., Simon, L., Messner, E.-M., Sander, L. B., & Baumeister, H. (2023) · Frontiers in Digital Health · doi:10.3389/fdgth.2023.1075266 · Cited by 10 · see citing works
Blended care in psychosis – A systematic review
Cohen, M., Roe, D., Savir, T., & Baumel, A. (2024) · Schizophrenia Research · doi:10.1016/j.schres.2024.03.041 · Cited by 10 · see citing works
Self-citationFowler, J. A., Buckley, L., Viskovich, S., Muir, M., & Dean, J. A. (2024) · Psychiatry Research · doi:10.1016/j.psychres.2024.115873 · Cited by 9 · see citing works
Designing for mental health in higher education: a collaborative approach to digital interventions
Severes, B., Barreto, M. L., & Esteves, A. (2024) · Behaviour and Information Technology · doi:10.1080/0144929x.2024.2436503 · Cited by 11 · see citing works
Engagement in Digital Self-management Interventions for Chronic Pain
MacLean, R. R., Shor, R., Reilly, E. D., Reuman, L., Solar, C. A., Halat, A. M., & Higgins, D. M. (2025) · Clinical Journal of Pain · doi:10.1097/ajp.0000000000001289 · Cited by 11 · see citing works
Goenjian, H. A., Pratap, A., Snipes, C., Hare, B. D., Kantrowitz, J. T., Dennis, T., McNeel, W., Ung, E., Besedina, O., Gupta, A., et al. (2025) · Schizophrenia · doi:10.1038/s41537-025-00659-1 · Cited by 9 · see citing works
Michalak, E. E., Barnes, S. J., Morton, E. E., O'Brien, H. L., Murray, G., Hole, R., & Meyer, D. (2022) · JMIR Research Protocols · doi:10.2196/36213 · Cited by 9 · see citing works
Wirkfaktoren und Veränderungsmechanismen digitaler psychotherapeutischer Interventionen
Domhardt, M., & Baumeister, H. (2021) · Psychotherapeut · doi:10.1007/s00278-021-00525-2 · Cited by 9 · see citing works
Levin, M. E., Mukasa, M. N., Bowers, E. M., Klimczak, K. S., & Aller, T. B. (2025) · Behavioral Sciences · doi:10.3390/bs15010075 · Cited by 8 · see citing works
Rajkumar, E., Hareesh, P. V., Lakshmi, K. N. V. S., Shaza, A., Gopi, A., & Romate, J. (2025) · Geopsychiatry · doi:10.1016/j.geopsy.2025.100010 · Cited by 9 · see citing works
Wanniarachchi, V. U., Greenhalgh, C., & Warren, J. (2025) · Internet Interventions · doi:10.1016/j.invent.2025.100884 · Cited by 10 · see citing works
Thapar, S., Nguyen, M., Khan, B. N., Fanaieyan, R., Kishimoto, V., Liu, R. H., Bolea-Alamanac, B. M., Leon-Carlyle, M., O’Riordan, A., Keresteci, M., et al. (2024) · JMIR Formative Research · doi:10.2196/50113 · Cited by 8 · see citing works
Vadhan, N. P., Padovano, H. T., Muench, F. J., Levak, S., & Allen, N. (2024) · Alcoholism Clinical and Experimental Research · doi:10.1111/acer.15414 · Cited by 8 · see citing works
Xiang, X., Turner, S., Ruiz-Sierra, S., Zheng, C., Ash, S., Kodkany, N., Dimoji, A., Gosdzinski, C., & Kubicek, G. (2024) · Clinical Gerontologist · doi:10.1080/07317115.2024.2395890 · Cited by 8 · see citing works
Renn, B. N., Walker, T. J., Edds, B., Roots, M., & Raue, P. J. (2024) · Journal of Affective Disorders · doi:10.1016/j.jad.2024.09.104 · Cited by 8 · see citing works
Can Digital Therapeutics Open a New Era of Sleep Medicine?
Park, K., Lee, S., & Lee, E. M. (2021) · Chronobiology in Medicine · doi:10.33069/cim.2021.0028 · Cited by 8 · see citing works
Baumel, A. (2025) · Frontiers in Digital Health · doi:10.3389/fdgth.2025.1541676 · Cited by 7 · see citing works
Self-citationBorghouts, J. W., Eikey, E. V., Leon, C. D., Schueller, S. M., Schneider, M. L., Stadnick, N. A., Zheng, K., Mukamel, D. B., & Sorkin, D. H. (2025) · JMIR Formative Research · doi:10.2196/73457 · Cited by 8 · see citing works
Simon, L., Steinmetz, L., Bendig, E., Küchler, A., Riemann, D. W., Ebert, D. D., Spiegelhalder, K., & Baumeister, H. (2025) · International Journal of Clinical and Health Psychology · doi:10.1016/j.ijchp.2025.100598 · Cited by 8 · see citing works
Levin, M. E., Aller, T. B., Klimczak, K. S., Donahue, M. L., & Knudsen, F. M. (2025) · Behaviour Research and Therapy · doi:10.1016/j.brat.2025.104729 · Cited by 5 · see citing works
Hentati, A., Isacsson, N. H., Rosén, A., Jernelöv, S., Kaldo, V., Ljótsson, B., Forsell, E., Lindefors, N., & Kraepelien, M. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-01839-0 · Cited by 5 · see citing works
Das, P. J., Kyndait, P., Jamir, S. I., Baishya, D., Kalita, J. K., & Komu, L. T. (2024) · International Journal of Scientific Research in Science and Technology · doi:10.32628/ijsrst24116186 · Cited by 5 · see citing works
Khan, B. N., Liu, R. H., Chu, C., Bolea-Alamanac, B. M., Nguyen, M., Thapar, S., Fanaieyan, R., Leon-Carlyle, M., Tadrous, M., Kurdyak, P. A., et al. (2024) · International Journal of Mental Health Systems · doi:10.1186/s13033-024-00651-9 · Cited by 6 · see citing works
Buntrock, C., Freund, J., Smit, F., Riper, H. M., Lehr, D., Boß, L., Berking, M., & Ebert, D. D. (2021) · Addiction · doi:10.1111/add.15718 · Cited by 5 · see citing works
Donahue, M. L., & Levin, M. E. (2024) · International Journal of Mental Health · doi:10.1080/00207411.2024.2321432 · Cited by 4 · see citing works
Kim, M., Moon, Y.-K., & Chung, K.-M. (2024) · JMIR mHealth and uHealth · doi:10.2196/50006 · Cited by 5 · see citing works
Ho, T. Q. A., Engel, L., Le, L. K., Melvin, G. A., Ride, J., Le, H., & Mihalopoulos, C. (2025) · Patient · doi:10.1007/s40271-025-00739-5 · Cited by 4 · see citing works
Kwok, G., Sharma, A., Mandato, I., & Devine, K. A. (2024) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph21050584 · Cited by 4 · see citing works
Reich, H., Hegerl, U., Rosenthal, A., & Allenhof, C. (2024) · Scientific Reports · doi:10.1038/s41598-024-62196-8 · Cited by 4 · see citing works
Zahed, K., Markert, C., & Sasangohar, F. (2025) · Applied Ergonomics · doi:10.1016/j.apergo.2025.104485 · Cited by 4 · see citing works
Parker, R., Padfield, P., Hanley, J., Pinnock, H., Kennedy, J., Stoddart, A., Hammersley, V., Sheikh, A., & McKinstry, B. (2020) · BMC Medical Research Methodology · doi:10.1186/s12874-021-01219-8 · Cited by 4 · see citing works
Yes, I can! Development and validation of the self-efficacy for self-help scale
Tomczyk, S., Heineck, S., McLaren, T., Peter, L.-J., Schomerus, G., Schmidt, S., & Muehlan, H. (2023) · Journal of Affective Disorders · doi:10.1016/j.jad.2023.03.028 · Cited by 4 · see citing works
Kikas, K., O'Moore, K., Miller, R. K., Matheson, J.-A. T., Li, S., Varghese, K., Baldwin, P., Cockayne, N., Whitton, A. E., & Newby, J. M. (2025) · JMIR Human Factors · doi:10.2196/69499 · Cited by 3 · see citing works
“In practice, many existing DMHIs have high attrition rates and low user engagement, especially when accessed in stand-alone self-guided formats [25,26] and in “real-world” naturalistic settings outside of controlled research studies [27].”
Peuters, C., DeSmet, A., Maenhout, L., Cardon, G., Debeer, D., & Crombez, G. (2025) · JMIR mhealth and uhealth · doi:10.2196/59041 · Cited by 5 · see citing works
Morton, E. E., Hole, R., O'Brien, H. L., LI, L. C., Barnes, S. J., & Michalak, E. E. (2025) · PLOS Digital Health · doi:10.1371/journal.pdig.0001017 · Cited by 3 · see citing works
Fowler, J. A., Buckley, L., Viskovich, S., & Dean, J. A. (2026) · Journal of Homosexuality · doi:10.1080/00918369.2025.2603459 · Cited by 3 · see citing works
Ryan, A. T., Stearns‐Yoder, K. A., & Brenner, L. A. (2023) · JMIR Formative Research · doi:10.2196/48365 · Cited by 3 · see citing works
Gamifying Mental Health Treatment: A Mobile Interface Design for PTSD Intervention
Doron, G., Derby, D. S., & Gamoran, A. (2025) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3706599.3720074 · Cited by 3 · see citing works
Areán, P. A., & Allred, R. (2022) · Elsevier eBooks · doi:10.1016/b978-0-323-90045-4.00011-3 · Cited by 3 · see citing works
Young adolescents’ interest in a mental health casual video game
Pine, R., Sutcliffe, K., McCallum, S., & Fleming, T. M. (2020) · doi:10.26686/wgtn.12904523 · Cited by 3 · see citing works
Günther, F., Wong, D., Elison, S., & Yau, C. (2023) · JAMIA Open · doi:10.1093/jamiaopen/ooad072 · Cited by 2 · see citing works
Baumel, A., & Aderka, I. M. (2025) · JMIR Research Protocols · doi:10.2196/71238 · Cited by 2 · see citing works
Self-citationBowers, E. M., Klimczak, K. S., Aller, T. B., & Levin, M. E. (2025) · Cognitive Behaviour Therapy · doi:10.1080/16506073.2025.2512971 · Cited by 2 · see citing works
Xiang, X.-L., Narbut, E., Zhang, X., Ash, S., Turner, S., An, R., Jester, J. M., Park, S., Habash, S., & Himle, J. A. (2025) · JMIR Research Protocols · doi:10.2196/72953 · Cited by 3 · see citing works
Takae, A., Sasaki, N., Imamura, K., & Nishi, D. (2025) · JMIR mhealth and uhealth · doi:10.2196/73512 · Cited by 2 · see citing works
Sextl-Plötz, T., Franke, M., Baumeister, H., & Ebert, D. D. (2023) · Springer eBooks · doi:10.1007/978-3-662-65816-1_3 · Cited by 2 · see citing works
- Authors
- Baumel, A., Edan, S., & Kane, J. M.
- Affiliations
- University of Haifa (IL)
- Northwell Health (US)
- Corresponding author
- Baumel, Amit
- Venue
- Translational Behavioral Medicine
- Publisher
- Oxford University Press (OUP)
- Published
- 2019-11-05
- Volume
- 9
- Issue
- 6
- Pages
- 1020-1033
- ISSN
- 1613-9860 (ISSN-L)
- 1869-6716
- OpenAlex
- W2989270862
- Cited by
- 143
- References
- 60
- Citations by year
- 2026: 17 · 2025: 38 · 2024: 26 · 2023: 17 · 2022: 21 · 2021: 14 · 2020: 10
- FWCI
- 12.74
- Type
- journal article
- Language
- en
- Open access
- closed
- Licence
- academic.oup.com/journals/pages/open_access/funder_policies/chorus/standard_publication_model (vor)
- Locations
- Translational Behavioral Medicine (publishedVersion)
- PubMed (publishedVersion)
- Topics
- Social Sciences › Psychology › Applied Psychology › Digital Mental Health Interventions
- Health Sciences › Health Professions › General Health Professions › Mobile Health and mHealth Applications
- Health Sciences › Medicine › Public Health, Environmental and Occupational Health › Telemedicine and Telehealth Implementation
- Keywords
- trial bias, user engagement
- MeSH terms
- Clinical Trials as Topic, Humans, Mental Health, Program Development, Telemedicine
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
