A multi-faceted approach to characterizing user behavior and experience in a digital mental health intervention
Open accessAbstract
Digital interventions offer great promise for supporting health-related behavior change. However, there is much that we have yet to learn about how people respond to them. In this study, we present a novel mixed-methods approach to analysis of the complex and rich data that digital interventions collect. We perform secondary analysis of IntelliCare, an intervention in which participants are able to try 14 different mental health apps over the course of eight weeks. The goal of our analysis is to characterize users' app use behavior and experiences, and is rooted in theoretical conceptualizations of engagement as both usage and user experience. In the first aim, we employ cluster analysis to identify subgroups of participants that share similarities in terms of the frequency of their usage of particular apps, and then employ other engagement measures to compare the clusters. We identified four clusters with different app usage patterns: Low Usage, High Usage, Daily Feats Users, and Day to Day users. Each cluster was distinguished by its overall frequency of app use, or the main app that participants used. In the second aim, we developed a computer-assisted text analysis and visualization method - message highlighting - to facilitate comparison of the clusters. Last, we performed a qualitative analysis using participant messages to better understand the mechanisms of change and usability of salient apps from the cluster analysis. Our novel approach, integrating text and visual analytics with more traditional qualitative analysis techniques, can be used to generate insights concerning the behavior and experience of users in digital health contexts, for subsequent personalization and to identify areas for improvement of intervention technologies.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The length of the program and the length of individual sessions play a role in participants' satisfaction and motivation to continue with the program.
“The length of the program as well as the length of individual sessions played a role in participants’ satisfaction and their motivation to continue with the program [88,138,147-151].”
From Barriers to and facilitators of user engagement with digital mental health interventions: Systematic review (2021) · cites this paper as [149]
The Discussion (page unavailable) suggests that shorter activity bursts might help the Low Usage group, offering limited support for a link between activity length and continued engagement. It does not report that either the overall program length or individual session length affected participants’ satisfaction or motivation to continue; the eight-week duration is described in Methods and Results, not evaluated as a factor. (LLM, conf=0.79)
Experimental — accuracy not yet measuredUsability issues formed a barrier to engaging with an intervention.
“Usability issues formed a barrier to engaging with an intervention [46,50, 78,84,148-150,157,159,170,224-228].”
From Barriers to and facilitators of user engagement with digital mental health interventions: Systematic review (2021) · cites this paper as [149]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
The PHQ-9: Validity of a brief depression severity measure
Kroenke, K., Spitzer, R. L., & Williams, J. B. W. (2001) · Journal of General Internal Medicine · doi:10.1046/j.1525-1497.2001.016009606.x · Cited by 44,910 · see citing works
A Brief Measure for Assessing Generalized Anxiety Disorder: The GAD-7
Spitzer, R. L., Kroenke, K., Williams, J. B. W., & Löwe, B. (2006) · Archives of Internal Medicine · doi:10.1001/archinte.166.10.1092 · Cited by 31,765 · cited 1× · see citing works
“Lastly, we rendered line charts depicting the outcomes of the participants of the study measured using the GAD-7[44] and PHQ-9[45].”
MethodThe Coding Manual for Qualitative Researchers
Saldaña, J. (2025) · SAGE Publications Ltd eBooks · doi:10.4135/9781036235611 · Cited by 19,156 · see citing works
Flow: the psychology of optimal experience
(1990) · Choice Reviews Online · doi:10.5860/choice.28-0597 · Cited by 12,652 · see citing works
Grounded theory research: Procedures, canons, and evaluative criteria
Corbin, J. M., & Strauss, A. (1990) · Qualitative Sociology · doi:10.1007/bf00988593 · Cited by 11,133 · see citing works
A General Inductive Approach for Analyzing Qualitative Evaluation Data
Thomas, D. R. (2006) · American Journal of Evaluation · doi:10.1177/1098214005283748 · Cited by 10,858 · cited 2× · see citing works
“We employed a general inductive qualitative coding procedure based on the Grounded Theory Methodology[47], in which the researcher engages in line-by-line coding of the data and develops conceptual codes[48,49].”
MethodData clustering: 50 years beyond K-means
Jain, A. K. (2009) · Pattern Recognition Letters · doi:10.1016/j.patrec.2009.09.011 · Cited by 9,622 · see citing works
Thorndike, R. L. (1953) · Psychometrika · doi:10.1007/bf02289263 · Cited by 2,102 · cited 1× · see citing works
“With the Elbow method, the variance is plotted against the number of clusters and the point at which increasing the number of clusters does not lead to substantial decreases in variance is selected[40,41].”
MethodWhat is user engagement? A conceptual framework for defining user engagement with technology
O'Brien, H. L., & Toms, E. G. (2008) · Journal of the American Society for Information Science and Technology · doi:10.1002/asi.20801 · Cited by 1,397 · see citing works
Perski, O., Blandford, A., West, R., & Michie, S. (2017) · Translational Behavioral Medicine · doi:10.1007/s13142-016-0453-1 · Cited by 1,193 · cited 4× · see citing works
“Mechanisms of change, which mediate the effect of engagement with the intervention and intervention effectiveness, include knowledge/information, motivation, beliefs and attitudes, and skill building, account for behavior change and subsequent symptom improvement[10,13].”
BackgroundInfluentialMohr, D. C., Cuijpers, P., & Lehman, K. (2011) · Journal of Medical Internet Research · doi:10.2196/jmir.1602 · Cited by 1,097 · cited 1× · see citing works
“The coaching protocol[29] was based on the Efficiency Model of Support [30] and supportive accountability[31].”
MethodSelf-citationLearning extraction patterns for subjective expressions
Riloff, E., & Wiebe, J. (2003) · Proceedings of the 2003 conference on Empirical methods in natural language processing - · doi:10.3115/1119355.1119369 · Cited by 1,059 · cited 1× · see citing works
“To facilitate a process of “discovery” of salient concepts from the text data, we employed a sentiment analysis lexicon comprised of over 6,000 words[46].”
MethodThe impact of guidance on Internet-based mental health interventions — A systematic review
Baumeister, H., Reichler, L., Munzinger, M., & Lin, J. (2014) · Internet Interventions · doi:10.1016/j.invent.2014.08.003 · Cited by 1,037 · cited 1× · see citing works
“More broadly, there are also issues and gaps in our knowledge of digital mental health interventions, such as the need to identify predictors of therapeutic success or failure[4], reasons for attrition and dropout[5], and determining active intervention components[6].”
BackgroundDeconstructing disengagement: analyzing learner subpopulations in massive open online courses
Kizilcec, R. F., Piech, C., & Schneider, E. (2013) · Proceedings of the Third International Conference on Learning Analytics and Knowledge · doi:10.1145/2460296.2460330 · Cited by 984 · cited 1× · see citing works
“…for diverse tasks ranging from segmentation of consumers by food choice[32,33], analyzing learner subpopulations in online educational environments[34,35], developing personas to inform the design of eHealth technologies [36,37], and identifying patterns of user engagement with a mobile app…”
BackgroundMichie, S., Yardley, L., West, R., Patrick, K., & Greaves, F. (2017) · Journal of Medical Internet Research · doi:10.2196/jmir.7126 · Cited by 865 · cited 1× · see citing works
“For example, the dimension, “Mechanisms”, does not provide a nuanced differentiation of mechanisms of change, which has been recognized as a need in the study of engagement of digital interventions[59].”
BackgroundCluster center initialization algorithm for K-means clustering
Khan, S. S., & Ahmad, A. (2004) · Pattern Recognition Letters · doi:10.1016/j.patrec.2004.04.007 · Cited by 691 · cited 1× · see citing works
“The k-means clustering method can be susceptible to the values that were used to initialize the clustering[42].”
MethodA Behavior Change Model for Internet Interventions
Ritterband, L. M., Thorndike, F. P., Cox, D. J., Kovatchev, B. P., & Gonder-Frederick, L. A. (2009) · Annals of Behavioral Medicine · doi:10.1007/s12160-009-9133-4 · Cited by 591 · cited 5× · see citing works
“In particular, we consider Short et al.’s model[1] and Ritterband’s model of Internet Interventions[13].”
BackgroundMethodInfluentialEmpirical Studies on Usability of mHealth Apps: A Systematic Literature Review
Zapata, B. C., Fernández-Alemán, J. L., Idri, A., & Toval, A. (2015) · Journal of Medical Systems · doi:10.1007/s10916-014-0182-2 · Cited by 585 · cited 1× · see citing works
“Usability is often conceptualized in terms of the effectiveness, efficiency, and satisfaction with which products can be used by users, to achieve particular goals within a specific context of use[54,55].”
BackgroundmHealth: A Mechanism to Deliver More Accessible, More Effective Mental Health Care
Price, M., Yuen, E. K., Goetter, E. M., Herbert, J. D., Forman, E. M., Acierno, R., & Ruggiero, K. J. (2014) · Clinical Psychology & Psychotherapy · doi:10.1002/cpp.1855 · Cited by 488 · cited 1× · see citing works
“…popularity and functionality of mobile devices offers great promise for enhancing the delivery of mental health services including: increasing access to and use of evidence-based care, informing and engaging users more actively in treatment, and enhancing care after treatment has concluded[2].”
BackgroundCalisir, F., & Calisir, F. (2004) · Computers in Human Behavior · doi:10.1016/j.chb.2003.10.004 · Cited by 474 · cited 1× · see citing works
“In this study, we adopted a broader conceptualization of usability which included characteristics such as helpfulness and usefulness, which have been seen as antecedents of satisfaction[56], and have been included in arguments and taxonomies for usability[57,58].”
MethodHow to evaluate technologies for health behavior change in HCI research
Klasnja, P., Consolvo, S., & Pratt, W. (2011) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/1978942.1979396 · Cited by 414 · see citing works
iPhone Apps for Smoking Cessation
Abroms, L. C., Padmanabhan, N., Thaweethai, L., & Phillips, T. (2011) · American Journal of Preventive Medicine · doi:10.1016/j.amepre.2010.10.032 · Cited by 365 · cited 1× · see citing works
“Previous methods to analyze text messages from digital interventions have often employed quantitative content analysis (e.g. [60,61]).”
MethodmHealth 2.0: Experiences, Possibilities, and Perspectives
Becker, S., Miron-Shatz, T., Schumacher, N., Krocza, J., Diamantidis, C., & Albrecht, U.-V. (2014) · JMIR mHealth and uHealth · doi:10.2196/mhealth.3328 · Cited by 354 · cited 1× · see citing works
“However, many challenges also exist for effective deployment, including high attrition rates, the digital divide, and intellectual capabilities of users[3].”
BackgroundChan, Y., & Walmsley, R. P. (1997) · Physical Therapy · doi:10.1093/ptj/77.12.1755 · Cited by 301 · cited 1× · see citing works
“To compare the clusters based on the first two measures, we employed the Kruskal-Wallis one-way analysis-of-variance-by-ranks test (H test), which does not assume that the variables are normally distributed[43].”
MethodIntelliCare: An Eclectic, Skills-Based App Suite for the Treatment of Depression and Anxiety
Mohr, D. C., Tomasino, K. N., Lattie, E. G., Palac, H. L., Kwasny, M. J., Weingardt, K., Karr, C. J., Kaiser, S. M., Rossom, R. C., Bardsley, L. R., et al. (2017) · Journal of Medical Internet Research · doi:10.2196/jmir.6645 · Cited by 301 · cited 4× · see citing works
“These include a sleep diary, a bedtime checklist, audio recordings to facilitate relaxation, and an alarm clock feature[27].”
BackgroundMethodInfluentialSelf-citationLehmann, J., Lalmas, M., Yom‐Tov, E., & Dupret, G. (2012) · Lecture notes in computer science · doi:10.1007/978-3-642-31454-4_14 · Cited by 267 · see citing works
Strecher, V., McClure, J., Alexander, G., Chakraborty, B., Nair, V., Konkel, J., Greene, S., Couper, M., Carlier, C., Wiese, C., et al. (2008) · Journal of Medical Internet Research · doi:10.2196/jmir.1002 · Cited by 264 · cited 2× · see citing works
“Usage data is also often conceptualized as a measure of adherence[19,20] or exposure to intervention content[14,21].”
BackgroundEngagement and retention: measuring breadth and depth of participant use of an online intervention
Couper, M. P., Alexander, G. L., Zhang, N., Little, R. J., Maddy, N., Nowak, M. A., McClure, J. B., Calvi, J. J., Rolnick, S. J., Stopponi, M. A., et al. (2010) · Journal of Medical Internet Research · doi:10.2196/jmir.1430 · Cited by 244 · see citing works
What Makes Internet Therapy Work?
Andersson, G., Carlbring, P., Berger, T., Almlöv, J., & Cuijpers, P. (2009) · Cognitive Behaviour Therapy · doi:10.1080/16506070902916400 · Cited by 234 · cited 1× · see citing works
“Though extant literature has shown that digital interventions work, we are not yet able to explain how, why, and for whom[7–9].”
BackgroundNew ISO Standards for Usability, Usability Reports and Usability Measures
Bevan, N., Carter, J., Earthy, J., Geis, T., & Harker, S. (2016) · Lecture notes in computer science · doi:10.1007/978-3-319-39510-4_25 · Cited by 223 · cited 1× · see citing works
“Usability is often conceptualized in terms of the effectiveness, efficiency, and satisfaction with which products can be used by users, to achieve particular goals within a specific context of use[54,55].”
BackgroundmHealth in the Wild: Using Novel Data to Examine the Reach, Use, and Impact of PTSD Coach
Owen, J. E., Jaworski, B. K., Kuhn, E., Makin-Byrd, K. N., Ramsey, K. M., & Hoffman, J. E. (2015) · JMIR Mental Health · doi:10.2196/mental.3935 · Cited by 216 · cited 1× · see citing works
“Still other research has employed log data to derive insights into users’ support needs, develop recommendations for incorporating new content, and identify points of attrition[23,24].”
Russian consumers’ motives for food choice
Honkanen, P., & Frewer, L. J. (2008) · Appetite · doi:10.1016/j.appet.2008.11.009 · Cited by 205 · cited 1× · see citing works
“…similar groups in data which has been employed in many disciplines, for diverse tasks ranging from segmentation of consumers by food choice[32,33], analyzing learner subpopulations in online educational environments[34,35], developing personas to inform the design of eHealth technologies…”
BackgroundTheme issue on e-mental health: A growing field in internet research
Riper, H., Andersson, G., Christensen, H., Cuijpers, P., Lange, A., & Eysenbach, G. (2010) · Journal of Medical Internet Research · doi:10.2196/jmir.1713 · Cited by 200 · cited 1× · see citing works
“More broadly, there are also issues and gaps in our knowledge of digital mental health interventions, such as the need to identify predictors of therapeutic success or failure[4], reasons for attrition and dropout[5], and determining active intervention components[6].”
BackgroundDoherty, G., Coyle, D., & Sharry, J. (2012) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/2207676.2208602 · Cited by 198 · see citing works
Yang, J.-J., Ning, C., Deb, C., Zhang, F., Cheong, D., Lee, S., Sekhar, C., & Tham, K. (2017) · Energy and Buildings · doi:10.1016/j.enbuild.2017.03.071 · Cited by 186 · cited 1× · see citing works
“With the Elbow method, the variance is plotted against the number of clusters and the point at which increasing the number of clusters does not lead to substantial decreases in variance is selected[40,41].”
MethodExamining the generalizability of the User Engagement Scale (UES) in exploratory search
O'Brien, H. L., & Toms, E. G. (2012) · Information Processing & Management · doi:10.1016/j.ipm.2012.08.005 · Cited by 180 · see citing works
Nijland, N., van Gemert-Pijnen, J. E., Kelders, S. M., Brandenburg, B. J., & Seydel, E. R. (2011) · Journal of Medical Internet Research · doi:10.2196/jmir.1603 · Cited by 175 · cited 1× · see citing works
“Other studies have examined usage trends to better characterize users in terms of needs and user preferences[15,22].”
BackgroundPaxling, B., Lundgren, S., Norman, A., Almlöv, J., Carlbring, P., Cuijpers, P., & Andersson, G. (2013) · Behavioural and Cognitive Psychotherapy · doi:10.1017/s1352465812000240 · Cited by 162 · cited 1× · see citing works
“Previous methods to analyze text messages from digital interventions have often employed quantitative content analysis (e.g. [60,61]).”
MethodChronic Health Conditions and Internet Behavioral Interventions
Schubart, J. R., Stuckey, H. L., GANESHAMOORTHY, A., & Sciamanna, C. (2011) · CIN Computers Informatics Nursing · doi:10.1097/ncn.0b013e3182155274 · Cited by 154 · see citing works
Challenges in Assessing Mobile Health App Quality
Grundy, Q. H., Wang, Z., & Bero, L. A. (2016) · American Journal of Preventive Medicine · doi:10.1016/j.amepre.2016.07.009 · Cited by 154 · see citing works
Neil, A. L., Batterham, P., Christensen, H., Bennett, K., & Griffiths, K. M. (2009) · Journal of Medical Internet Research · doi:10.2196/jmir.1050 · Cited by 152 · cited 1× · see citing works
“Usage data is also often conceptualized as a measure of adherence[19,20] or exposure to intervention content[14,21].”
BackgroundRealizing the Potential of Behavioral Intervention Technologies
Schueller, S. M., Muñoz, R. F., & Mohr, D. C. (2013) · Current Directions in Psychological Science · doi:10.1177/0963721413495872 · Cited by 152 · see citing works
Self-citationDefining Participant Exposure Measures in Web-Based Health Behavior Change Programs
Danaher, B. G., Boles, S. M., Akers, L., Gordon, J. S., & Severson, H. H. (2006) · Journal of Medical Internet Research · doi:10.2196/jmir.8.3.e15 · Cited by 145 · cited 3× · see citing works
“Usage data is also often conceptualized as a measure of adherence[19,20] or exposure to intervention content[14,21].”
BackgroundSCHUBART, J. R., STUCKEY, H. L., GANESHAMOORTHY, A., & SCIAMANNA, C. N. (2011) · CIN: Computers, Informatics, Nursing · doi:10.1097/ncn.0b013e3182065eed · Cited by 135 · cited 1× · see citing works
“The Engagement (usage) category was used to represent the conceptualization of engagement as participants’ efforts to complete the intervention as prescribed, which is often called adherence in the behavioral science literature on digital interventions (e.g. [52,53]).”
Holden, R. J., Kulanthaivel, A., Purkayastha, S., Goggins, K. M., & Kripalani, S. (2017) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2017.10.006 · Cited by 131 · cited 1× · see citing works
“…tasks ranging from segmentation of consumers by food choice[32,33], analyzing learner subpopulations in online educational environments[34,35], developing personas to inform the design of eHealth technologies [36,37], and identifying patterns of user engagement with a mobile app intervention[38].”
BackgroundUptake and usage of IntelliCare: A publicly available suite of mental health and well-being apps
Lattie, E. G., Schueller, S. M., Sargent, E., Stiles-Shields, C., Tomasino, K. N., Corden, M. E., Begale, M., Karr, C. J., & Mohr, D. C. (2016) · Internet Interventions · doi:10.1016/j.invent.2016.06.003 · Cited by 113 · cited 2× · see citing works
“Consistent use is encouraged using streaks, in which consistent use over a number of days levels the user up to another set of activities[26].”
BackgroundSelf-citationSimmons, W. M., & Zoetewey, M. W. (2012) · Technical Communication Quarterly · doi:10.1080/10572252.2012.673953 · Cited by 95 · cited 1× · see citing works
“In this study, we adopted a broader conceptualization of usability which included characteristics such as helpfulness and usefulness, which have been seen as antecedents of satisfaction[56], and have been included in arguments and taxonomies for usability[57,58].”
MethodDesigning engaging online behaviour change interventions: A proposed model of user engagement
Short, C. E., Rebar, A. L., Plotnikoff, R. C., & Vandelanotte, C. (2015) · Adelaide Research & Scholarship (AR&S) (University of Adelaide) · Cited by 89 · see citing works
Kelders, S. M., Bohlmeijer, E. T., & Van Gemert-Pijnen, J. E. (2013) · Journal of Medical Internet Research · doi:10.2196/jmir.2258 · Cited by 88 · cited 1× · see citing works
“Usage data is also often conceptualized as a measure of adherence[19,20] or exposure to intervention content[14,21].”
Van Gemert-Pijnen, J. E., Kelders, S. M., & Bohlmeijer, E. T. (2014) · Journal of Medical Internet Research · doi:10.2196/jmir.2991 · Cited by 84 · cited 1× · see citing works
“Still other research has employed log data to derive insights into users’ support needs, develop recommendations for incorporating new content, and identify points of attrition[23,24].”
Baltierra, N., Muessig, K. E., Pike, E. C., LeGrand, S., Bull, S. S., & Hightow‐Weidman, L. (2015) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2015.12.015 · Cited by 79 · see citing works
An overview on the emerging area of identification, characterization, and assessment of health apps
Paglialonga, A., Lugo, A., & Santoro, E. (2018) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2018.05.017 · Cited by 77 · cited 1× · see citing works
“Recent literature has observed that there is an increased need to identify, characterize, and assess health-related apps[62,63].”
BackgroundUsing personas to tailor educational messages to the preferences of coronary heart disease patients
Vosbergen, S., Mulder-Wiggers, J. M. R., Lacroix, J. P. W., Kemps, H., Kraaijenhagen, R. A., Jaspers, M., & Peek, N. B. (2014) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2014.09.004 · Cited by 72 · cited 1× · see citing works
“…tasks ranging from segmentation of consumers by food choice[32,33], analyzing learner subpopulations in online educational environments[34,35], developing personas to inform the design of eHealth technologies [36,37], and identifying patterns of user engagement with a mobile app intervention[38].”
BackgroundAnalyzing Engagement in a Web-Based Intervention Platform Through Visualizing Log-Data
Morrison, C., & Doherty, G. (2014) · Journal of Medical Internet Research · doi:10.2196/jmir.3575 · Cited by 62 · cited 2× · see citing works
“Previous research has observed that a critical challenge to the analysis of digital interventions is the organization of log data for analysis[25].”
BackgroundInternet Interventions for Mental Health and Addictions: Current Findings and Future Directions
Cunningham, J. A., Gulliver, A., Farrer, L., Bennett, K., & Carron‐Arthur, B. (2014) · Current Psychiatry Reports · doi:10.1007/s11920-014-0521-5 · Cited by 57 · cited 1× · see citing works
“More broadly, there are also issues and gaps in our knowledge of digital mental health interventions, such as the need to identify predictors of therapeutic success or failure[4], reasons for attrition and dropout[5], and determining active intervention components[6].”
BackgroundRahman, Q. A., Janmohamed, T., Pirbaglou, M., Ritvo, P., Heffernan, J. M., Clarke, H. A., & Katz, J. D. (2017) · JMIR mhealth and uhealth · doi:10.2196/mhealth.7871 · Cited by 47 · cited 2× · see citing works
“For example, previous research that employed cluster analysis to study user engagement with a pain management app using variables relating to sociodemographic characteristics, medical history and medication regimen/ use[38].”
BackgroundMethodKwasny, M., Schueller, S. M., Lattie, E. G., Gray, E., & Mohr, D. C. (2018) · JMIR Mental Health · doi:10.2196/11572 · Cited by 37 · see citing works
Self-citationDoherty, G., Coyle, D., & Sharry, J. (2012) · University of Limerick Institutional Repository (University of Limerick) · doi:10.34961/1154 · Cited by 25 · see citing works
Wise, A. F., Marbouti, F., Speer, J., & Hsiao, Y. T. (2011) · Computer Supported Collaborative Learning · doi:10.22318/cscl2011.88 · Cited by 14 · see citing works
Flagged first, then by citations
2 of the 61 references couldn’t be matched to a known paper.
Research works printed but not linked: 2
- [33]Honkanen P, Frewer L. Russian consumers’ motives for food choice. Appetite 2009 4; 52(2): 363– 371. [doi: 10.1016/j.appet.2008.11.009] 19073227 · DOI
- [52]Bennett-Levy J, Richards D, Farrand P, Christensen H, Griffiths K, Kavanagh D, Klein B, Lau MA, Proudfoot J, Ritterband L, White J, Williams C. Oxford Guide to Low Intensity CBT Interventions. OUP Oxford; 2010 ISBN:978-0-19-102935-6
Cited by
Sorted by citations
Borghouts, J., Eikey, E., Mark, G., De Leon, C., Schueller, S. M., Schneider, M., Stadnick, N., Zheng, K., Mukamel, D., & Sorkin, D. H. (2021) · Journal of Medical Internet Research · doi:10.2196/24387 · Cited by 1,069 · see citing works
“Usability issues formed a barrier to engaging with an intervention [46,50, 78,84,148-150,157,159,170,224-228].”
BackgroundEngagement and retention in digital mental health interventions: a narrative review
Boucher, E. M., & Raiker, J. S. (2024) · BMC Digital Health · doi:10.1186/s44247-024-00105-9 · Cited by 201 · see citing works
“In one such study, Chen et al. examined patterns of usage across the Intellicare platform, which houses 13 different DMHIs [25].”
BackgroundWu, A., Scult, M. A., Barnes, E. D., Betancourt, J. A., Falk, A. E., & Gunning, F. M. (2021) · npj Digital Medicine · doi:10.1038/s41746-021-00386-8 · Cited by 164 · see citing works
Patient experiences with technology enabled care across healthcare settings- a systematic review
Leonardsen, A. L., Hardeland, C., Helgesen, A. K., & Grøndahl, V. A. (2020) · BMC Health Services Research · doi:10.1186/s12913-020-05633-4 · Cited by 139 · see citing works
Human-Computer Interaction in Digital Mental Health
Balcombe, L., & Leo, D. D. (2022) · Informatics · doi:10.3390/informatics9010014 · Cited by 146 · see citing works
“…of web-based and smartphone intervention users with the intent to increase engagement—the data analysis specified differences between the overall intervention and different aspects of it with the aim to demonstrate how passive data can help to individualize treatment and improve/assure quality [49].”
BackgroundChien, I., Enrique, A., Palacios, J., Regan, T., Keegan, D., Carter, D., Tschiatschek, S., Nori, A., Thieme, A., Richards, D., et al. (2020) · JAMA Network Open · doi:10.1001/jamanetworkopen.2020.10791 · Cited by 111 · see citing works
Chikersal, P., Belgrave, D. C. M., Doherty, G. J., Enrique, Á., Palacios, J. E., Richards, D., & Thieme, A. (2020) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376341 · Cited by 63 · see citing works
“Clustering & Text Mining in (Large-Scale) Mental Health Data Clustering is an unsupervised ML technique to identify behavioral patterns in data through commonalities in each data piece; it is often used to identify features in unstructured data ( e.g. [2, 14, 68]).”
MethodWang, X., Markert, C., & Sasangohar, F. (2021) · Human Factors The Journal of the Human Factors and Ergonomics Society · doi:10.1177/0018720821998110 · Cited by 61 · see citing works
O'Brien, H. L., Morton, E., Kampen, A., Barnes, S. J., & Michalak, E. E. (2020) · BJPsych Open · doi:10.1192/bjo.2020.72 · Cited by 57 · see citing works
Human Support in App-Based Cognitive Behavioral Therapies for Emotional Disorders: Scoping Review
Bernstein, E. E., Weingarden, H. M., Wolfe, E. C., Hall, M. D., Snorrason, Í., & Wilhelm, S. (2022) · Journal of Medical Internet Research · doi:10.2196/33307 · Cited by 55 · see citing works
“In total, 3 trials of the IntelliCare suite or its individual apps, designed to help patients with depressive or anxious symptoms, described the relationship between coaching and objective metrics of engagement [38,51,53].”
BackgroundInfluentialOr, C. K., Holden, R. J., & Valdez, R. S. (2022) · Automation, collaboration, and e-services · doi:10.1007/978-3-031-10788-7_6 · Cited by 48 · see citing works
Bell, L., Garnett, C., Bao, Y., Cheng, Z., Qian, T., Perski, O., Potts, H. W. W., & Williamson, E. (2023) · JMIR mHealth and uHealth · doi:10.2196/38342 · Cited by 35 · see citing works
Consumer Acceptance of App Push Notifications: Systematic Review on the Influence of Frequency
Wohllebe, A. (2020) · International Journal of Interactive Mobile Technologies (iJIM) · doi:10.3991/ijim.v14i13.14563 · Cited by 33 · see citing works
“The frequency of notifications also increases the overall use of the app [25], [29], [32], whereby particularly active users of a smartphone app are more open to higher notification frequencies [27].”
BackgroundBell, L., Garnett, C. V., Qian, T., Perski, O., Williamson, E., & POTTS, H. W. W. (2020) · Journal of Medical Internet Research · doi:10.2196/23369 · Cited by 33 · see citing works
“In previous digital health research, visualizations have delivered at a glance insight from mass volume and time-varying data, including more sophisticated displays of spatiotemporal, contextual, and event-centric outcomes [34-38].”
BackgroundAdaptive Systems for Internet-Delivered Psychological Treatments
Mukhiya, S. K., Wake, J. D., İnal, Y., & Lamo, Y. (2020) · IEEE Access · doi:10.1109/access.2020.3002793 · Cited by 28 · see citing works
Mata-Greve, F., Johnson, M., Pullmann, M. D., Friedman, E. C., Fillipo, I. G., Comtois, K. A., & Areán, P. A. (2021) · JMIR Mental Health · doi:10.2196/28360 · Cited by 19 · see citing works
“As Mohr and colleagues [17] have noted, digital mental health service use could be improved if intervention developers better understood what features people felt were important to have, the usability of these tools, and what role these services should have in the context of mental wellness [18-20].”
BackgroundMethodCheung, L., Thomas, P. C., Brvar, E., & Rowe, S. L. (2024) · JMIR Mental Health · doi:10.2196/57795 · Cited by 19 · see citing works
“…the product, such as visual and auditory feedback; (6) identification, how users perceive and connect with the product, including their sense of ownership; and (7) perceived user control and evocation, the cognitive responses generated in relation to the product, and their consequences [34, 35].”
Aschbacher, K., Rivera, L. M., Hornstein, S., Nelson, B. W., Forman‐Hoffman, V. L., & Peiper, N. C. (2023) · Psychosomatic Medicine · doi:10.1097/psy.0000000000001230 · Cited by 18 · see citing works
“, therapist messages, videos, group chats, heart rate variability biofeedback) was beyond the scope of this study because of the significant methodological complexity, future digital phenotyping studies will be necessary to better understand differential engagement in DMHIs (49,50).”
BackgroundResultStecher, C. D., Berardi, V., Fowers, R., Christ, J., Chung, Y., & Huberty, J. (2021) · Journal of Medical Internet Research · doi:10.2196/27282 · Cited by 15 · see citing works
“To characterize app use before and after completing the survey, we calculated the number of total sessions used, number of unique sessions used, duration of use, start time, and whether any app use occurred on a given day [27-31].”
Drews‐Windeck, E., Greenwood, K., & Cavanagh, K. (2023) · Journal of Clinical Psychology · doi:10.1002/jclp.23523 · Cited by 15 · see citing works
Ranjbartabar, H., Richards, D., BILGIN, A. A. B., Kutay, C., & Mascarenhas, S. F. (2020) · Multimodal Technologies and Interaction · doi:10.3390/mti4030055 · Cited by 13 · see citing works
“[29] used log data from a health app to understand user engagement.”
MethodNicol, G. E., Ricchio, A., Metts, C. L., Yingling, M. D., Ramsey, A. T., Schweiger, J. A., Miller, J. P., & Lenze, E. J. (2020) · JMIR mhealth and uhealth · doi:10.2196/18609 · Cited by 11 · see citing works
“In contrast, more traditional usability assessments conducted at the end of an intervention, even when paired with utilization data, are subject to recall bias and do not capture how end users experience the app in real time [47].”
BackgroundPerski, O., Watson, N. L., Mull, K. E., & Bricker, J. B. (2021) · Nicotine & Tobacco Research · doi:10.1093/ntr/ntab008 · Cited by 12 · see citing works
User Engagement in an Online Digital Health Intervention to Promote Problem Solving
O'Brien, H. L., Chen, A., Kaneshiro, J., & Zaslavsky, O. (2024) · Interacting with Computers · doi:10.1093/iwc/iwae030 · Cited by 11 · see citing works
Self-citationK‐Means Centroids Initialization Based on Differentiation Between Instances Attributes
Khan, A. A., Bashir, M. S., Batool, A., Raza, M. S., & Bashir, M. A. (2024) · International Journal of Intelligent Systems · doi:10.1155/2024/7086878 · Cited by 9 · see citing works
Resilience in the Surgical Scheduling to Support Adaptive Scheduling System
Wiyartanti, L., Lim, C. H., Park, M.-W., Kim, J. K., Kwon, G. H., & Kim, L. (2020) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph17103511 · Cited by 9 · see citing works
Aye, L. M., Tan, M., Schaefer, A., Thurairajasingam, S., Geldsetzer, P., Soon, L.-K., Reininghaus, U., Bärnighausen, T., & Su, T. T. (2024) · Digital Health · doi:10.1177/20552076241278313 · Cited by 12 · see citing works
“24,25 Despite the wealth of knowledge in digitally delivered psychotherapeutic techniques, several important questions remain in this fi eld.”
BackgroundAgeing and Alcohol: Drinking Typologies among Older Adults
ROCHE, A. M., Harrison, N. J., Chapman, J., Kostadinov, V., & Woodman, R. J. (2020) · Journal of Aging and Health · doi:10.1177/0898264320936953 · Cited by 7 · see citing works
Bernstein, E. E., Greenberg, J. L., Weingarden, H. M., Snorrason, Í., Summers, B. J., Williams, J., Quist, R. E., Curtiss, J. E., Harrison, O., & Wilhelm, S. (2024) · Internet Interventions · doi:10.1016/j.invent.2024.100743 · Cited by 7 · see citing works
“Although it is known that coaches typically send significantly more messages than users (Mohr et al., 2017; Carolan et al., 2017; Chen et al., 2019; Goldin et al., 2019; Graham et al., 2020; Newman et al., 2020), it remains unclear whether the overall number of exchanges with, or messages sent to,…”
BackgroundÖzdemir, M. C., Mõttus, M., & Lamas, D. (2024) · Applied Sciences · doi:10.3390/app15010014 · Cited by 6 · see citing works
Measuring and Fostering Engagement with Mental Health e-Coaches
Callejas, Z., Griol, D., Benghazi, K., Noguera, M., Chollet, G., Torres, M. I., & Esposito, A. A. (2020) · Companion Publication of the International Conference on Multimodal Interaction (ICMI Companion) · doi:10.1145/3395035.3425316 · Cited by 5 · see citing works
“evidence-based care, engage patients more actively and establish a communication line that can be active after treatment [8].”
BackgroundAdjei-Boateng, D., & Ikoh, C. L. (2025) · Cureus · doi:10.7759/cureus.85943 · Cited by 6 · see citing works
“For example, many apps track user behavior, such as the amount of time spent on the platform or engagement with certain features [88,106].”
Examining Engagement and Usability in an Online Discussion Platform for Older Adults
Han, S., Teng, A. K., Lin, S., Demiris, G., Zaslavsky, O., & Chen, A. (2023) · CIN Computers Informatics Nursing · doi:10.1097/cin.0000000000001001 · Cited by 5 · see citing works
Self-citationBeutter, C. N. L., Zeller, K., Martens, U. M., Pfleiderer, B., & Fegeler, C. (2023) · JMIR Cancer · doi:10.2196/44985 · Cited by 5 · see citing works
“Since the literature states that user engagement should only be measured if usability is already evaluated at a high level [26,34,35], we decided to assess user engagement only in the beta test after the previous usability tests.”
BackgroundEmotional anthropomorphism of notifications and app engagement: does the usage frequency matter?
Li, Y., Hwang, I., & Choi, J. (2025) · Journal of Research in Interactive Marketing · doi:10.1108/jrim-10-2024-0488 · Cited by 4 · see citing works
Reyes, R. C., Sevilla, R. V., Zapanta, G. S., Merin, J. V., Maaliw, R. R., III, & Santiago, A. F. (2022) · IEEE International Conference on Electronics, Computing and Communication Technologies (CONECCT) · doi:10.1109/conecct55679.2022.9865757 · Cited by 4 · see citing works
“Such methods can provide valuable information, which can eventually be examined to characterize employee conduct [3] and identify the construction site's safety environment and attitude [4].”
BackgroundMapping User Trajectories to Examine Behavior and Outcomes in Digital Health Intervention Data
Chen, A., Chang, J. H., Hallinan, S., & Mohr, D. C. (2019) · Workshop on Visual Analytics in Healthcare (VAHC) · doi:10.1109/vahc47919.2019.8945038 · Cited by 4 · see citing works
“examine app usage[10], [22], in this work, we employ a flowbased visualization to group participants by trajectories and”
MethodSelf-citationBaldwin, P. A., Black, M., Newby, J. M., Brown, L., Scott, N., Shrestha, T., Cockayne, N. L., Tennant, J., Harvey, S. B., & Christensen, H. M. (2022) · BMJ Innovations · doi:10.1136/bmjinnov-2021-000807 · Cited by 4 · see citing works
Opie, J. E., Vuong, A. T., McIntosh, J., & Kuntsche, S. (2025) · Mental Health and Digital Technologies · doi:10.1108/mhdt-07-2025-0050 · Cited by 2 · see citing works
Peluang Penelitian UI/UX pada Pengembangan Aplikasi Mobile: Systematic Literature Review
Sufandi, U. U., Trihapningsari, D., & Mellysa, W. (2022) · Techno Com · doi:10.33633/tc.v21i3.6059 · Cited by 3 · see citing works
Liu, C., Fuller‐Tyszkiewicz, M., Linardon, J., Jarman, H. K., & Messer, M. (2025) · Behavior Therapy · doi:10.1016/j.beth.2025.09.008 · Cited by 2 · see citing works
A Digital Mental Health Intervention for Children and Parents Using a User-Centred Design
Mahlous, A. R., & Okkali, B. (2022) · Advances in Human-Computer Interaction · doi:10.1155/2022/4322177 · Cited by 2 · see citing works
“As [11] states that in this way, there has been an extraordinary intrigue in collaborating with digital interventions and numerous definitions of support have developed.”
Backgroundvan Schalkwyk, A., Grobbelaar, S. S. S., Vermeulen, E., & Herselman, M. E. (2022) · IEEE Access · doi:10.1109/access.2022.3224038 · Cited by 2 · see citing works
Parra-Soto, S. L., Ortega, A., Celis-Morales, C. A., & Kain, J. (2020) · Global Pediatric Health · doi:10.1177/2333794x20961575 · Cited by 2 · see citing works
“Seventeen (17) questions were included with a Likert scale, based on the methodology proposed by Martin et al.”
MethodApproaches to Measuring User Engagement
O'Brien, H. L. (2025) · Synthesis lectures on information concepts, retrieval, and services · doi:10.1007/978-3-031-80916-3_5 · Cited by 1 · see citing works
Kelly, M. M., Moore, P. S., Zai, A., & Allison, J. J. (2026) · JMIR AI · doi:10.2196/85419 · Cited by 1 · see citing works
“According to Internet Addicts Anonymous, signs of AI-specific addiction include compulsive use of chatbots or voice-based agents, disruption of real-life relationships, emotional dysregulation when disconnected, and preference for AI interaction over human contact [8].”
Human Factors and Organizational Issues
Senathirajah, Y., Sylvia Pelayo, A., & Issues, S. E. F. T. I. Y. S. O. H. F. A. O. (2020) · Yearbook of Medical Informatics · doi:10.1055/s-0040-1702012
“, [8] made use of a multi-function suite of 13 clinical apps to address depression and other mental health issues, by measuring experience and user engagement via usage patterns, mechanisms of change, and usability.”
MethodHUMAN-COMPUTER INTERACTION IN MEDICAL EDUCATION
Mohamed, M. M. A., Darabee, H. A. M., & Soykan, E. (2021) · Near East University Online Journal of Education · doi:10.32955/neuje.v4i1.282
A scoping review of the use of log data for evaluating mobile Apps (Preprint)Preprint
van Schalkwyk, A., Grobbelaar, S. S. S., & Vermeulen, E. (2021) · doi:10.2196/preprints.31407
Conceptual Approaches to User Engagement
O'Brien, H. L. (2025) · Synthesis lectures on information concepts, retrieval, and services · doi:10.1007/978-3-031-80916-3_2
Silva, L. M., dos Santos, M. I. Q. L., Silva, K. D., Seixas, B. F. D. A., Chiacchio, F. B. B., & Chiacchio, A. D. (2025) · Revista de Gestão Social e Ambiental · doi:10.24857/rgsa.v19n4-083
Jung, G., Lim, H., Han, K., Kim, H., & Lee, U. (2025) · JMIR Human Factors · doi:10.2196/64967 · Cited by 1 · see citing works
“Group-level analysis has been widely used in previous studies on large-scale health data [13,24,51].”
Evia, Á. F. V. D. L. G., Eberle, J. W., Baee, S., Wolfe, E. C., Boukhechba, M., Funk, D. H., Teachman, B. A., & Barnes, L. E. (2025) · PLOS Digital Health · doi:10.1371/journal.pdig.0000945
“…features because cluster analysis’s use of multiple (vs. single or composite) metrics and its exploratory nature can suggest clusters that differ based on potentially more complex engagement patterns in the data than can be represented by single metrics or predicted a priori [16,17,19,35].”
Ballester, L., Gili, M., Roca, M., Forteza-Rey, I., Soto‐Sanz, V., Vilagut, G., Amigo, F., Moreno‐Küstner, B., Mortier, P., Piqueras, J. A., et al. (2025) · BMC Public Health · doi:10.1186/s12889-025-24752-3 · Cited by 1 · see citing works
Fässler, L., Ersöz, B., Hahne, I. M., Koop, S., & Böge, K. (2026) · Frontiers in Psychology · doi:10.3389/fpsyg.2025.1727156
Scherr, T., Hardcastle, A., Moore, C. P., Majji, D., Kuhns, L. M., Garofalo, R., & Schnall, R. (2026) · PLOS Digital Health · doi:10.1371/journal.pdig.0001452
“Although paradata has been employed in small pilot projects by our group and a few others [10–12], its substantial potential remains largely untapped in digital health, underscoring a gap in the utilization of digital tools in healthcare [13].”
Herold, M., Rodríguez‐Muñoz, A., Antino, M., & Simbula, S. (2026) · Stress and Health · doi:10.1002/smi.70203
- Authors
- Chen, A. T., Wu, S., Tomasino, K. N., Lattie, E. G., & Mohr, D. C.
- Affiliations
- University of Washington (US)
- Northwestern University (US)
- Corresponding author
- Chen, Annie T.
- Venue
- Journal of Biomedical Informatics
- Publisher
- Elsevier BV
- Published
- 2019-06
- Volume
- 94
- Pages
- 103187
- ISSN
- 1532-0464 (ISSN-L)
- PMC
- PMC6662914
- OpenAlex
- W2940596780
- Cited by
- 58
- References
- 61
- Citations by year
- 2026: 4 · 2025: 11 · 2024: 7 · 2023: 5 · 2022: 8 · 2021: 8 · 2020: 14 · 2019: 1
- FWCI
- 8.75
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Journal of Biomedical Informatics (publishedVersion, cc-by-nc-nd)
- PubMed (publishedVersion)
- PubMed Central (acceptedVersion)
- Funders
- NIMH (R01 MH100482, K08 MH112878)
- National Institute of Mental Health
- Topics
- Social Sciences › Psychology › Applied Psychology › Digital Mental Health Interventions
- Health Sciences › Health Professions › General Health Professions › Mobile Health and mHealth Applications
- Physical Sciences › Computer Science › Human-Computer Interaction › Innovative Human-Technology Interaction
- Keywords
- digital mental health interventions, user engagement, app usage behavior, user experience, mental health apps, mechanisms of change, usability, personalization
- MeSH terms
- Attitude to Computers, Mental Disorders, Humans, Mental Health, Telemedicine, Mobile Applications
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
