Exploring user learnability and learning performance in an app for depression: usability study
Open accessAbstract
BACKGROUND: Mental health apps tend to be narrow in their functioning, with their focus mostly being on tracking, management, or psychoeducation. It is unclear what capability such apps have to facilitate a change in users, particularly in terms of learning key constructs relating to behavioral interventions. Thought Challenger (CBITs, Chicago) is a skill-building app that engages users in cognitive restructuring, a core component of cognitive therapy (CT) for depression. OBJECTIVE: The purpose of this study was to evaluate the learnability and learning performance of users following initial use of Thought Challenger. METHODS: Twenty adults completed in-lab usability testing of Thought Challenger, which comprised two interactions with the app. Learnability was measured via completion times, error rates, and psychologist ratings of user entries in the app; learning performance was measured via a test of CT knowledge and skills. Nonparametric tests were conducted to evaluate the difference between individuals with no or mild depression to those with moderate to severe depression, as well as differences in completion times and pre- and posttests. RESULTS: Across the two interactions, the majority of completion times were found to be acceptable (5 min or less), with minimal errors (1.2%, 10/840) and successful completion of CT thought records. Furthermore, CT knowledge and skills significantly improved after the initial use of Thought Challenger (P=.009). CONCLUSIONS: The learning objectives for Thought Challenger during initial uses were successfully met in an evaluation with likely end users. The findings therefore suggest that apps are capable of providing users with opportunities for learning of intervention skills.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The intervention promptly proceeds to the subsequent stage once the user reports the initial NAT they identify, which may unintentionally lead to the omission of other important NAT.
“The intervention promptly proceeds to the subsequent stage once the user reports the initial NAT they identify, which may unintentionally lead to the omission of other important NAT (13), potentially compromising effectiveness (14).”
From A virtual reality-based self-guided training on identification of negative automatic thoughts in healthy adults: a mixed-methods feasibility study (2024) · cites this paper as [13]
The Methods section (“Thought Challenger”; page unavailable) describes a sequence in which the user enters a maladaptive thought before moving to reflective questions. It does not say that the app advances promptly or automatically after the first thought, or that this sequence may cause users to omit other important negative automatic thoughts. The separate effectiveness clause is attributed to citation (14), not this paper. (LLM, conf=0.78)
Experimental — accuracy not yet measuredAfter training, participants demonstrated a higher level of proficiency in identifying NAT in their homework and a better understanding of cognitive theory, surpassing previous research on APP intervention conducted in non-clinical populations.
“After training, participants demonstrated a higher level of proficiency in identifying NAT in their homework and a better understanding of cognitive theory, surpassing previous research on APP intervention conducted in non-clinical populations (13).”
From A virtual reality-based self-guided training on identification of negative automatic thoughts in healthy adults: a mixed-methods feasibility study (2024) · cites this paper as [13]
Split into 2 atoms — 0 confirmed, 2 partially supported, 0 disputed, 0 misattributed, 0 unverifiable. Expand the claim to see per-atom verdicts.
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Harris, P. A., Taylor, R., Thielke, R., Payne, J., Gonzalez, N., & Conde, J. G. (2009) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2008.08.010 · Cited by 51,384 · see citing works
The PHQ-9: A New Depression Diagnostic and Severity Measure
Kroenke, K., & Spitzer, R. L. (2002) · Psychiatric Annals · doi:10.3928/0048-5713-20020901-06 · Cited by 5,898 · see citing works
(1993) · Elsevier eBooks · doi:10.1016/c2009-0-21512-1 · Cited by 5,533 · see citing works
Health App Use Among US Mobile Phone Owners: A National Survey
Krebs, P., & Duncan, D. T. (2015) · JMIR mHealth and uHealth · doi:10.2196/mhealth.4924 · Cited by 1,274 · see citing works
Recurrence after recovery from major depressive disorder during 15 years of observational follow-up
Mueller, T. I., Leon, A. C., Keller, M. B., Solomon, D. A., Endicott, J., Coryell, W., Warshaw, M., & Maser, J. D. (1999) · American Journal of Psychiatry · doi:10.1176/ajp.156.7.1000 · Cited by 897 · see citing works
Usability of mobile applications: literature review and rationale for a new usability model
Harrison, R., Flood, D., & Duce, D. (2013) · Journal of Interaction Science · doi:10.1186/2194-0827-1-1 · Cited by 736 · see citing works
Mohr, D. C., Burns, M. N., Schueller, S. M., Clarke, G., & Klinkman, M. (2013) · General Hospital Psychiatry · doi:10.1016/j.genhosppsych.2013.03.008 · Cited by 693 · see citing works
Challenges, Methodologies, and Issues in the Usability Testing of Mobile Applications
Zhang, D., & Adipat, B. (2005) · International Journal of Human-Computer Interaction · doi:10.1207/s15327590ijhc1803_3 · Cited by 613 · see citing works
Interfacing Thought: Cognitive Aspects of Human-Computer Interaction
Carroll, J. M. (2003) · Medical Entomology and Zoology · Cited by 407 · see citing works
Ben‐Zeev, D., Kaiser, S. M., Brenner, C. J., Begale, M. J., Duffecy, J. L., & Mohr, D. C. (2013) · Psychiatric Rehabilitation Journal · doi:10.1037/prj0000019 · Cited by 306 · see citing works
IntelliCare: An Eclectic, Skills-Based App Suite for the Treatment of Depression and Anxiety
Mohr, D. C., Tomasino, K. N., Lattie, E. G., Palac, H. L., Kwasny, M. J., Weingardt, K., Karr, C. J., Kaiser, S. M., Rossom, R. C., Bardsley, L. R., et al. (2017) · Journal of Medical Internet Research · doi:10.2196/jmir.6645 · Cited by 301 · see citing works
A method to standardize usability metrics into a single score
Sauro, J., & Kindlund, E. (2005) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/1054972.1055028 · Cited by 299 · see citing works
Torous, J., Friedman, R., & Keshavan, M. (2014) · JMIR mHealth and uHealth · doi:10.2196/mhealth.2994 · Cited by 287 · see citing works
Finding a Depression App: A Review and Content Analysis of the Depression App Marketplace
Shen, N., Levitan, M.-J., Johnson, A., Bender, J. L., Hamilton-Page, M., Jadad, A. (. R., & Wiljer, D. (2015) · JMIR mHealth and uHealth · doi:10.2196/mhealth.3713 · Cited by 282 · see citing works
Usability of Commercially Available Mobile Applications for Diverse Patients
Sarkar, U., Gourley, G. I., Lyles, C. R., Tieu, L., Clarity, C., Newmark, L., Singh, K., & Bates, D. W. (2016) · Journal of General Internal Medicine · doi:10.1007/s11606-016-3771-6 · Cited by 257 · see citing works
Mohr, D. C., Schueller, S. M., Riley, W. T., Brown, C. H., Cuijpers, P., Duan, N., Kwasny, M. J., Stiles-Shields, C., & Cheung, K. (2015) · Journal of Medical Internet Research · doi:10.2196/jmir.4391 · Cited by 216 · see citing works
mHealth in the Wild: Using Novel Data to Examine the Reach, Use, and Impact of PTSD Coach
Owen, J. E., Jaworski, B. K., Kuhn, E., Makin-Byrd, K. N., Ramsey, K. M., & Hoffman, J. E. (2015) · JMIR Mental Health · doi:10.2196/mental.3935 · Cited by 216 · cited 1× · see citing works
“The possibility for such errors within an app may be one reason that behavioral health apps show low rates of retention when deployed in public marketplaces [45].”
BackgroundAgile science: creating useful products for behavior change in the real world
Hekler, E. B., Klasnja, P., Riley, W. T., Buman, M. P., Huberty, J., Rivera, D. E., & Martin, C. A. (2016) · Translational Behavioral Medicine · doi:10.1007/s13142-016-0395-7 · Cited by 208 · see citing works
Partial remission, residual symptoms, and relapse in depression
Paykel, E. S. (2008) · Dialogues in Clinical Neuroscience · doi:10.31887/dcns.2008.10.4/espaykel · Cited by 194 · see citing works
On second thought: Where the action is in cognitive therapy for depression
Barber, J. P., & DeRubeis, R. J. (1989) · Cognitive Therapy and Research · doi:10.1007/bf01173905 · Cited by 184 · see citing works
Hundt, N. E., Mignogna, J., Underhill, C., & Cully, J. A. (2012) · Behavior Therapy · doi:10.1016/j.beth.2012.10.001 · Cited by 143 · see citing works
Development and Initial Testing of a Multimedia Program for Computer-Assisted Cognitive Therapy
Wright, J. H., Wright, A. S., Salmon, P. M., Beck, A. T., Kuykendall, J., Goldsmith, L. J., & Zickel, M. B. (2002) · American Journal of Psychotherapy · doi:10.1176/appi.psychotherapy.2002.56.1.76 · Cited by 124 · see citing works
Current research and trends in the use of smartphone applications for mood disorders
Torous, J., & Powell, A. C. (2015) · Internet Interventions · doi:10.1016/j.invent.2015.03.002 · Cited by 121 · see citing works
Relationship Between Homework Completion and Outcome in Cognitive Behaviour Therapy
Rees, C. S., McEvoy, P. M., & Nathan, P. R. (2005) · Cognitive Behaviour Therapy · doi:10.1080/16506070510011548 · Cited by 116 · see citing works
Uptake and usage of IntelliCare: A publicly available suite of mental health and well-being apps
Lattie, E. G., Schueller, S. M., Sargent, E., Stiles-Shields, C., Tomasino, K. N., Corden, M. E., Begale, M., Karr, C. J., & Mohr, D. C. (2016) · Internet Interventions · doi:10.1016/j.invent.2016.06.003 · Cited by 113 · see citing works
Exploring mental health providers' interest in using web and mobile-based tools in their practices
Schueller, S. M., Washburn, J. J., & Price, M. (2016) · Internet Interventions · doi:10.1016/j.invent.2016.06.004 · Cited by 98 · see citing works
Self-citationMobile Learning: Transforming the delivery of education and training
Gorham, R. (2010) · International journal of e-learning & distance education · Cited by 93 · see citing works
What might get in the way: Barriers to the use of apps for depression
Stiles-Shields, C., Montague, E., Lattie, E. G., Kwasny, M. J., & Mohr, D. C. (2017) · DIGITAL HEALTH · doi:10.1177/2055207617713827 · Cited by 68 · see citing works
Self-citationJudd, L. L. (1999) · Archives of General Psychiatry · doi:10.1001/archpsyc.56.8.764 · Cited by 67 · see citing works
Judd, L. L., Paulus, M. P., & Zeller, P. J. (1999) · Archives of General Psychiatry · doi:10.1001/archpsyc.56.8.763 · Cited by 51 · see citing works
User Experience Evaluation of a Smoking Cessation App in People With Serious Mental Illness
Vilardaga, R., Rizo, J. L., Kientz, J. A., McDonell, M. G., Ries, R. K., & Sobel, K. (2015) · Nicotine & Tobacco Research · doi:10.1093/ntr/ntv256 · Cited by 45 · see citing works
Learning cognitive behavior therapy
Gumport, N. B., Williams, J. J., & Harvey, A. G. (2015) · Journal of Behavior Therapy and Experimental Psychiatry · doi:10.1016/j.jbtep.2015.03.015 · Cited by 40 · see citing works
Cognitive Behavior Therapy, Basics and Beyond, Second Edition
Davidson, J. (2013) · Journal of Psychiatric Practice · doi:10.1097/01.pra.0000432605.83964.85 · Cited by 33 · see citing works
Mohr, D. C., Montague, E. N. H., Stiles‐Shields, C., Kaiser, S. M., Brenner, C. J., Palac, H., Carty-Fickes, E., & Duffecy, J. (2015) · EAI Endorsed Transactions on Pervasive Health and Technology · doi:10.4108/icst.pervasivehealth.2015.259042 · Cited by 32 · see citing works
Evaluating Mobile Applications: A Spreadsheet Case Study
Flood, D., Harrison, R., Iacob, C., & Duce, D. (2012) · International Journal of Mobile Human Computer Interaction · doi:10.4018/jmhci.2012100103 · Cited by 29 · see citing works
Evidence-based Health Informatics and the Scientific Development of the Field
Wyatt Jeremy, C. (2016) · Studies in health technology and informatics · doi:10.3233/978-1-61499-635-4-14 · Cited by 20 · see citing works
Luchini, K., Quintana, C., & Soloway, E. M. (2003) · ACM Conference on Human Factors in Computing Systems (CHI) · doi:10.1145/642667.642668 · Cited by 2 · see citing works
Flagged first, then by citations
2 of the 39 references couldn’t be matched to a known paper.
Research works printed but not linked: 1
- [19]Tullis T Albert B Measuring the user experience: Collecting, analyzing, and presenting usability metrics, second edition 2008 Burlington, MA Morgan Kaufmann
Cited by
Sorted by citations
Lattie, E. G., Adkins, E. C., Winquist, N., Stiles-Shields, C., Wafford, Q. E., & Graham, A. K. (2019) · Journal of Medical Internet Research · doi:10.2196/12869 · Cited by 843 · see citing works
Smartphone Apps for the Treatment of Mental Disorders: Systematic Review
Miralles, I., Granell, C., Díaz-Sanahuja, L., Van Woensel, W., Bretón-López, J., Mira, A., Castilla, D., & Casteleyn, S. (2020) · JMIR mHealth and uHealth · doi:10.2196/14897 · Cited by 188 · see citing works
Hornstein, S., Zantvoort, K., Lueken, U., Funk, B., & Hilbert, K. (2023) · Frontiers in Digital Health · doi:10.3389/fdgth.2023.1170002 · Cited by 134 · see citing works
Vilardaga, R., Rizo, J., Zeng, E., Kientz, J. A., Ries, R., Otis, C., & Hernandez, K. (2018) · JMIR Serious Games · doi:10.2196/games.8881 · Cited by 90 · see citing works
A mobile app–based intervention for depression: end-user and expert usability testing study
Fuller-Tyszkiewicz, M., Richardson, B., Klein, B., Skouteris, H., Christensen, H., Austin, D., Castle, D., Mihalopoulos, C., O'Donnell, R., Arulkadacham, L., et al. (2018) · JMIR Mental Health · doi:10.2196/mental.9445 · Cited by 98 · see citing works
Holden, R. J., Campbell, N. L., Abebe, E., Clark, D. O., Ferguson, D. Y., Bodke, K., Boustani, M. A., & Callahan, C. M. (2019) · Research in Social and Administrative Pharmacy · doi:10.1016/j.sapharm.2019.02.011 · Cited by 64 · see citing works
Cheung, K., Ling, W., Karr, C. J., Weingardt, K. R., Schueller, S. M., & Mohr, D. C. (2018) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocy023 · Cited by 61 · see citing works
Stiles-Shields, C., Montague, E., Kwasny, M. J., & Mohr, D. C. (2019) · Psychological Services · doi:10.1037/ser0000261 · Cited by 59 · see citing works
Wainer, A., Arnold, Z. E., Leonczyk, C., & Soorya, L. V. (2021) · Molecular Autism · doi:10.1186/s13229-021-00443-9 · Cited by 57 · see citing works
Perin, S., Buckley, R. F., Pase, M. P., Yassi, N., Lavale, A., Wilson, P. H., Schembri, A. J., Maruff, P. T., & Lim, Y. Y. (2020) · Alzheimer s & Dementia Translational Research & Clinical Interventions · doi:10.1002/trc2.12043 · Cited by 49 · see citing works
Gamification Acceptance for Learners with Different E-Skills
Panagiotarou, A., Stamatiou, Y., Pierrakeas, C., & Kameas, A. (2020) · International Journal of Learning, Teaching and Educational Research · doi:10.26803/ijlter.19.2.16 · Cited by 36 · see citing works
Kruzan, K. P., Meyerhoff, J., Biernesser, C. L., Goldstein, T. R., Reddy, M. C., & Mohr, D. C. (2021) · JMIR Mental Health · doi:10.2196/31367 · Cited by 30 · see citing works
Towards Understanding the Usability Attributes of AI-Enabled eHealth Mobile Applications
Alzahrani, A., Gay, V. C., Alturki, R., & AlGhamdi, M. J. (2021) · Journal of Healthcare Engineering · doi:10.1155/2021/5313027 · Cited by 19 · see citing works
RetractedDesign recommendations for chatbots to support people with depression
de Souza, P. M., Pires, I. D. C., Motti, V. G., Caseli, H. D. M., Neto, J. B. B., Martini, L. C., & Néris, V. P. D. A. (2022) · Brazilian Symposium on Human Factors in Computing Systems (IHC) · doi:10.1145/3554364.3559119 · Cited by 19 · see citing works
Sinclair, R., Nodi, S. S., & Kabir, M. A. (2023) · Computers and Electronics in Agriculture · doi:10.1016/j.compag.2023.108527 · Cited by 19 · see citing works
Yee, L. M., Leziak, K., Jackson, J., Niznik, C. M., Saber, R., Yeh, C., & Simon, M. A. (2022) · Diabetes Spectrum · doi:10.2337/ds22-0014 · Cited by 13 · see citing works
Learnability Assessment of Speech-Based Intelligent Personal Assistants by Older Adults
Islam, M. U., & Chaudry, B. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-34866-2_24 · Cited by 8 · see citing works
Identifying Usability Challenges of eHealth Applications for People with Mental Disorders
Blankenhagel, K. J. (2019) · International Conference on Pervasive Computing Technologies for Healthcare · doi:10.1145/3329189.3329195 · Cited by 7 · see citing works
Stiles‐Shields, C., Garcia, B., Villota, K., Wartman, E. C., Winning, A. M., & Holmbeck, G. N. (2019) · JMIR Pediatrics and Parenting · doi:10.2196/15153 · Cited by 6 · see citing works
Neumayr, C., Voderholzer, U., & Schlegl, S. (2020) · Verhaltenstherapie · doi:10.1159/000510954 · Cited by 5 · see citing works
Ganapathy, T., Othman, M. K., & Al-Hababi, A. Y. S. (2021) · Journal of Visual Art and Design · doi:10.5614/j.vad.2021.13.1.2 · Cited by 3 · see citing works
Understanding E-Mental Health for People with Depression: An Evaluation Study
Blankenhagel, K. J., Werner, J., Mayer, G., Schultz, J., & Zarnekow, R. (2021) · Lecture notes of the Institute for Computer Sciences, Social Informatics and Telecommunications Engineering · doi:10.1007/978-3-030-70569-5_3 · Cited by 3 · see citing works
Scaling Our Impact: Emerging Human Factors Applications Addressing Societal Challenges
Rogers, C. C., Valdez, R. S., Gilbert, J. E., Lange-Morales, K., Rogers, W. A., Thatcher, A., & Wooldridge, A. R. (2020) · Proceedings of the Human Factors and Ergonomics Society Annual Meeting · doi:10.1177/1071181320641200 · Cited by 3 · see citing works
Yang, B., Liao, C., Yang, Y., Shi, B., Zhang, C., & Li, C. (2024) · Frontiers in Psychiatry · doi:10.3389/fpsyt.2024.1479207 · Cited by 2 · see citing works
Stiles‐Shields, C., & Holmbeck, G. N. (2018) · Journal of Pediatric Psychology · doi:10.1093/jpepsy/jsy018 · Cited by 2 · see citing works
Android-Based Mobile Grade Viewer Application Using PHP
Villanueva, C. A., Cacayuran, K. C. A., & Munar, E. A. (2021) · IEEE International Conference on Humanoid, Nanotechnology, Information Technology, Communication and Control, Environment, and Management (HNICEM) · doi:10.1109/hnicem54116.2021.9731936
Investigating mobile blended learning adoption with usability factors: an empirical study
Chand, S. S., & Kumar, B. A. (2024) · Interactive Technology and Smart Education · doi:10.1108/itse-08-2024-0182 · Cited by 2 · see citing works
- Authors
- Stiles-Shields, C., Montague, E., Lattie, E. G., Schueller, S. M., Kwasny, M. J., & Mohr, D. C.
- Affiliations
- Northwestern University (US)
- ORCID
- Venue
- JMIR Human Factors
- Publisher
- JMIR Publications Inc.
- Published
- 2017-08-11
- Volume
- 4
- Issue
- 3
- Pages
- e18
- ISSN
- 2292-9495 (ISSN-L)
- PMC
- PMC5573426
- OpenAlex
- W2743667149
- Cited by
- 29
- References
- 39
- Citations by year
- 2024: 2 · 2023: 3 · 2022: 2 · 2021: 7 · 2020: 6 · 2019: 4 · 2018: 5
- FWCI
- 4.69
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- Locations
- JMIR Human Factors (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Funders
- National Institutes of Health (RR025780)
- National Institute of Mental Health (R01MH100482, F31MH106321, R01MH095753, K08MH102336)
- National Center for Research Resources (RR-025780, UL1RR025780)
- Topics
- Social Sciences › Psychology › Applied Psychology › Digital Mental Health Interventions
- Physical Sciences › Computer Science › Human-Computer Interaction › Usability and User Interface Design
- Health Sciences › Health Professions › General Health Professions › Mobile Health and mHealth Applications
- Keywords
- mental health apps, depression, cognitive restructuring, cognitive behavioral therapy, learnability, learning performance, usability, thought records, digital mental health interventions
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
