Experiences of a Speech-enabled Conversational Agent for the Self-report of Well-being among People Living with Affective Disorders: An In-the-Wild Study
Open accessThe DOI matches its Crossref record; 43 of the 43 references it cites were found in a registry or an index.
Abstract
The growing commercial success of smart speaker devices following recent advancements in speech recognition technology has surfaced new opportunities for collecting self-reported health and well-being data. Speech-enabled conversational agents (CAs) in particular, deployed in home environments using just such systems, may offer increasingly intuitive and engaging means of self-report. To date, however, few real-world studies have examined users’ experiences of engaging in the self-report of mental health using such devices or the challenges of deploying these systems in the home context. With these aims in mind, this article recounts findings from a 4-week “in-the-wild” study during which 20 individuals with depression or bipolar disorder used a speech-enabled CA named “Sofia” to maintain a daily diary log, responding also to the World Health Organization–Five Well-Being Index WHO-5 scale every 2 weeks. Thematic analysis of post-study interviews highlights actions taken by participants to overcome CAs’ limitations, diverse personifications of a speech-enabled agent, and unique forms of valuing of this system among users’ personal and social circles. These findings serve as initial evidence for the potential of CAs to support the self-report of mental health and well-being, while highlighting the need to address outstanding technical limitations in addition to design challenges of conversational pattern matching, filling unmet interpersonal gaps, and the use of self-report CAs in the at-home social context. Based on these insights, we discuss implications for the future design of CAs to support the self-report of mental health and well-being.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Using thematic analysis in psychology
Braun, V., & Clarke, V. (2006) · Qualitative Research in Psychology · doi:10.1191/1478088706qp063oa · Cited by 194,414
The WHO-5 well-being index: A systematic review of the literature
Topp, C. W., Østergaard, S. D., Søndergaard, S., & Bech, P. (2015) · Psychotherapy and Psychosomatics · doi:10.1159/000376585 · Cited by 4,708
ELIZA—a computer program for the study of natural language communication between man and machine
Weizenbaum, J. (1966) · Communications of the ACM · doi:10.1145/365153.365168 · Cited by 4,452 · cited 1×
“This irst of many text-based Natural Language Processing (NLP) programs to come provided pre-determined answers to user queries in mimicry of a Rogerian psychotherapist [91].”
BackgroundFitzpatrick, K. K., Darcy, A., & Vierhile, M. (2017) · JMIR Mental Health · doi:10.2196/mental.7785 · Cited by 2,730 · cited 1×
“, for example, developed a fully automated chatbot named ‘Woebot’ and studied its feasibility, acceptability, and eicacy in delivering a self-help CBT program for college students with symptoms of anxiety and depression [29].”
BackgroundConversational agents in healthcare: a systematic review
Laranjo, L., Dunn, A. G., Tong, H. L., Kocaballi, A. B., Chen, J., Bashir, R., Surian, D., Gallego, B., Magrabi, F., Lau, A. Y. S., et al. (2018) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocy072 · Cited by 1,388 · cited 1×
“…CAs remain in many ways nascent technologies, many HCI and healthcare researchers have already begun to explore CAs’ potential to support mental health and wellbeing in a variety of contexts and in various forms, from diagnosis to symptom monitoring, treatment and intervention [31, 44, 49, 84].”
BackgroundWho Sees Human?: The Stability and Importance of Individual Differences in Anthropomorphism
Waytz, A., Cacioppo, J., & Epley, N. (2010) · Perspectives on Psychological Science · doi:10.1177/1745691610369336 · Cited by 1,228
Luger, E., & Sellen, A. (2016) · Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems · doi:10.1145/2858036.2858288 · Cited by 1,176 · cited 2×
“And inally, studies have shown that users often establish high expectations for CAs [52], anticipating emotional exchanges, relationship building, and human-to-human like conversations [16], which often leads to a drastic decline in use when CAs cannot meet users’ expectations [16, 52].”
BackgroundInkster, B., Sarda, S., & Subramanian, V. (2018) · doi:10.2196/preprints.12106 · Cited by 1,063
Chatbots and Conversational Agents in Mental Health: A Review of the Psychiatric Landscape
Vaidyam, A. N., Wisniewski, H., Halamka, J. D., Kashavan, M. S., & Torous, J. B. (2019) · The Canadian Journal of Psychiatry · doi:10.1177/0706743719828977 · Cited by 1,048 · cited 3×
“While the results of these preliminary studies are promising, therapeutic applications of CAs remain at a very early stage and there are many technical, ethical and medical challenges which must be overcome before they can be ofered as a viable alternative for those in need of support [84].”
BackgroundIt's only a computer: virtual humans increase willingness to disclose
Lucas, G. M., Gratch, J., King, A., & Morency, L.-P. (2014) · Computers in Human Behavior · doi:10.1016/j.chb.2014.04.043 · Cited by 977 · cited 1×
“It has been suggested that this more human-like conversational mode of interaction may enable users to form a relationship with CAs [7, 88], and that CAs could in turn potentially serve as means to collect more accurate, honest and insightful information from users [23, 50].”
BackgroundCuijpers, P., Donker, T., van Straten, A., Li, J., & Andersson, G. (2010) · Psychological Medicine · doi:10.1017/s0033291710000772 · Cited by 832
Voice Interfaces in Everyday Life
Porcheron, M., Fischer, J. E., Reeves, S., & Sharples, S. (2018) · Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3174214 · Cited by 717 · cited 1×
“While others have begun to explore questions of user experience, trust, feedback, self-relection and learning during the collection of self-reports of mental and physical health and wellbeing [42, 69, 87] as well as social engagement [64, 72, 74].”
BackgroundLau, J., Zimmerman, B., & Schaub, F. (2018) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3274371 · Cited by 568
Harari, G. M., Lane, N. D., Wang, R., Crosier, B. S., Campbell, A. T., & Gosling, S. D. (2016) · Perspectives on Psychological Science · doi:10.1177/1745691616650285 · Cited by 515
Establishing the computer-patient working alliance in automated health behavior change interventions
Bickmore, T., Gruber, A., & Picard, R. (2005) · Patient Education and Counseling · doi:10.1016/j.pec.2004.09.008 · Cited by 494 · cited 3×
“It has been suggested that this more human-like conversational mode of interaction may enable users to form a relationship with CAs [7, 88], and that CAs could in turn potentially serve as means to collect more accurate, honest and insightful information from users [23, 50].”
BackgroundResultWhat Makes a Good Conversation?: Challenges in Designing Truly Conversational Agents
Clark, L., Pantidi, N., Cooney, O., Doyle, P., Garaialde, D., Edwards, J., Spillane, B., Gilmartin, E., Murad, C., Munteanu, C., et al. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300705 · Cited by 461 · cited 1×
“Several researchers have provided initial guidelines to address these limitations in support of the design of efective CAs [19, 43, 57, 77, 89], including for health and wellbeing management [75].”
BackgroundPradhan, A., Mehta, K., & Findlater, L. (2018) · Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3174033 · Cited by 448
The Experience Sampling Method on Mobile Devices
van Berkel, N., Ferreira, D., & Kostakos, V. (2018) · ACM Computing Surveys · doi:10.1145/3123988 · Cited by 432
Weiss, L. A., Westerhof, G. J., & Bohlmeijer, E. T. (2016) · PLOS ONE · doi:10.1371/journal.pone.0158092 · Cited by 405 · cited 10×
“Therapies entailing practices of self-report and self-monitoring such as Cognitive Behavioral Therapy (CBT) and Behavioral Activation (BA) are known to be efective in treating ADs [21, 79, 90].”
BackgroundInfluentialMiner, A. S., Milstein, A., Schueller, S., Hegde, R., Mangurian, C., & Linos, E. (2016) · JAMA Internal Medicine · doi:10.1001/jamainternmed.2016.0400 · Cited by 368
Wen, C. K. F., Schneider, S., Stone, A. A., & Spruijt-Metz, D. (2017) · Journal of Medical Internet Research · doi:10.2196/jmir.6641 · Cited by 350 · cited 1×
“91) Ð a igure comparable to rates of ‘compliance’ reported in prior EMA studies using mobile devices [92].”
ResultThe Personalization of Conversational Agents in Health Care: Systematic Review
Kocaballi, A. B., Berkovsky, S., Quiroz, J. C., Laranjo, L., Tong, H. L., Rezazadegan, D., Briatore, A., & Coiera, E. (2019) · Journal of Medical Internet Research · doi:10.2196/15360 · Cited by 332 · cited 1×
“Studies have shown that speech, as a primary mode of communication, can engage users in more natural and human-to-human-like conversations, suggesting the potential of these systems to improve user engagement and self-report response quality while fostering more honest and insightful forms of self-disclosure [23, 50]…”
BackgroundSocial Dialongue with Embodied Conversational Agents
Bickmore, T., & Cassell, J. (2005) · Text, Speech and Language Technology · doi:10.1007/1-4020-3933-6_2 · Cited by 324 · cited 1×
“A remote ‘Wizard-of-Oz’ study of another multi-modal agent designed to provide longitudinal social support to isolated older adults similarly reported high levels of user acceptance and satisfaction with the system [6].”
BackgroundReporting Mental Health Symptoms: Breaking Down Barriers to Care with Virtual Human Interviewers
Lucas, G. M., Rizzo, A., Gratch, J., Scherer, S., Stratou, G., Boberg, J., & Morency, L.-P. (2017) · Frontiers in Robotics and AI · doi:10.3389/frobt.2017.00051 · Cited by 308
Conversational agents in the treatment of mental health problems: mixed-method systematic review
Gaffney, H., Mansell, W., & Tai, S. (2019) · JMIR Mental Health · doi:10.2196/14166 · Cited by 275
Hoermann, S., McCabe, K. L., Milne, D. N., & Calvo, R. A. (2017) · Journal of Medical Internet Research · doi:10.2196/jmir.7023 · Cited by 263 · cited 1×
“Initial research suggests that by aiding in the assessment and treatment of mental illnesses including ADs, CAs may prove able to play a role in overcoming these challenges of access and interaction [34, 84].”
BackgroundTorous, J., Staples, P., Shanahan, M., Lin, C., Peck, P., Keshavan, M., & Onnela, J.-P. (2015) · JMIR Mental Health · doi:10.2196/mental.3889 · Cited by 258 · cited 1×
“In order to render the WHO-5 questionnaire more conversational in nature, slight variations were incorporated into the wording of the preamble and the questions, as previously practiced in digital mental health research [82].”
MethodResponse to a relational agent by hospital patients with depressive symptoms
Bickmore, T. W., Mitchell, S. E., Jack, B. W., Paasche-Orlow, M. K., Pfeifer, L. M., & O’Donnell, J. (2010) · Interacting with Computers · doi:10.1016/j.intcom.2009.12.001 · Cited by 218
Patterns for How Users Overcome Obstacles in Voice User Interfaces
Myers, C., Furqan, A., Nebolsky, J., Caro, K., & Zhu, J. (2018) · Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3173580 · Cited by 213 · cited 2×
“…that users themselves apply tactics including hyper-articulation and exaggeration, increased volume, use of diferent utterances or simpliied words, and reformulation strategies, such as addition or substitution, removal, and re-ordering of words to alleviate conversational barriers [15, 58, 62].”
BackgroundAnthropomorphism, agency, and ethopoeia: computers as social actors
Nass, C., Steuer, J., Tauber, E., & Reeder, H. (1993) · INTERACT '93 and CHI '93 conference companion on Human factors in computing systems - CHI '93 · doi:10.1145/259964.260137 · Cited by 162 · cited 1×
“Other studies have demonstrated that human-CA interactions tend to follow prevailing social norms despite users’ awareness that their interaction partner is a machine [59, 70].”
BackgroundPersonification of the Amazon Alexa: BFF or a Mindless Companion
Lopatovska, I., & Williams, H. (2018) · Proceedings of the 2018 Conference on Human Information Interaction&Retrieval - CHIIR '18 · doi:10.1145/3176349.3176868 · Cited by 160
Corbett, E., & Weber, A. (2016) · Proceedings of the 18th International Conference on Human-Computer Interaction with Mobile Devices and Services · doi:10.1145/2935334.2935386 · Cited by 148 · cited 1×
“Typically, users have to know in advance what they can or cannot say to the CA [20, 30, 65] which often makes it cognitively demanding for users to interact [65].”
BackgroundTechnology and mental health: The role of artificial intelligence
Lovejoy, C. A. (2019) · European Psychiatry · doi:10.1016/j.eurpsy.2018.08.004 · Cited by 129
Philip, P., Micoulaud-Franchi, J.-A., Sagaspe, P., Sevin, E. D., Olive, J., Bioulac, S., & Sauteraud, A. (2017) · Scientific Reports · doi:10.1038/srep42656 · Cited by 128 · cited 1×
“Finally, Philip et al. contrasted the diagnostic performance of an embodied CA and a psychiatrist in relation to major depressive disorder among 179 outpatient participants [63].”
BackgroundVirtual training and coaching of health behavior: Example from mindfulness meditation training
Hudlicka, E. (2013) · Patient Education and Counseling · doi:10.1016/j.pec.2013.05.007 · Cited by 104 · cited 1×
“The multi-modal embodied system ‘Virtual Mindfulness Coach’, for example, was developed and evaluated for its efectiveness in training and coaching mindfulness meditation [35].”
BackgroundPyae, A., & Joelsson, T. N. (2018) · Proceedings of the 20th International Conference on Human-Computer Interaction with Mobile Devices and Services Adjunct · doi:10.1145/3236112.3236130 · Cited by 99
Social support agents for older adults: longitudinal affective computing in the home
Ring, L., Shi, L., Totzke, K., & Bickmore, T. (2015) · Journal on Multimodal User Interfaces · doi:10.1007/s12193-014-0157-0 · Cited by 97
Sebastian, J., & Richards, D. (2017) · Computers in Human Behavior · doi:10.1016/j.chb.2017.03.071 · Cited by 96 · cited 1×
“While others have begun to explore questions of user experience, trust, feedback, self-relection and learning during the collection of self-reports of mental and physical health and wellbeing [42, 69, 87] as well as social engagement [64, 72, 74].”
BackgroundOnce a Kind Friend is Now a Thing: Understanding How Conversational Agents at Home are Forgotten
Cho, M., Lee, S.-S., & Lee, K.-P. (2019) · Proceedings of the 2019 on Designing Interactive Systems Conference · doi:10.1145/3322276.3322332 · Cited by 90
Health and Fitness Apps for Hands-Free Voice-Activated Assistants: Content Analysis
Chung, A. E., Griffin, A. C., Selezneva, D., & Gotz, D. (2018) · JMIR mHealth and uHealth · doi:10.2196/mhealth.9705 · Cited by 70 · cited 1×
“While speech-enabled CAs are gaining widespread adoption in daily use [11, 18, 71], we cannot speak about the potential of CAs to support practices of self-report without mentioning the current limitations of these technologies, which are often signiicant in nature [16, 52, 55, 76, 77].”
BackgroundWhy doesn't it work?: voice-driven interfaces and young children's communication repair strategies
Cheng, Y., Yen, K., Chen, Y., Chen, S., & Hiniker, A. (2018) · Proceedings of the 17th ACM Conference on Interaction Design and Children · doi:10.1145/3202185.3202749 · Cited by 69 · cited 1×
“…that users themselves apply tactics including hyper-articulation and exaggeration, increased volume, use of diferent utterances or simpliied words, and reformulation strategies, such as addition or substitution, removal, and re-ordering of words to alleviate conversational barriers [15, 58, 62].”
BackgroundPersuasive Synthetic Speech: Voice Perception and User Behaviour
Dubiel, M., Halvey, M., Gallegos, P. O., & King, S. (2020) · Proceedings of the 2nd Conference on Conversational User Interfaces · doi:10.1145/3405755.3406120 · Cited by 28 · cited 1×
“Limitations to date identiied by researchers include misinterpretation and failed recognition of user utterances resulting in confusion and further errors [58, 68, 77], CAs’ present inability to engage in dynamic conversations [16], the need to specify in advance what users can or cannot say during interaction [20, 30…”
BackgroundAlexa depression and anxiety self-tests: a preliminary analysis of user experience and trust
Quiroz, J. C., Bongolan, T., & Ijaz, K. (2020) · Adjunct Proceedings of the 2020 ACM International Joint Conference on Pervasive and Ubiquitous Computing and Proceedings of the 2020 ACM International Symposium on Wearable Computers · doi:10.1145/3410530.3414374 · Cited by 16 · cited 1×
“In related work, Quiroz et al. developed an Alexa Skill enabling users to better express their emotions, complete self-assessments for depression and anxiety, and receive suggestions for improving their current state-of-mind [69].”
Background
Cited by
Exploring the Ethical Challenges of Conversational AI in Mental Health Care: Scoping Review
Rahsepar Meadi, M., Sillekens, T., Metselaar, S., van Balkom, A., Bernstein, J., & Batelaan, N. (2025) · JMIR Mental Health · doi:10.2196/60432 · Cited by 211
Moylan, K., & Doherty, K. (2025) · Journal of Medical Internet Research · doi:10.2196/67114 · Cited by 46
Self-citationCho, Y., Rai, S., Ungar, L., Sedoc, J., & Guntuku, S. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.698 · Cited by 33
Kuhail, M. A., Al-Shamaileh, O., Farooq, S., Shahin, H., Abdelzaher, F., & Thomas, J. (2025) · Human Behavior and Emerging Technologies · doi:10.1155/hbe2/9942295 · Cited by 11
Striegl, J., Rottmann, S., & Loitsch, C. (2024) · Lecture notes in networks and systems · doi:10.1007/978-3-031-66329-1_14 · Cited by 4
Metadata
- Authors
- Maharjan, R., Doherty, K., Rohani, D. A., Bækgaard, P., & Bardram, J. E.
- Venue
- ACM Transactions on Interactive Intelligent Systems
- Published
- 2022-06-30
- Volume
- 12
- Issue
- 2
- Pages
- 1-29
- ISSN
- 2160-6455 (ISSN-L)
- 2160-6463
- OpenAlex
- W4286273951
- Cited by
- 29
- References
- 43
- FWCI
- 5.04
- Type
- journal article
- Language
- en
- Open access
- bronze
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.