Validity of Chatbot Use for Mental Health Assessment: Experimental Study
Open accessThe DOI matches its Crossref record; 46 of the 46 references it cites were found in a registry or an index.
Abstract
Background: Mental disorders in adolescence and young adulthood are major public health concerns. Digital tools such as text-based conversational agents (ie, chatbots) are a promising technology for facilitating mental health assessment. However, the human-like interaction style of chatbots may induce potential biases, such as socially desirable responding (SDR), and may require further effort to complete assessments. Objective: This study aimed to investigate the convergent and discriminant validity of chatbots for mental health assessments, the effect of assessment mode on SDR, and the effort required by participants for assessments using chatbots compared with established modes. Methods: In a counterbalanced within-subject design, we assessed 2 different constructs—psychological distress (Kessler Psychological Distress Scale and Brief Symptom Inventory-18) and problematic alcohol use (Alcohol Use Disorders Identification Test-3)—in 3 modes (chatbot, paper-and-pencil, and web-based), and examined convergent and discriminant validity. In addition, we investigated the effect of mode on SDR, controlling for perceived sensitivity of items and individuals’ tendency to respond in a socially desirable way, and we also assessed the perceived social presence of modes. Including a between-subject condition, we further investigated whether SDR is increased in chatbot assessments when applied in a self-report setting versus when human interaction may be expected. Finally, the effort (ie, complexity, difficulty, burden, and time) required to complete the assessments was investigated. Results: A total of 146 young adults (mean age 24, SD 6.42 years; n=67, 45.9% female) were recruited from a research panel for laboratory experiments. The results revealed high positive correlations (all P<.001) of measures of the same construct across different modes, indicating the convergent validity of chatbot assessments. Furthermore, there were no correlations between the distinct constructs, indicating discriminant validity. Moreover, there were no differences in SDR between modes and whether human interaction was expected, although the perceived social presence of the chatbot mode was higher than that of the established modes (P<.001). Finally, greater effort (all P<.05) and more time were needed to complete chatbot assessments than for completing the established modes (P<.001). Conclusions: Our findings suggest that chatbots may yield valid results. Furthermore, an understanding of chatbot design trade-offs in terms of potential strengths (ie, increased social presence) and limitations (ie, increased effort) when assessing mental health were established.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Faul, F., Erdfelder, E., Lang, A.-G., & Buchner, A. (2007) · Behavior Research Methods · doi:10.3758/bf03193146 · Cited by 65,971
SAUNDERS, J. B., AASLAND, O. G., BABOR, T. F., DE LA FUENTE, J. R., & GRANT, M. (1993) · Addiction · doi:10.1111/j.1360-0443.1993.tb02093.x · Cited by 13,597 · cited 1×
“We assessed alcohol use by applying the Alcohol Use Disorders Identification Test (AUDIT)–3 questionnaire [59,60], which has been shown to perform similarly well as the AUDIT-10 in detecting risky drinking behavior [60].”
MethodThe Psychology of Survey Response
Tourangeau, R., Rips, L. J., & Rasinski, K. (2000) · doi:10.1017/cbo9780511819322 · Cited by 4,734
Machines and Mindlessness: Social Responses to Computers
Nass, C., & Moon, Y. (2000) · Journal of Social Issues · doi:10.1111/0022-4537.00153 · Cited by 4,022 · cited 1×
“This is particularly relevant, as there is evidence that individuals preconsciously attribute human characteristics to chatbots because of increased perceived social presence [27-30].”
BackgroundMental health of young people: A global public-health challenge
Patel, V., Flisher, A. J., Hetrick, S., & McGorry, P. (2007) · The Lancet · doi:10.1016/s0140-6736(07)60368-7 · Cited by 3,126 · cited 1×
“Thus, mental health in young people is a major public health concern [2].”
BackgroundSensitive questions in surveys.
Tourangeau, R., & Yan, T. (2007) · Psychological Bulletin · doi:10.1037/0033-2909.133.5.859 · Cited by 3,057 · cited 2×
“Previous evidence indicates that SDR may increase when individuals expect their responses to be immediately reviewed and evaluated by a researcher [33,43,44].”
BackgroundFitzpatrick, K. K., Darcy, A., & Vierhile, M. (2017) · JMIR Mental Health · doi:10.2196/mental.7785 · Cited by 2,730 · cited 3×
“Text-based conversational agents (ie, chatbots) are a promising digital technology in this context [6-12].”
BackgroundMethodMethods of coping with social desirability bias: A review
Nederhof, A. J. (1985) · European Journal of Social Psychology · doi:10.1002/ejsp.2420150303 · Cited by 2,340 · cited 1×
“Although an increase in perceived social presence in face-to-face interviews has been found to increase response biases [32-35], self-reported assessments associated with reduced social presence have demonstrated reliability and validity compared with, for example, face-to-face assessments [36-40].”
BackgroundGlobal burden of disease in young people aged 10–24 years: a systematic analysis
Gore, F. M., Bloem, P. J., Patton, G. C., Ferguson, J., Joseph, V., Coffey, C., Sawyer, S. M., & Mathers, C. D. (2011) · The Lancet · doi:10.1016/s0140-6736(11)60512-6 · Cited by 2,301 · cited 1×
“Mental disorders are a leading cause of disease burden in high-income countries and first emerge in adolescence and young adulthood [1].”
BackgroundInformation richness. A new approach to managerial behavior and organization design
Daft, R. L., & Lengel, R. H. (1983) · doi:10.21236/ada128980 · Cited by 2,263
Nass, C., Steuer, J., & Tauber, E. R. (1994) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/191666.191703 · Cited by 2,148 · cited 1×
“If chatbots are perceived as human actors [42,45], this may lead individuals to believe that their responses are immediately reviewed and evaluated.”
Conversational agents in healthcare: a systematic review
Laranjo, L., Dunn, A. G., Tong, H. L., Kocaballi, A. B., Chen, J., Bashir, R., Surian, D., Gallego, B., Magrabi, F., Lau, A. Y. S., et al. (2018) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocy072 · Cited by 1,388 · cited 2×
“Irrespective of the popularity of chatbots, reviews of their application in the context of (mental) health emphasize the quasi-experimental nature of studies and the need to empirically evaluate their impact [7,16,23-26].”
BackgroundEstablishing and maintaining long-term human-computer relationships
Bickmore, T. W., & Picard, R. W. (2005) · ACM Transactions on Computer-Human Interaction · doi:10.1145/1067860.1067867 · Cited by 1,333 · cited 3×
“For example, it has been shown that users may develop a strong social relationship with a chatbot when it expresses empathetic support [21,51,85,87-90].”
BackgroundLuger, E., & Sellen, A. (2016) · Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems · doi:10.1145/2858036.2858288 · Cited by 1,176
Go, E., & Sundar, S. S. (2019) · Computers in Human Behavior · doi:10.1016/j.chb.2019.01.020 · Cited by 1,089 · cited 3×
“The chatbot incorporated the following social cues to further increase perceived social presence [28,30]: an anthropomorphic icon [50], the capability to engage in small talk [51], a dynamically calculated response delay based on the length of the response [30], and a typing indicator (3 moving dots indicating that a…”
BackgroundInkster, B., Sarda, S., & Subramanian, V. (2018) · doi:10.2196/preprints.12106 · Cited by 1,063
Chatbots and Conversational Agents in Mental Health: A Review of the Psychiatric Landscape
Vaidyam, A. N., Wisniewski, H., Halamka, J. D., Kashavan, M. S., & Torous, J. B. (2019) · The Canadian Journal of Psychiatry · doi:10.1177/0706743719828977 · Cited by 1,048 · cited 3×
“Irrespective of the popularity of chatbots, reviews of their application in the context of (mental) health emphasize the quasi-experimental nature of studies and the need to empirically evaluate their impact [7,16,23-26].”
BackgroundFrom Eliza to XiaoIce: Challenges and opportunities with social chatbots
Shum, H.-Y., He, X.-D., & Li, D. (2018) · Frontiers of Information Technology & Electronic Engineering · doi:10.1631/fitee.1700826 · Cited by 749 · cited 1×
“For example, it has been shown that users may develop a strong social relationship with a chatbot when it expresses empathetic support [21,51,85,87-90].”
BackgroundHo, A., Hancock, J., & Miner, A. S. (2018) · Journal of Communication · doi:10.1093/joc/jqy026 · Cited by 629 · cited 1×
“As the assessment modes between conditions did not differ, the results show that the application of a chatbot results in valid responses, regardless of whether the data are immediately reviewed and evaluated by a human actor [70,83].”
BackgroundAlmost human: Anthropomorphism increases trust resilience in cognitive agents.
de Visser, E. J., Monfort, S. S., McKendrick, R., Smith, M. A. B., McKnight, P. E., Krueger, F., & Parasuraman, R. (2016) · Journal of Experimental Psychology: Applied · doi:10.1037/xap0000092 · Cited by 614
McTear, M., Callejas, Z., & Griol, D. (2016) · doi:10.1007/978-3-319-32967-3 · Cited by 594
User Experiences of Social Support From Companion Chatbots in Everyday Contexts: Thematic Analysis
Ta, V., Griffith, C., Boatfield, C., Wang, X., Civitello, M., Bader, H., DeCero, E., & Loggarakis, A. (2020) · Journal of Medical Internet Research · doi:10.2196/16235 · Cited by 561
Abd-Alrazaq, A. A., Rababeh, A., Alajlani, M., Bewick, B. M., & Househ, M. (2020) · Journal of Medical Internet Research · doi:10.2196/16021 · Cited by 503 · cited 4×
“The use of chatbots for mental health assessment is an emerging field, and robust investigations of their positive and potential negative effects are required [16].”
BackgroundInfluentialDisclosure of sensitive behaviors across self-administered survey modes: a meta-analysis
Gnambs, T., & Kaspar, K. (2015) · Behavior Research Methods · doi:10.3758/s13428-014-0533-4 · Cited by 422 · cited 1×
“Although an increase in perceived social presence in face-to-face interviews has been found to increase response biases [32-35], self-reported assessments associated with reduced social presence have demonstrated reliability and validity compared with, for example, face-to-face assessments [36-40].”
BackgroundSlade, T., Grove, R., & Burgess, P. (2011) · Australian & New Zealand Journal of Psychiatry · doi:10.3109/00048674.2010.543653 · Cited by 372
Ly, K. H., Ly, A.-M., & Andersson, G. (2017) · Internet Interventions · doi:10.1016/j.invent.2017.10.002 · Cited by 368
"I Hear You, I Feel You": Encouraging Deep Self-disclosure through a Chatbot
Lee, Y.-C., Yamashita, N., Huang, Y., & Fu, W. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376175 · Cited by 339 · cited 1×
“A promising approach may be to enable reciprocal self-disclosure, in which the chatbot reveals sensitive information, as this has been shown to result in a reciprocal effect on promoting individuals’ self-disclosure [70], as well as perceived intimacy and enjoyment [74].”
BackgroundChatbots in the fight against the COVID-19 pandemic
Miner, A. S., Laranjo, L., & Kocaballi, A. B. (2020) · npj Digital Medicine · doi:10.1038/s41746-020-0280-0 · Cited by 322
Conversational agents in the treatment of mental health problems: mixed-method systematic review
Gaffney, H., Mansell, W., & Tai, S. (2019) · JMIR Mental Health · doi:10.2196/14166 · Cited by 275
Morris, R. R., Kouddous, K., Kshirsagar, R., & Schueller, S. M. (2018) · Journal of Medical Internet Research · doi:10.2196/10148 · Cited by 273
Kretzschmar, K., Tyroll, H., Pavarini, G., Manzini, A., Singh, I., & NeurOx Young People’s Advisory Group (2019) · Biomedical Informatics Insights · doi:10.1177/1178222619829083 · Cited by 270
Greer, S., Ramo, D., Chang, Y.-J., Fu, M., Moskowitz, J., & Haritatos, J. (2019) · JMIR mHealth and uHealth · doi:10.2196/15018 · Cited by 265 · cited 1×
“Text-based conversational agents (ie, chatbots) are a promising digital technology in this context [6-12].”
BackgroundEffectiveness of an Empathic Chatbot in Combating Adverse Effects of Social Exclusion on Mood
de Gennaro, M., Krumhuber, E. G., & Lucas, G. (2020) · Frontiers in Psychology · doi:10.3389/fpsyg.2019.03061 · Cited by 265
Hoermann, S., McCabe, K. L., Milne, D. N., & Calvo, R. A. (2017) · Journal of Medical Internet Research · doi:10.2196/jmir.7023 · Cited by 263 · cited 4×
“Text-based conversational agents (ie, chatbots) are a promising digital technology in this context [6-12].”
BackgroundMethodInfluentialChaix, B., Bibault, J.-E., Pienkowski, A., Delamon, G., Guillemassé, A., Nectoux, P., & Brouard, B. (2019) · JMIR Cancer · doi:10.2196/12856 · Cited by 227
Bendig, E., Erb, B., Schulze-Thuesing, L., & Baumeister, H. (2022) · Verhaltenstherapie · doi:10.1159/000501812 · Cited by 214 · cited 2×
“Irrespective of the popularity of chatbots, reviews of their application in the context of (mental) health emphasize the quasi-experimental nature of studies and the need to empirically evaluate their impact [7,16,23-26].”
BackgroundArtificial Intelligence-Assisted Online Social Therapy for Youth Mental Health
D'Alfonso, S., Santesteban-Echarri, O., Rice, S., Wadley, G., Lederman, R., Miles, C., Gleeson, J., & Alvarez-Jimenez, M. (2017) · Frontiers in Psychology · doi:10.3389/fpsyg.2017.00796 · Cited by 192 · cited 1×
“However, psychological help remains difficult to access [3].”
BackgroundSchuetzler, R. M., Giboney, J. S., Grimes, G. M., & Nunamaker, J. F. (2018) · Decision Support Systems · doi:10.1016/j.dss.2018.08.011 · Cited by 154 · cited 6×
“If chatbots are perceived as human actors [42,45], this may lead individuals to believe that their responses are immediately reviewed and evaluated.”
BackgroundResultInfluentialBickmore, T. W., Utami, D., Matsuyama, R., & Paasche-Orlow, M. K. (2016) · Journal of Medical Internet Research · doi:10.2196/jmir.5239 · Cited by 153
Bérubé, C., Schachner, T., Keller, R., Fleisch, E., v Wangenheim, F., Barata, F., & Kowatsch, T. (2021) · Journal of Medical Internet Research · doi:10.2196/25933 · Cited by 133 · cited 1×
“However, more research on the design of voice-based conversational agents for mental health assessment is required [77].”
BackgroundJungmann, S. M., Klan, T., Kuhn, S., & Jungmann, F. (2019) · JMIR Formative Research · doi:10.2196/13863 · Cited by 127
Oh, J., Jang, S., Kim, H., & Kim, J.-J. (2020) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2020.104171 · Cited by 124 · cited 1×
“In addition, chatbots have been investigated in several studies and applied to assess or monitor mental health [16], deliver information for improving mental health literacy [9,14,15,17], and assist and compound therapy sessions as guided or blended care [18-22].”
BackgroundRelational Agents in Clinical Psychiatry
Bickmore, T., & Gruber, A. (2010) · Harvard Review of Psychiatry · doi:10.3109/10673221003707538 · Cited by 112 · cited 1×
“In addition, chatbots have been investigated in several studies and applied to assess or monitor mental health [16], deliver information for improving mental health literacy [9,14,15,17], and assist and compound therapy sessions as guided or blended care [18-22].”
BackgroundScholten, M. R., Kelders, S. M., & Van Gemert-Pijnen, J. E. (2017) · Journal of Medical Internet Research · doi:10.2196/jmir.7351 · Cited by 97 · cited 3×
“For example, it has been shown that users may develop a strong social relationship with a chatbot when it expresses empathetic support [21,51,85,87-90].”
BackgroundMartínez-Miranda, J. (2017) · Journal of Medical Systems · doi:10.1007/s10916-017-0784-6 · Cited by 51
Zand, A., Sharma, A., Stokes, Z., Reynolds, C., Montilla, A., Sauk, J., & Hommes, D. (2020) · Journal of Medical Internet Research · doi:10.2196/15589 · Cited by 48
Cited by
Chakraborty, C., Bhattacharya, M., Pal, S., & Lee, S.-S. (2024) · Current Research in Biotechnology · doi:10.1016/j.crbiot.2023.100164 · Cited by 274
D'Amico, R. S., White, T. G., Shah, H. A., & Langer, D. J. (2023) · Neurosurgery · doi:10.1227/neu.0000000000002414 · Cited by 118
Hua, Y., Siddals, S., Ma, Z., Galatzer‐Levy, I., Xia, W., Hau, C., Na, H., Flathers, M., Linardon, J., Ayubcha, C., et al. (2025) · World Psychiatry · doi:10.1002/wps.21352 · Cited by 117
Cho, Y., Rai, S., Ungar, L., Sedoc, J., & Guntuku, S. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.698 · Cited by 33
Kuhail, M. A., Al-Shamaileh, O., Farooq, S., Shahin, H., Abdelzaher, F., & Thomas, J. (2025) · Human Behavior and Emerging Technologies · doi:10.1155/hbe2/9942295 · Cited by 11
Metadata
- Authors
- Schick, A., Feine, J., Morana, S., Maedche, A., & Reininghaus, U.
- Venue
- JMIR mHealth and uHealth
- Publisher
- JMIR Publications Inc.
- Published
- 2022-10-31
- Volume
- 10
- Issue
- 10
- Pages
- e28082
- ISSN
- 2291-5222 (ISSN-L)
- OpenAlex
- W4307877758
- Cited by
- 58
- References
- 46
- FWCI
- 9.08
- Type
- journal article
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.