Validity of Chatbot Use for Mental Health Assessment: Experimental Study
Open accessThe DOI matches its Crossref record; 46 of the 46 references it cites were found in a registry or an index.
Abstract
Background: Mental disorders in adolescence and young adulthood are major public health concerns. Digital tools such as text-based conversational agents (ie, chatbots) are a promising technology for facilitating mental health assessment. However, the human-like interaction style of chatbots may induce potential biases, such as socially desirable responding (SDR), and may require further effort to complete assessments. Objective: This study aimed to investigate the convergent and discriminant validity of chatbots for mental health assessments, the effect of assessment mode on SDR, and the effort required by participants for assessments using chatbots compared with established modes. Methods: In a counterbalanced within-subject design, we assessed 2 different constructs—psychological distress (Kessler Psychological Distress Scale and Brief Symptom Inventory-18) and problematic alcohol use (Alcohol Use Disorders Identification Test-3)—in 3 modes (chatbot, paper-and-pencil, and web-based), and examined convergent and discriminant validity. In addition, we investigated the effect of mode on SDR, controlling for perceived sensitivity of items and individuals’ tendency to respond in a socially desirable way, and we also assessed the perceived social presence of modes. Including a between-subject condition, we further investigated whether SDR is increased in chatbot assessments when applied in a self-report setting versus when human interaction may be expected. Finally, the effort (ie, complexity, difficulty, burden, and time) required to complete the assessments was investigated. Results: A total of 146 young adults (mean age 24, SD 6.42 years; n=67, 45.9% female) were recruited from a research panel for laboratory experiments. The results revealed high positive correlations (all P<.001) of measures of the same construct across different modes, indicating the convergent validity of chatbot assessments. Furthermore, there were no correlations between the distinct constructs, indicating discriminant validity. Moreover, there were no differences in SDR between modes and whether human interaction was expected, although the perceived social presence of the chatbot mode was higher than that of the established modes (P<.001). Finally, greater effort (all P<.05) and more time were needed to complete chatbot assessments than for completing the established modes (P<.001). Conclusions: Our findings suggest that chatbots may yield valid results. Furthermore, an understanding of chatbot design trade-offs in terms of potential strengths (ie, increased social presence) and limitations (ie, increased effort) when assessing mental health were established.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Faul, F., Erdfelder, E., Lang, A.-G., & Buchner, A. (2007) · Behavior Research Methods · doi:10.3758/bf03193146 · Cited by 65,971
SAUNDERS, J. B., AASLAND, O. G., BABOR, T. F., DE LA FUENTE, J. R., & GRANT, M. (1993) · Addiction · doi:10.1111/j.1360-0443.1993.tb02093.x · Cited by 13,597 · cited 1×
“We assessed alcohol use by applying the Alcohol Use Disorders Identification Test (AUDIT)–3 questionnaire [59,60], which has been shown to perform similarly well as the AUDIT-10 in detecting risky drinking behavior [60].”
MethodThe Psychology of Survey Response
Tourangeau, R., Rips, L. J., & Rasinski, K. (2000) · doi:10.1017/cbo9780511819322 · Cited by 4,734
Machines and Mindlessness: Social Responses to Computers
Nass, C., & Moon, Y. (2000) · Journal of Social Issues · doi:10.1111/0022-4537.00153 · Cited by 4,022 · cited 1×
“This is particularly relevant, as there is evidence that individuals preconsciously attribute human characteristics to chatbots because of increased perceived social presence [27-30].”
BackgroundMental health of young people: A global public-health challenge
Patel, V., Flisher, A. J., Hetrick, S., & McGorry, P. (2007) · The Lancet · doi:10.1016/s0140-6736(07)60368-7 · Cited by 3,126 · cited 1×
“Thus, mental health in young people is a major public health concern [2].”
BackgroundSensitive questions in surveys.
Tourangeau, R., & Yan, T. (2007) · Psychological Bulletin · doi:10.1037/0033-2909.133.5.859 · Cited by 3,057 · cited 2×
“Previous evidence indicates that SDR may increase when individuals expect their responses to be immediately reviewed and evaluated by a researcher [33,43,44].”
BackgroundFitzpatrick, K. K., Darcy, A., & Vierhile, M. (2017) · JMIR Mental Health · doi:10.2196/mental.7785 · Cited by 2,730 · cited 3×
“Text-based conversational agents (ie, chatbots) are a promising digital technology in this context [6-12].”
BackgroundMethodMethods of coping with social desirability bias: A review
Nederhof, A. J. (1985) · European Journal of Social Psychology · doi:10.1002/ejsp.2420150303 · Cited by 2,340 · cited 1×
“Although an increase in perceived social presence in face-to-face interviews has been found to increase response biases [32-35], self-reported assessments associated with reduced social presence have demonstrated reliability and validity compared with, for example, face-to-face assessments [36-40].”
BackgroundGlobal burden of disease in young people aged 10–24 years: a systematic analysis
Gore, F. M., Bloem, P. J., Patton, G. C., Ferguson, J., Joseph, V., Coffey, C., Sawyer, S. M., & Mathers, C. D. (2011) · The Lancet · doi:10.1016/s0140-6736(11)60512-6 · Cited by 2,301 · cited 1×
“Mental disorders are a leading cause of disease burden in high-income countries and first emerge in adolescence and young adulthood [1].”
BackgroundInformation richness. A new approach to managerial behavior and organization design
Daft, R. L., & Lengel, R. H. (1983) · doi:10.21236/ada128980 · Cited by 2,263
Cited by
Chakraborty, C., Bhattacharya, M., Pal, S., & Lee, S.-S. (2024) · Current Research in Biotechnology · doi:10.1016/j.crbiot.2023.100164 · Cited by 274
D'Amico, R. S., White, T. G., Shah, H. A., & Langer, D. J. (2023) · Neurosurgery · doi:10.1227/neu.0000000000002414 · Cited by 118
Hua, Y., Siddals, S., Ma, Z., Galatzer‐Levy, I., Xia, W., Hau, C., Na, H., Flathers, M., Linardon, J., Ayubcha, C., et al. (2025) · World Psychiatry · doi:10.1002/wps.21352 · Cited by 117
Cho, Y., Rai, S., Ungar, L., Sedoc, J., & Guntuku, S. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.698 · Cited by 33
Kuhail, M. A., Al-Shamaileh, O., Farooq, S., Shahin, H., Abdelzaher, F., & Thomas, J. (2025) · Human Behavior and Emerging Technologies · doi:10.1155/hbe2/9942295 · Cited by 11
Unlocking AI Chatbot Potential in Healthcare: Trust-Enhanced DeLone & McLean IS Success Model
Sarhan, M. Y., Alarify, M., & Khojah, M. (2026) · Healthcare · doi:10.3390/healthcare14101324 · Cited by 2
“…vaccine hesitancy [16–19] improving diabetic patient education [20,21], providing emotional support for patient [22,23], assisting in remote mental health monitoring [24–26], aiding tobacco cessation [27–30], enhancing drug compliance, and alleviating symptoms of depression and anxiety [7,31,32].”
Metadata
- Authors
- Schick, A., Feine, J., Morana, S., Maedche, A., & Reininghaus, U.
- Venue
- JMIR mHealth and uHealth
- Publisher
- JMIR Publications Inc.
- Published
- 2022-10-31
- Volume
- 10
- Issue
- 10
- Pages
- e28082
- ISSN
- 2291-5222 (ISSN-L)
- OpenAlex
- W4307877758
- Cited by
- 58
- References
- 46
- FWCI
- 9.08
- Type
- journal article
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.