CoChat
Sign Up
Cited by 6Digital Mental Health InterventionsSocial Robot Interaction and HRIEating Disorders and Behaviors

A Field Test of Popular Chatbots’ Responses To Questions Concerning Negative Body Image

Open access
An, Ruopeng, Byron, Christopher W, Chen, Chen, Xiang, Xiaoling·2023·Health Behavior Research
DOI 10.4148/2572-1836.1164
Open this paper in CoChat — free accountOpen PDF

Abstract

Background: Chatbots are computer programs, often built upon large artificial intelligence models, that employ dialogue systems to enable online, natural language conversations with users via text, speech, or both. Body image, broadly defined as a combination of thoughts and feelings about one’s physical appearance, has been implicated in many risk behaviors and health problems, especially among adolescents and young adults. Little is known about how chatbots respond to questions about body image. Methods: This study assessed the responses of 14 widely-used chatbots (eight companion and six therapeutic chatbots) to ten body image-related questions developed upon validated instruments. Chatbots’ responses were documented, with qualities systematically assessed by nine pre-determined criteria. Results: The overall quality of the chatbots’ responses was modest (an average score of five out of nine), with substantial variations in the content and quality of responses across chatbots (individual scores ranging from one to eight). Companion and therapeutic chatbots systematically differed in their responses (e.g., focusing on comforting users vs. trying to identify the causes of negative body image and recommending potential remedies). Some therapeutic chatbots recognized potential mental health crises (self-harm) in test users’ messages. Conclusion: Substantial heterogeneities in the responses were present across chatbots and assessment criteria. Adolescents and young adults struggling with body image could be vulnerable to misleading or biased remarks made by chatbots. Still, the technical and supervision challenges to prevent those adverse consequences remain paramount and unsolved.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.4148/2572-1836.1164

Cites

12 of 12 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Across 14 widely used LLM-CBs, the quality of responses to mental health queries ranges widely, indicating a lack of consistency.

    “Furthermore, across 14 widely used LLM-CBs, the quality of responses to mental health queries ranges widely, indicating a lack of consistency70.”

    From A scoping review on the mental health harms of LLM-based chatbots (2026) · cites this paper as [70]

    The Discussion reports substantial variation in response quality across 14 popular companion and therapeutic chatbots. The study tested body image-related questions, however, so it does not establish the same pattern for mental health queries generally or identify all 14 as LLM-based chatbots. (LLM, conf=0.84)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 40

  1. Health Behavior Research, Vol. 6, No. 1 [2023], Art. 3 https://newprairiepress.org/hbr/vol6/iss1/3 DOI: 10.4148/2572-1836.1164 Health Behavior Research, Vol. 6, No. 1 [2023], Art. 3 https://newprairiepress.org/hbr/vol6/iss1/3 DOI: 10.4148/2572-1836.1164 Health Behavior Research, Vol. 6, No. 1 [2023], Art. 3 https://newprairiepress.org/hbr/vol6/iss1/3 DOI: 10.4148/2572-1836.1164 Health Behavior Research, Vol. 6, No. 1 [2023], Art. 3 https://newprairiepress.org/hbr/vol6/iss1/3 DOI: 10.4148/2572-1836.1164 Health Behavior Research, Vol. 6, No. 1 [2023], Art. 3 https://newprairiepress.org/hbr/vol6/iss1/3 DOI: 10.4148/2572-1836.1164 Health Behavior Research, Vol. 6, No. 1 [2023], Art. 3 https://newprairiepress.org/hbr/vol6/iss1/3 DOI: 10.4148/2572-1836.1164 Health Behavior Research, Vol. 6, No. 1 [2023], Art. 3 https://newprairiepress.org/hbr/vol6/iss1/3 DOI: 10.4148/2572-1836.1164 Health Behavior Research, Vol. 6, No. 1 [2023], Art. 3 https://newprairiepress.org/hbr/vol6/iss1/3 DOI: 10.4148/2572-1836.1164 References · DOI
  2. Anantrasirichai, N., & Bull, D. (2021). Artificial intelligence in the creative industries: A review. Artificial Intelligence Review, 55, 589-656. https://doi.org/10.1007/s10462-021- 10039-7 · DOI · Paper page
  3. Baclic, O., Tunis, M., Young, K., Doan, C., Swerdfeger, H., & Schonfeld, J. (2020). Challenges and opportunities for public health made possible by advances in natural language processing. Canada Communicable Disease Report, 46(6), 161-168. https://doi.org/10.14745/ccdr.v46i06a02 · DOI
  4. Bandura A. (2004). Health promotion by social cognitive means. Health Education & Behavior, 31(2), 143-164. https://doi.org/10.1177/10901981042636 · DOI · Paper page
  5. Beilharz, F., Sukunesan, S., Rossell, S. L., Kulkarni, J., & Sharp, G. Development of a positive body image chatbot (KIT) with young people and parents/carers: Qualitative focus group study. (2021). Journal of Internet Medical Research, 23(6). https://doi.org/10.2196/27807 · DOI
  6. Bishop, J. M. (2021). Artificial intelligence is stupid and causal reasoning will not fix it. Frontiers in Psychology, 11(513474). https://doi.org/10.3389/fpsyg.2020.5134 · DOI
  7. Bucchianeri, M. M., Arikian, A. J., Hannan, P. J., Eisenberg, M. E., & Neumark- Sztainer, D. (2013). Body dissatisfaction from adolescence to young adulthood: Findings from a 10-year longitudinal study. Body Image, 10(1), 1-7. https://doi.org/10.1016/j.bodyim.2012.0 9.001 · DOI · Paper page
  8. Caldarini, G., Jaf, S., & McGarry, K. (2022). A literature survey of recent advances in chatbots. Information, 1, 06657. https://doi.org/10.48550/arXiv.2201.066 · DOI
  9. Celi, L. A., Cellini, J., Charpignon, M. L., Dee, E. C., Dernoncourt, F., Eber, R., Mitchell, W. G., Moukheiber, L., Schirmer, J., Situ, J., Paguio, J., Park, J., Wawira, J. G., & Yao S. (2022) Sources of bias in artificial intelligence that perpetuate healthcare disparities -a global review. PLoS Digit Health, 1(3), e0000022. https://doi.org/10.1371/journal.pdig.000 0022 · DOI · Paper page
  10. D'Alfonso, S., Santesteban-Echarri, O., Rice, S., Wadley, G., Lederman, R., Miles C., Gleeson, J., & Alvarez-Jimenez, M. (2017). Artificial intelligence-assisted online social therapy for youth mental health. Frontiers in Psychology, 8(796). https://doi.org/10.3389/fpsyg.2017.0079 · DOI · Paper page
  11. Davis, K. (1995). Reshaping the female body: The dilemma of cosmetic surgery. Routledge. https://doi.org/10.4324/9780203700129 · DOI
  12. Devlin, J., Chang, M. W., Lee, K., & Toutanova, K. (2018). Bert: Pre-training of deep bidirectional transformers for language understanding. Cornell University. https://doi.org/10.48550/arXiv.1810.048 · DOI · Paper page
  13. Dohnt, H., & Tiggemann, M. (2006). The contribution of peer and media influences to the development of body satisfaction and self-esteem in young girls: A prospective study. Developmental Psychology, 42(5), 929- 936. https://doi.org/10.1037/0012- 1649.42.5.929 · DOI
  14. Ferrand, J., Hockensmith, R., Houghton, R. F., & Walsh-Buhi, E. R. (2020). Evaluating smart assistant responses for accuracy and misinformation regarding human papillomavirus vaccination: Content analysis study. Journal of Medical Internet Research, 22(8), e19018. https://www.jmir.org/2020/8/e19018 26 Health Behavior Research, Vol. 6, No. 1 [2023], Art. 3 https://newprairiepress.org/hbr/vol6/iss1/3 DOI: 10.4148/2572-1836.1164
  15. Goh, Y. S., Ow Yong, J., Chee, B., Kuek, J., & Ho, C. (2022). Machine learning in health promotion and behavioral change: Scoping review. Journal of Medical Internet Research, 24(6), e35831. https://doi.org/10.2196/35831 · DOI
  16. Grove, C. (2021) Co-developing a mental health and wellbeing chatbot with and for young people. Frontiers in Psychiatry, 11. https://doi.org/10.3389/fpsyt.2020.60604 · DOI
  17. Han, X., Zhang, Z., Ding, N., Gu, Y., Liu, X., Huo, Y., Qiu, J., Yao, Y., Zhang, A., Zhang, L., Han, W., Huang, M., Jin, Q., Lan, Y., Liu, Y., Liu, Z., Lu, Z., Qiu, X., Song, R., … Zhu, J. (2021). Pre-trained models: Past, present and future. AI Open, 2, 225-250. https://doi.org/10.1016/j.aiopen.2021.08. 002 · DOI · Paper page
  18. Hughes, R. T., Zhu, L., & Bednarz, T. (2021). Generative adversarial networks-enabled human-artificial intelligence collaborative applications for creative and design industries: a systematic review of current approaches and trends. Frontiers in Artificial Intelligence, 4, 60234. https://doi.org/10.3389/frai.2021.604234 · DOI · Paper page
  19. King, T. K., Matacin, M., White, K. S., & Marcus, B. H. (2005). A prospective examination of body image and smoking cessation in women. Body Image, 2(1), 19-28. https://psycnet.apa.org/doi/10.1016/j.bod yim.2005.01.003 · DOI
  20. Kipela, L. S., Becker, C. B., Wesley, N., & Stewart, T. (2015). Body image in adult women: Moving beyond the younger years. Advances in Eating Disorders, 3(2), 144-164. https://dx.doi.org/10.1080%2F21662630 .2015.1012728 · DOI
  21. Kochling, A., & Wehner, M. C. (2020). Discriminated by an algorithm: A systematic review of discrimination and fairness by algorithmic decision-making in the context of HR recruitment and HR development. Business Research, 13, 795-848. https://doi.org/10.1007/s40685- 020-00134-w · DOI · Paper page
  22. Levine, M. P., & Piran, N. (2004). The role of body image in the prevention of eating disorders. Body Image, 1(1), 57-70. https://doi.org/10.1016/s1740- 1445(03)00006-8 · DOI
  23. Liggett, G., Grogan, S., & Burwitz, L. (2003). The effectiveness of a six-week 16 aerobic dance intervention on body image dissatisfaction among adolescent females [Conference Session]. British Psychological Society, Division of Health 18 Psychology Conference, Stafford, United Kingdom.
  24. Mercenier, H., Wiard, V., & Dufrasne, M. (2021). Teens, social media, and fake news. In G. López-García, D. Palau- Sampio, B. Palomo, E. Campos- Domínguez, & P. Masip, (Eds.), Politics of Disinformation: The Influence of Fake News on the Public Sphere. Wiley Online Library. https://doi.org/10.1002/9781119743347. · DOI
  25. ch12 · DOI
  26. Mierzwa, S., Souidi, S., Conroy, T., Abusyed, M., Watari, H., & Allen, T. (2019). On the potential, feasibility, and effectiveness of chat bots in public health research going forward. Online Journal of Public Health Informatics, 11(2). https://dx.doi.org/10.5210%2Fojphi.v11i 2.9998 · DOI
  27. Oh, Y. J., Zhang, J., Fang, M. L., & Fukuoka, Y. (2021). A systematic review of artificial intelligence chatbots for promoting physical activity, healthy diet, and weight loss. International Journal of Behavioral Nutrition and Physical Activity, 18(160). https://doi.org/10.1186/s12966-021- 01224-6 · DOI · Paper page
  28. Panch, T., Pearson-Stuttard, J., Greaves, F., & Atun, R. (2019). Artificial intelligence: Opportunities and risks for public health. The Lancet. Digital Health, 1(1), e13- e14. https://doi.org/10.1016/S2589- 7500(19)30002-0 · DOI
  29. Panch, T., Szolovits, P., & Atun, R. Artificial intelligence, machine learning and health systems. (2018). Journal of Global Health, 8(2). https://dx.doi.org/10.7189%2Fjogh.08.0 20303 28 · DOI
  30. Health Behavior Research, Vol. 6, No. 1 [2023], Art. 3 https://newprairiepress.org/hbr/vol6/iss1/3 DOI: 10.4148/2572-1836.1164
  31. Papapicco, C., Lamanna, I., & D'Errico, F. (2022). Adolescents' vulnerability to fake news and to racial hoaxes: A qualitative analysis on Italian sample. Multimodal Technologies and Interaction, 6(3), 20, https://doi.org/10.3390/mti6030020 · DOI
  32. Park, S. M., & Kim, A. Y. G. (2022). A metaverse: Taxonomy, components, applications, and open challenges. IEEE Access, 10. https://doi.org/10.1109/ACCESS.2021.3 140175 · DOI
  33. Quittkat, H. L., Hartmann, A. S., Dusing, R., Buhlmann, U., & Vocks, S. (2019). Body dissatisfaction, importance of appearance, and body appreciation in men and women over the lifespan. Frontiers in Psychiatry, 10(864). https://dx.doi.org/10.3389%2Ffpsyt.201 9.00864 · DOI
  34. Scheibmeir, J. A., & Malaiya, Y. K. (2021). Social media analytics of the Internet of Things. Discover Internet of Things, 1(16). https://doi.org/10.1007/s43926- 021-00016-5 · DOI
  35. Tort-Nasarre, G., Pocallet, M. P., & Artigues- Barbera, E. (2021). The meaning and factors that influence the concept of body image: Systematic review and meta- ethnography from the perspectives of adolescents. International Journal of Environmental Research and Public Health, 18(1140). https://dx.doi.org/10.3390%2Fijerph180 31140 · DOI
  36. Voelker, D. K., Reel, J. J., & Greenleaf, C. (2015). Weight status and body image perceptions in adolescents: Current perspectives. Adolescent Health, Medicine and Therapeutics, 6, 149-158. https://dx.doi.org/10.2147%2FAHMT.S 68344 · DOI
  37. Wang, L., Zhang, Y., Wang, D., Tong, X., Liu, T., Zhang, S., Huang, J., Zhang, L., Chen, L., Fan, H., & Clarke, M. (2021). Artificial intelligence for COVID-19: A systematic review. Frontiers in Medicine, 8, 704256. https://doi.org/10.3389/fmed.2021.70425 · DOI
  38. Wright, S., Grogan, S., & Hunter, G. (2000). Motivations for anabolic steroid use among bodybuilders. Journal of Health Psychology, 5(4), 566-571. https://doi.org/10.1177/13591053000050 0413 · DOI
  39. Zhu, Q., & Luo, J. (2022). Generative pre- trained transformer for design concept generation: An exploration. Proceedings of the Design Society, 2, 1825-1834. https://doi.org/10.1017/pds.2022.185 30 · DOI
  40. Health Behavior Research, Vol. 6, No. 1 [2023], Art. 3 https://newprairiepress.org/hbr/vol6/iss1/3 DOI: 10.4148/2572-1836.1164

Cited by

Sorted by citations

Authors
An, R., Byron, C. W., Chen, C., & Xiang, X.
Affiliations
Washington University in St. Louis (US)
Corresponding author
An, Ruopeng
ORCID
Venue
Health Behavior Research
Publisher
Kansas State University
Published
2023-01-03
Volume
6
Issue
1
ISSN
2572-1836 (ISSN-L)
DOI
10.4148/2572-1836.1164
OpenAlex
W4313575150
Cited by
6
References
12
Citations by year
2026: 2 · 2025: 1 · 2024: 2 · 2023: 1
FWCI
1.55
Type
journal article
Language
en
Open access
Licence
cc-by-nc
Locations
  • Health Behavior Research (publishedVersion, cc-by-nc)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
  • New Prairie Press (Kansas State University) (submittedVersion, cc-by-nc)
Topics
  • Social Sciences › Psychology › Applied Psychology › Digital Mental Health Interventions
  • Social Sciences › Psychology › Social Psychology › Social Robot Interaction and HRI
  • Social Sciences › Psychology › Clinical Psychology › Eating Disorders and Behaviors
Keywords
chatbots, negative body image, body image, therapeutic chatbots, self-harm detection, adolescents, young adults, mental health, large language models, dialogue systems

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review