CoChat
Sign Up
Cited by 60Innovative Human-Technology InteractionMobile Health and mHealth ApplicationsArtificial Intelligence in Healthcare and Education

Beyond self-diagnosis: how a chatbot-based symptom checker should respond

Open access
You, Yue, Tsai, Chun-Hua, Li, Yao, Ma, Fenglong, Heron, Christopher, Gui, Xinning·2023·ACM Transactions on Computer-Human Interaction
DOI 10.1145/3589959
Open this paper in CoChat — free accountOpen PDF

Abstract

Chatbot-based symptom checker (CSC) apps have become increasingly popular in healthcare. These apps engage users in human-like conversations and offer possible medical diagnoses. The conversational design of these apps can significantly impact user perceptions and experiences, and may influence medical decisions users make and the medical care they receive. However, the effects of the conversational design of CSCs remain understudied, and there is a need to investigate and enhance users’ interactions with CSCs. In this article, we conducted a two-stage exploratory study using a human-centered design methodology. We first conducted a qualitative interview study to identify key user needs in engaging with CSCs. We then performed an experimental study to investigate potential CSC conversational design solutions based on the results from the interview study. We identified that emotional support, explanations of medical information, and efficiency were important factors for users in their interactions with CSCs. We also demonstrated that emotional support and explanations could affect user perceptions and experiences, and they are context-dependent. Based on these findings, we offer design implications for CSC conversations to improve the user experience and health-related decision-making.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1145/3589959

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

The citation lists haven’t been fetched yet

References 96 · Cited by 60

Research works printed but not linked: 53

  1. Amal Abdulrahman, Deborah Richards, and Ayse Aysin Bilgin. 2021. Reason explanation for encouraging behaviour change intention. In Proceedings of the 20th International Conference on Autonomous Agents and MultiAgent Systems. 68-77.
  2. Gillian Cameron, David Cameron, Gavin Megaw, Raymond Bond, Maurice Mulvenna, Siobhan O'Neill, Cherie Armour, and Michael McTear. 2017. Towards a chatbot for digital counselling. In Proceedings of the 31st International BCS Human Computer Interaction Conference. 1-7.
  3. Zhongxia Chen, Xiting Wang, Xing Xie, Mehul Parsana, Akshay Soni, Xiang Ao, and Enhong Chen. 2020. Towards explainable conversational recommendation. In Proceedings of the IJCAI. 2994-3000.
  4. Hyojin Chin, Lebogang Wame Molefi, and Mun Yong Yi. 2020. Empathy is all you need: How a conversational agent should respond to verbal abuse. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems. 1-13.
  5. Hyojin Chin and Mun Yong Yi. 2019. Should an agent be ignoring it? A study of verbal abuse types and conversational agents' response styles. In Proceedings of the Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems. 1-6.
  6. Khairul Azhar Mat Daud, Nik Zulkarnaen Khidzir, Ahmad Rasdan Ismail, and Fadhilahanim Aryani Abdullah. 2018. Validity and reliability of instrument to measure social media skills among small and medium entrepreneurs at Pengkalan Datu River. International Journal of Development and sustainability 7, 3 (2018), 1026-1037.
  7. Asbjørn Følstad and Marita Skjuve. 2019. Chatbots for customer service: User experience and motivation. In Proceed- ings of the 1st International Conference on Conversational User Interfaces. 1-9.
  8. Yang Gao, Zhengyu Pan, Honghao Wang, and Guanling Chen. 2018. Alexa, my love: Analyzing reviews of Amazon echo. In Proceedings of the 2018 IEEE SmartWorld, Ubiquitous Intelligence & Computing, Advanced & Trusted Comput- ing, Scalable Computing & Communications, Cloud & Big Data Computing, Internet of People and Smart City Innovation. IEEE, 372-380.
  9. Radhika Garg and Subhasree Sengupta. 2020. Conversational technologies for in-home learning: Using co-design to understand children's and parents' perspectives. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems. 1-13.
  10. Ulrich Gnewuch, Stefan Morana, Marc T. P. Adam, and Alexander Maedche. 2018. Faster is not always better: Un- derstanding the effect of dynamic response delays in human-chatbot interaction. In Proceedings of the 26th European Conference on Information Systems: Beyond Digitization-Facets of Socio-Technical Change. 143975.
  11. Mouadh Guesmi, Mohamed Amine Chatti, Laura Vorgerd, Shoeb Joarder, Shadi Zumor, Yiqi Sun, Fangzheng Ji, and Arham Muslim. 2021. On-demand personalized explanation for transparent recommendation. In Proceedings of the Adjunct Proceedings of the 29th ACM Conference on User Modeling, Adaptation and Personalization. 246-252.
  12. Sandra G. Hart and Lowell E. Staveland. 1988. Development of NASA-TLX (task load index): Results of empirical and theoretical research. In Proceedings of the Advances in Psychology. 139-183.
  13. Shafquat Hussain, Omid Ameri Sianaki, and Nedal Ababneh. 2019. A survey on conversational agents/chatbots clas- sification and design techniques. In Proceedings of the Workshops of the International Conference on Advanced Infor- mation Networking and Applications. Springer, 946-956.
  14. Angel Hsing-Chi Hwang and Andrea Stevenson Won. 2021. IdeaBot: Investigating social facilitation in human- machine team creativity. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems. 1-16.
  15. Youjin Hwang, Donghoon Shin, Sion Baek, Bongwon Suh, and Joonhwan Lee. 2021. Applying the Persona of user's family member and the doctor to the conversational agents for healthcare. In CHI 2020 Workshop on Conversational Agents for Health and Wellbeing.
  16. Mohit Jain, Ramachandra Kota, Pratyush Kumar, and Shwetak N. Patel. 2018. Convey: Exploring the use of a context view for chatbots. In Proceedings of the 2018 Chi Conference on Human Factors in Computing Systems. 1-6.
  17. Junhan Kim, Yoojung Kim, Byungjoon Kim, Sukyung Yun, Minjoon Kim, and Joongseek Lee. 2018. Can a machine tend to teenagers' emotional needs? A study with conversational agents. In Proceedings of the Extended Abstracts of the 2018 CHI Conference on Human Factors in Computing Systems. 1-6.
  18. Junhan Kim, Jana Muhic, Lionel Peter Robert, and Sun Young Park. 2022. Designing chatbots with black americans with chronic conditions: Overcoming challenges against COVID-19. In Proceedings of the CHI Conference on Human Factors in Computing Systems. 1-17.
  19. Soomin Kim, Jinsu Eun, Changhoon Oh, Bongwon Suh, and Joonhwan Lee. 2020. Bot in the bunch: Facilitating group chat discussion by improving efficiency and participation with a chatbot. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems. 1-13.
  20. Soomin Kim, Joonhwan Lee, and Gahgene Gweon. 2019. Comparing data from chatbot and web surveys: Effects of platform and conversational style on survey response quality. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems. 1-12.
  21. A Baki Kocaballi, Juan C. Quiroz, Liliana Laranjo, Dana Rezazadegan, Rafal Kocielnik, Leigh Clark, Q. Vera Liao, Sun Young Park, Robert J. Moore, and Adam Miner. 2020. Conversational agents for health and wellbeing. In Pro- ceedings of the Extended Abstracts of the 2020 CHI Conference on Human Factors in Computing Systems. 1-8.
  22. Sari Kujala, Iiris Hörhammer, Riitta Hänninen-Ervasti, Tarja Heponiemi. 2020. Health professionals' experiences of the benefits and challenges of online symptom checkers. In Proceedings of the MIE. 966-970.
  23. Raina Langevin, Ross J. Lordon, Thi Avrahami, Benjamin R. Cowan, Tad Hirsch, and Gary Hsieh. 2021. Heuristic eval- uation of conversational agents. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems. 1-15.
  24. Minha Lee, Sander Ackermans, Nena Van As, Hanwen Chang, Enzo Lucas, and Wijnand IJsselsteijn. 2019. Caring for Vincent: A chatbot for self-compassion. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems. 1-13.
  25. Q. Vera Liao, Daniel Gruen, and Sarah Miller. 2020. Questioning the AI: Informing design practices for explainable AI user experiences. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems. 1-15.
  26. Mingming Liu, Qicheng Ding, Yu Zhang, Guoguang Zhao, Changjian Hu, Jiangtao Gong, Penghui Xu, Yu Zhang, Liuxin Zhang, and Qianying Wang. 2020. Cold comfort matters-how channel-wise emotional strategies help in a customer service chatbot. In Proceedings of the Extended Abstracts of the 2020 CHI Conference on Human Factors in Computing Systems. 1-7.
  27. Nika Mozafari, Welf H. Weiger, and Maik Hammerschmidt. 2020. The chatbot disclosure dilemma: Desirable and undesirable effects of disclosing the non-human identity of chatbots. In Proceedings of the ICIS.
  28. Andreea Muresan and Henning Pohl. 2019. Chats with bots: Balancing imitation and engagement. In Proceedings of the Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems. 1-6.
  29. Fedelucio Narducci, Marco de Gemmis, Pasquale Lops, and Giovanni Semeraro. 2018. Improving the user experience with a conversational recommender system. In Proceedings of the International Conference of the Italian Association for Artificial Intelligence. Springer, 528-538.
  30. Stefan Olafsson, Teresa K. O'Leary, and Timothy W. Bickmore. 2020. Motivating health behavior change with hu- morous virtual agents. In Proceedings of the 20th ACM International Conference on Intelligent Virtual Agents. 1-8.
  31. Hyanghee Park and Joonhwan Lee. 2020. Can a conversational agent lower sexual violence victims' burden of self- disclosure?. In Proceedings of the Extended Abstracts of the 2020 CHI Conference on Human Factors in Computing Systems. 1-8.
  32. Sihang Qiu, Ujwal Gadiraju, and Alessandro Bozzon. 2020. Improving worker engagement through conversational microtask crowdsourcing. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems. 1-12.
  33. Emilee Rader, Kelley Cotter, and Janghee Cho. 2018. Explanations as mechanisms for supporting algorithmic trans- parency. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. 1-13.
  34. Mireia Ribera and Agata Lapedriza. 2019. Can we do better explanations? A proposal of user-centered explainable AI.. In Proceedings of the IUI Workshops, Vol. 2327. 38.
  35. Hyeyoung Ryu, Soyeon Kim, Dain Kim, Sooan Han, Keeheon Lee, and Younah Kang. 2020. Simple and steady inter- actions win the healthy mentality: Designing a chatbot service for the elderly. Proceedings of the ACM on Human- Computer Interaction 4, CSCW2 (2020), 1-25.
  36. Samiha Samrose, Kavya Anbarasu, Ajjen Joshi, and Taniya Mishra. 2020. Mitigating Boredom Using An Empathetic Conversational Agent. In Proceedings of the 20th ACM International Conference on Intelligent Virtual Agents. 1-8.
  37. Briane Paul V. Samson and Yasuyuki Sumi. 2020. Are two heads better than one? Exploring two-party conversations for car navigation voice guidance. In Proceedings of the Extended Abstracts of the 2020 CHI Conference on Human Factors in Computing Systems. 1-9.
  38. Shruti Sannon, Brett Stoll, Dominic DiFranzo, Malte Jung, and Natalya N. Bazarova. 2018. How personification and interactivity influence stress-related disclosures to conversational agents. In Proceedings of the Companion of the 2018 ACM Conference on Computer Supported Cooperative Work and Social Computing. 285-288.
  39. Jessica Schroeder, Chelsey Wilkes, Kael Rowan, Arturo Toledo, Ann Paradiso, Mary Czerwinski, Gloria Mark, and Marsha M. Linehan. 2018. Pocket skills: A conversational mobile web app to support dialectical behavioral therapy. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. 1-15.
  40. Kristen M. Scott, Simone Ashby, and Julian Hanna. 2020. "Human, all too human": NOAA weather radio and the emotional impact of synthetic voices. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems. 1-9.
  41. Ameneh Shamekhi, Q Vera Liao, Dakuo Wang, Rachel K. E. Bellamy, and Thomas Erickson. 2018. Face value? Ex- ploring the effects of embodiment for a group facilitation agent. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. 1-13.
  42. Weiyan Shi, Xuewei Wang, Yoo Jung Oh, Jingwen Zhang, Saurav Sahay, and Zhou Yu. 2020. Effects of persuasive dialogues: Testing bot identities and inquiry strategies. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems. 1-13.
  43. Kacper Sokol and Peter A. Flach. 2018. Glass-box: Explaining AI decisions with counterfactual statements through conversation with a voice-enabled virtual assistant. In Proceedings of the IJCAI. 5868-5870.
  44. Yuan Sun and S. Shyam Sundar. 2022. Exploring the effects of interactive dialogue in improving user control for explainable online symptom checkers. In Proceedings of the CHI Conference on Human Factors in Computing Systems Extended Abstracts. 1-7.
  45. Ella Tallyn, Hector Fried, Rory Gianni, Amy Isard, and Chris Speed. 2018. The ethnobot: Gathering ethnographies in the age of IoT. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. 1-13.
  46. Chun-Hua Tsai and Peter Brusilovsky. 2021. The effects of controllability and explainability in a social recommender system. User Modeling and User-Adapted Interaction 31, 3 (2021), 591-627.
  47. Chun-Hua Tsai, Yue You, Xinning Gui, Yubo Kou, and John M. Carroll. 2021. Exploring and promoting diagnostic transparency and explainability in online symptom checkers. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems. 1-17.
  48. Danding Wang, Qian Yang, Ashraf Abdul, and Brian Y. Lim. 2019. Designing theory-driven user-centric explainable AI. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems. 1-15.
  49. Weiyu Wang and Keng Siau. 2018. Living with artificial intelligence-developing a theory on trust in health chatbots. In Proceedings of the 16th Annual Pre-ICIS Workshop on HCI Research in MIS. Association for Information Systems San Francisco, CA.
  50. Justin D. Weisz, Mohit Jain, Narendra Nath Joshi, James Johnson, and Ingrid Lange. 2019. BigBlueBot: Teaching strategies for successful human-agent interactions. In Proceedings of the 24th International Conference on Intelligent User Interfaces. 448-459.
  51. Xi Yang and Marco Aurisicchio. 2021. Designing conversational agents: A self-determination theory approach. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems. 1-16.
  52. Yue You and Xinning Gui. 2020. Self-diagnosis through AI-enabled chatbot-based symptom checkers: User experi- ences and design considerations. In Proceedings of the AMIA Annual Symposium Proceedings, Vol. 2020. American Medical Informatics Association, 1354.
  53. Yue You, Yubo Kou, Xianghua Ding, and Xinning Gui. 2021. The medical authority of AI: A study of AI-enabled consumer-facing health technology. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Sys- tems. 1-16.

Authors
You, Y., Tsai, C.-H., Li, Y., Ma, F., Heron, C., & Gui, X.
Affiliations
  • Pennsylvania State University (US)
  • University of Nebraska at Omaha (US)
  • University of Central Florida (US)
ORCID
Venue
ACM Transactions on Computer-Human Interaction
Publisher
Association for Computing Machinery (ACM)
Published
2023-08-31
Volume
30
Issue
4
Pages
1-44
ISSN
  • 1073-0516 (ISSN-L)
  • 1557-7325
DOI
10.1145/3589959
OpenAlex
W4362472235
Cited by
60
References
96
Citations by year
2026: 17 · 2025: 20 · 2024: 20 · 2023: 3
FWCI
10.39
Type
journal article
Language
en
Open access
Licence
www.acm.org/publications/policies/copyright_policy#Background (vor)
Locations
  • ACM Transactions on Computer-Human Interaction (publishedVersion)
  • DigitalCommons@UNO (submittedVersion)
  • ScholarSphere (Penn State Libraries) (submittedVersion)
Funders
Penn State College of IST’s (150000004308)
Topics
  • Physical Sciences › Computer Science › Human-Computer Interaction › Innovative Human-Technology Interaction
  • Health Sciences › Health Professions › General Health Professions › Mobile Health and mHealth Applications
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
Keywords
conversational design, self-diagnosis, emotional support, user experience, health decision-making, human-centered design, chatbot responses

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review