A Conversation Analysis of Non-Progress and Coping Strategies with a Banking Task-Oriented Chatbot
The DOI matches its Crossref record; 30 of the 30 references it cites were found in a registry or an index.
Abstract
Task-oriented chatbots are becoming popular alternatives for fulfilling users' needs, but few studies have investigated how users cope with conversational 'non-progress' (NP) in their daily lives. Accordingly, we analyzed a three-month conversation log between 1,685 users and a task-oriented banking chatbot. In this data, we observed 12 types of conversational NP; five types of content that was unexpected and challenging for the chatbot to recognize; and 10 types of coping strategies. Moreover, we identified specific relationships between NP types and strategies, as well as signs that users were about to abandon the chatbot, including 1) three consecutive incidences of NP, 2) consecutive use of message reformulation or switching subjects, and 3) using message reformulation as the final strategy. Based on these findings, we provide design recommendations for task-oriented chatbots, aimed at reducing NP, guiding users through such NP, and improving user experiences to reduce the cessation of chatbot use.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“However, such chatbots are usually unable to handle user messages that are out of the pre-defined intents ( e.g. , a user attempts to have small talk with the banking chatbot) [57].”
From Leveraging Large Language Models to Power Chatbots for Collecting User Self-Reported Data (2024) · cites this paper as [57]
In the citing paper
“Task-oriented chatbots usually have limited number of pre-defined user intents to accomplish a specific goal.”
From Leveraging Large Language Models to Power Chatbots for Collecting User Self-Reported Data (2024) · cites this paper as [57]
Citations
Cites
Luger, E., & Sellen, A. (2016) · Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems · doi:10.1145/2858036.2858288 · Cited by 1,176
Designing the user interface: strategies for effective human-computer interaction
Briggs, J. S. (1987) · Information and Software Technology · doi:10.1016/0950-5849(87)90040-1 · Cited by 1,146
Characterizing browsing strategies in the World-Wide web
Catledge, L. D., & Pitkow, J. E. (1995) · Computer Networks and ISDN Systems · doi:10.1016/0169-7552(95)00043-7 · Cited by 937 · cited 1×
“In line with prior research [6,14] and our definition of abandonment of the chatbot, we defined temporarily quitting the conversation as exiting it for more than 30 minutes but less than 10 days.”
MethodA Theoretical Framework for Conversational Search
Radlinski, F., & Craswell, N. (2017) · Proceedings of the 2017 Conference on Conference Human Information Interaction and Retrieval · doi:10.1145/3020165.3020183 · Cited by 439
Beyond the session timeout: automatic hierarchical segmentation of search topics in query logs
Jones, R., & Klinkner, K. L. (2008) · Proceedings of the 17th ACM conference on Information and knowledge management · doi:10.1145/1458082.1458176 · Cited by 385
Evaluating and Informing the Design of Chatbots
Jain, M., Kumar, P., Kota, R., & Patel, S. N. (2018) · Proceedings of the 2018 Designing Interactive Systems Conference · doi:10.1145/3196709.3196735 · Cited by 376
Chatbots, Humbots, and the Quest for Artificial General Intelligence
Grudin, J., & Jacques, R. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300439 · Cited by 275 · cited 1×
“Chatbots have been deployed in online platforms in various fields [9], and in 2018, more than 300,000 chatbots were said to be active on Facebook Messenger alone [5].”
BackgroundResilient Chatbots: Repair Strategy Preferences for Conversational Breakdowns
Ashktorab, Z., Jain, M., Liao, Q. V., & Weisz, J. D. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300484 · Cited by 224 · cited 5×
“Meanwhile, people’s willingness to use CAs were found to be influenced by several variables, such as systems’ low reliability and users’ poor mental models of what a VUI is capable of [20], and conversation breakdowns during the interaction with chatbots [4,12].”
BackgroundInfluentialPatterns for How Users Overcome Obstacles in Voice User Interfaces
Myers, C., Furqan, A., Nebolsky, J., Caro, K., & Zhu, J. (2018) · Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3173580 · Cited by 213 · cited 5×
“Secondly, while both Stent et al.[29] and Myers et al. [23] found users most often used hyper-articulation, and the latter added that simplification was also often used to cope with recognition errors, our dataset revealed very infrequent use of repeating and removing words, the two coping…”
BackgroundInfluentialAnalysing technological affordances of online interactions using conversation analysis
Meredith, J. (2017) · Journal of Pragmatics · doi:10.1016/j.pragma.2017.03.001 · Cited by 164
I'm Sorry, Dave, I'm Afraid I Can't Do That: Chatbot Perception and Expectations
Zamora, J. (2017) · Proceedings of the 5th International Conference on Human Agent Interaction · doi:10.1145/3125739.3125766 · Cited by 164
(2010) · doi:10.5555/1734454.1734544 · Cited by 163
Gracefully mitigating breakdowns in robotic services
Lee, M. K., Kiesler, S., Forlizzi, J., Srinivasa, S., & Rybski, P. (2010) · 2010 5th ACM/IEEE International Conference on Human-Robot Interaction (HRI) · doi:10.1109/hri.2010.5453195 · Cited by 121
FarmChat: A Conversational Agent to Answer Farmer Queries
Jain, M., Kumar, P., Bhansali, I., Liao, Q. V., Truong, K., & Patel, S. (2018) · Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies · doi:10.1145/3287048 · Cited by 117
Liao, Q. V., Mas-ud Hussain, M., Chandar, P., Davis, M., Khazaeni, Y., Crasso, M. P., Wang, D., Muller, M., Shami, N. S., & Geyer, W. (2018) · Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3173577 · Cited by 116 · cited 2×
“Researchers have studied how users interact with CAs [7,16,17,20,21,30,32], including chatbots.”
BackgroundHow Do You Want Your Chatbot? An Exploratory Wizard-of-Oz Study with Young, Urban Indians
Medhi Thies, I., Menon, N., Magapu, S., Subramony, M., & O’Neill, J. (2017) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-67744-6_28 · Cited by 104
Chaves, A. P., & Gerosa, M. A. (2020) · Figshare · doi:10.6084/m9.figshare.13207221.v1 · Cited by 97
Chaves, A. P., & Gerosa, M. A. (2020) · Figshare · doi:10.6084/m9.figshare.13207221 · Cited by 96
Adapting speaking after evidence of misrecognition: Local and global hyperarticulation
Stent, A. J., Huffman, M. K., & Brennan, S. E. (2007) · Speech Communication · doi:10.1016/j.specom.2007.07.005 · Cited by 93
Liao, Q. V., Davis, M., Geyer, W., Muller, M., & Shami, N. S. (2016) · Proceedings of the 2016 ACM Conference on Designing Interactive Systems · doi:10.1145/2901790.2901842 · Cited by 92
Jiang, J., Jeng, W., & He, D. (2013) · Proceedings of the 36th international ACM SIGIR conference on Research and development in information retrieval · doi:10.1145/2484028.2484092 · Cited by 84
Machine Learning for Dialog State Tracking: A Review
Henderson, M. (2015) · Cited by 66
Where's the "party" in "multi-party"? analyzing the structure of small-group sociable talk
Aoki, P. M., Szymanski, M. H., Plurkowski, L., Thornton, J. D., Woodruff, A., & Yi, W. (2006) · ACM Conference on Computer Supported Cooperative Work (CSCW) · doi:10.1145/1180875.1180934 · Cited by 57
An analysis of web-chat in an outsourced customer service account in the Philippines
Lockwood, J. (2017) · English for Specific Purposes · doi:10.1016/j.esp.2017.04.001 · Cited by 47
“Here's the link”: Hyperlinking in service-focused chat interaction
Stommel, W., Paulus, T. M., & Atkins, D. P. (2017) · Journal of Pragmatics · doi:10.1016/j.pragma.2017.02.009 · Cited by 44
Engelhardt, S., Hansson, E., & Leite, I. (2017) · Cited by 38
BigBlueBot: teaching strategies for successful human-agent interactions
Weisz, J. D., Jain, M., Joshi, N. N., Johnson, J., & Lange, I. (2019) · Proceedings of the 24th International Conference on Intelligent User Interfaces · doi:10.1145/3301275.3302290 · Cited by 38
Managing Uncertainty in Time Expressions for Virtual Assistants
Rong, X., Fourney, A., Brewer, R. N., Morris, M. R., & Bennett, P. N. (2017) · Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3025453.3025674 · Cited by 30
Voice User Interface Design Patterns.
Schnelle, D., & Lyardet, F. (2006) · European Conference on Pattern Languages of Programs · Cited by 26
The role of spoken feedback in experiencing multimodal interfaces as human-like
Qvarfordt, P., Jönsson, A., & Dahlbäck, N. (2003) · INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION · doi:10.1145/958432.958478 · Cited by 21
Research works printed but not linked: 4
- Adobe. 2019 Digital Trends: Financial Services in Focus. Retrieved September 20, 2019 from https://www.adobe.com/content/dam/acom/uk/modal -offers/2019/DT-Report-2019/Econsultancy-2019Digital-Trends-Financial-Services.pdf
- 10.1016/0169--7552(95)00043--7 · doi:10.1016/0169--7552(95)00043--7
- Ana Paula Chaves and Marco Aurelio Gerosa. 2019. How should my chatbot interact? A survey on human-chatbot interaction design. arXiv:1904.02743 [cs] (April 2019). Retrieved September 20, 2019 from http://arxiv.org/abs/1904.02743 · arXiv
- Matthew Lombard, Jennifer Snyder-Duch, and Cheryl Bracken. 2005. Practical Resources for Assessing and Reporting Intercoder Reliability in Content Analysis Research Projects. Retrieved April 19, (January 2005).
Cited by
Rapp, A., Curti, L., & Boldi, A. (2021) · International Journal of Human-Computer Studies · doi:10.1016/j.ijhcs.2021.102630 · Cited by 643
Haugeland, I. K. F., Følstad, A., Taylor, C., & Bjørkli, C. A. (2022) · International Journal of Human-Computer Studies · doi:10.1016/j.ijhcs.2022.102788 · Cited by 235
Conversational Agents: Goals, Technologies, Vision and Challenges
Allouch, M., Azaria, A., & Azoulay, R. (2021) · Sensors · doi:10.3390/s21248448 · Cited by 168
“[192] investigated how users cope with conversations with chatbots that do not make any progress in the field of customer support.”
BackgroundIntelligent Agents with LLM-based Process Automation
Guan, Y., Wang, D., Chu, Z., Wang, S., Ni, F., Song, R., & Zhuang, C. (2024) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3637528.3671646 · Cited by 144
Khennouche, F., Elmir, Y., Himeur, Y., Djebari, N., & Amira, A. (2024) · Expert Systems with Applications · doi:10.1016/j.eswa.2024.123224 · Cited by 124
Metadata
- Authors
- Li, C.-H., Yeh, S.-F., Chang, T.-J., Tsai, M.-H., Chen, K., & Chang, Y.-J.
- Affiliations
- National Yang Ming Chiao Tung University (TW)
- Venue
- Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems
- Publisher
- ACM
- Published
- 2020-04-21
- Pages
- 1-12
- OpenAlex
- W3030085124
- Cited by
- 57
- References
- 29
- Citations by year
- 2026: 2 · 2025: 13 · 2024: 13 · 2023: 12 · 2022: 9 · 2021: 8
- FWCI
- 2.77
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- CHI Conference on Human Factors in Computing Systems (publishedVersion)
- Funders
- Ministry of Science and Technology, Taiwan (108-2218-E-009 -050)
- Topics
- Social Sciences › Business, Management and Accounting › Marketing › AI in Service Interactions
- Physical Sciences › Computer Science › Artificial Intelligence › Speech and dialogue systems
- Social Sciences › Psychology › Social Psychology › Social Robot Interaction and HRI
- Keywords
- task-oriented chatbots, banking chatbots, user experience, conversation logs
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.