How to Guide Task-oriented Chatbot Users, and When: A Mixed-methods Study of Combinations of Chatbot Guidance Types and Timings
The DOI matches its Crossref record; 24 of the 24 references it cites were found in a registry or an index.
Abstract
The popularity of task-oriented chatbots is constantly growing, but smooth conversational progress with them remains profoundly challenging. In recent years, researchers have argued that chatbot systems should include guidance for users on how to converse with them. Nevertheless, empirical evidence about what to place in such guidance, and when to deliver it, has been lacking. Using a mixed-methods approach that integrates results from a between-subjects experiment and a reflection session, this paper compares the effectiveness of eight combinations of two guidance types (example-based and rule-based) at four guidance timings (service-onboarding, task-intro, after-failure, and upon-request), as measured by users’ task performance, improvement on subsequent tasks, and subjective experience. It establishes that each guidance type and timing has particular strengths and weaknesses, thus that each type/timing combination has a unique impact on performance metrics, learning outcomes, and user experience. On that basis, it presents guidance-design recommendations for future task-oriented chatbots.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
A guideline of selecting and reporting intraclass correlation coefficients for reliability research
Koo, T. K., & Li, M. Y. (2016) · Journal of Chiropractic Medicine · doi:10.1016/j.jcm.2016.02.012 · Cited by 31,709
A Comparison of Alternative Tests of Significance for the Problem of $m$ Rankings
Friedman, M. (1940) · The Annals of Mathematical Statistics · doi:10.1214/aoms/1177731944 · Cited by 2,785 · cited 2×
“A Friedman test [25] indicated that diferences among the rankings were statistically signifcant (�2(8)=320.”
BackgroundMethodGuidelines for Human-AI Interaction
Amershi, S., Weld, D., Vorvoreanu, M., Fourney, A., Nushi, B., Collisson, P., Suh, J., Iqbal, S., Bennett, P. N., Inkpen, K., et al. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300233 · Cited by 2,439 · cited 5×
“Therefore, while some prior work [3, 38] concluded that guidance should be placed at the initial stage of interaction, the present study specifcally clarifes that the example-based guidance would be better provided at the introduction of the task and not at service-onboarding.”
BackgroundInfluentialTask Complexity: A Review and Analysis
Campbell, D. J. (1988) · The Academy of Management Review · doi:10.5465/amr.1988.4306775 · Cited by 1,634
Luger, E., & Sellen, A. (2016) · Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems · doi:10.1145/2858036.2858288 · Cited by 1,176 · cited 2×
“Additionally, given that learnability is an important aspect of usability of an interactive system [60] and one that has previously been used for assessing whether a conversational user interface allows users to achieve successful interaction [52], we also measured learnability of the chatbot.”
BackgroundBrandtzaeg, P. B., & Følstad, A. (2017) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-70284-1_30 · Cited by 788 · cited 1×
“the question of whether catering to such preferences would yield empirically better communicative outcomes remains unanswered, despite a high proportion of chatbots use being outcome-focused [7, 9, 23].”
BackgroundChaves, A. P., & Gerosa, M. A. (2021) · International Journal of Human–Computer Interaction · doi:10.1080/10447318.2020.1841438 · Cited by 572 · cited 1×
“pectations by progressively issuing them clear guidance about AI capability and functionality [3, 15, 21, 33, 35, 38, 48, 52, 56, 58].”
BackgroundToward an Instructionally Oriented Theory of Example‐Based Learning
Renkl, A. (2014) · Cognitive Science · doi:10.1111/cogs.12086 · Cited by 509
Evaluating XAI: A comparison of rule-based and example-based explanations
van der Waa, J., Nieuwburg, E., Cremers, A., & Neerincx, M. (2021) · Artificial Intelligence · doi:10.1016/j.artint.2020.103404 · Cited by 393 · cited 1×
“[74] compared rule-based and example-based explanations of an Explainable AI system that generated explanations for decision support systems (DSS), and found that while the examplebased ones were better at encouraging people’s compliance with the system’s advice, rule-based explanations helped the participants more ac…”
ResultLee, S., & Choi, J. (2017) · International Journal of Human-Computer Studies · doi:10.1016/j.ijhcs.2017.02.005 · Cited by 388
Evaluating and Informing the Design of Chatbots
Jain, M., Kumar, P., Kota, R., & Patel, S. N. (2018) · Proceedings of the 2018 Designing Interactive Systems Conference · doi:10.1145/3196709.3196735 · Cited by 376
Learning from Worked-Out Examples: The Effects of Example Variability and Elicited Self-Explanations
Renkl, A., Stark, R., Gruber, H., & Mandl, H. (1998) · Contemporary Educational Psychology · doi:10.1006/ceps.1997.0959 · Cited by 374
Training wheels in a user interface
Carroll, J. M., & Carrithers, C. (1984) · Communications of the ACM · doi:10.1145/358198.358218 · Cited by 345 · cited 1×
“guidance may be unexpected, annoying and/or distracting [12, 14, 62], so providing users with guidance upon request might be more in line with their expectations [22].”
BackgroundChatbots: changing user needs and motivations
Brandtzaeg, P. B., & Følstad, A. (2018) · Interactions · doi:10.1145/3236669 · Cited by 322
Chatbots, Humbots, and the Quest for Artificial General Intelligence
Grudin, J., & Jacques, R. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300439 · Cited by 275 · cited 1×
“more than 300,000 chatbots were live on Facebook Messenger, and were commonly used to transact business [6] in multiple domains [28].”
BackgroundRules and instances in foreign language learning: Interactions of explicit and implicit knowledge
Ellis, N. (1993) · European Journal of Cognitive Psychology · doi:10.1080/09541449308520120 · Cited by 269 · cited 2×
“two types of guidance have been mentioned in other felds, such as language learning [20], and our results show that these distinctions also hold when it comes to guiding people’s communication with task-oriented chatbots.”
BackgroundResilient Chatbots: Repair Strategy Preferences for Conversational Breakdowns
Ashktorab, Z., Jain, M., Liao, Q. V., & Weisz, J. D. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300484 · Cited by 224 · cited 4×
“extensively used in prior chatbot research [4, 38, 47, 50].”
BackgroundInfluentialThe effects of example-based explanations in a machine learning interface
Cai, C. J., Jongejan, J., & Holbrook, J. (2019) · Proceedings of the 24th International Conference on Intelligent User Interfaces · doi:10.1145/3301275.3302289 · Cited by 220 · cited 1×
“Example-based guidance, for example, has frequently been recommended or adopted in various kinds of intelligent user interface [10, 38, 40, 72].”
BackgroundPatterns for How Users Overcome Obstacles in Voice User Interfaces
Myers, C., Furqan, A., Nebolsky, J., Caro, K., & Zhu, J. (2018) · Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3173580 · Cited by 213 · cited 1×
“pectations by progressively issuing them clear guidance about AI capability and functionality [3, 15, 21, 33, 35, 38, 48, 52, 56, 58].”
BackgroundInteracting meaningfully with machine learning systems: Three experiments
Stumpf, S., Rajaram, V., Li, L., Wong, W.-K., Burnett, M., Dietterich, T., Sullivan, E., & Herlocker, J. (2009) · International Journal of Human-Computer Studies · doi:10.1016/j.ijhcs.2009.03.004 · Cited by 184
Case study—Embodied virtual agents: An analysis on reasons for failure
Ben Mimoun, M. S., Poncin, I., & Garnier, M. (2012) · Journal of Retailing and Consumer Services · doi:10.1016/j.jretconser.2012.07.006 · Cited by 136 · cited 2×
“estimation or other misunderstanding of their chatbot’s capability [37, 38, 44, 48, 55].”
BackgroundFarmChat: A Conversational Agent to Answer Farmer Queries
Jain, M., Kumar, P., Bhansali, I., Liao, Q. V., Truong, K., & Patel, S. (2018) · Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies · doi:10.1145/3287048 · Cited by 117
Application Programming Interface Documentation: What Do Software Developers Want?
Meng, M., Steinhardt, S., & Schubert, A. (2018) · Journal of Technical Writing and Communication · doi:10.1177/0047281617721853 · Cited by 78
Improving Conversations: Lessons Learnt from Manual Analysis of Chatbot Dialogues
Kvale, K., Sell, O. A., Hodnebrog, S., & Følstad, A. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-39540-7_13 · Cited by 51
Cited by
AI Knowledge: Improving AI Delegation through Human Enablement
Pinski, M., Adam, M., & Benlian, A. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3580794 · Cited by 62
Chen, C., Lee, S., Jang, E., & Sundar, S. S. (2024) · Proceedings of the Second International Symposium on Trustworthy Autonomous Systems · doi:10.1145/3686038.3686060 · Cited by 21
Neshaei, S. P., Rietsche, R., Su, X., & Wambsganss, T. (2024) · Proceedings of the 29th International Conference on Intelligent User Interfaces · doi:10.1145/3640543.3645169 · Cited by 9
“The users may also benefit from an always-available help button, which offers guidance if they get confused or become unsure how to progress ( 𝐹 6 , also backed by the literature [114]).”
BackgroundGnewuch, U., & Reinkemeier, F. (2026) · MIS Quarterly · doi:10.25300/misq/2025/18742 · Cited by 5
Metadata
- Authors
- Yeh, S.-F., Wu, M.-H., Chen, T.-Y., Lin, Y.-C., Chang, X., Chiang, Y.-H., & Chang, Y.-J.
- Venue
- CHI Conference on Human Factors in Computing Systems
- Publisher
- ACM
- Published
- 2022-04-29
- Pages
- 1-16
- OpenAlex
- W4225011935
- Cited by
- 35
- References
- 24
- FWCI
- 4.49
- Type
- conference paper
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.