How to Guide Task-oriented Chatbot Users, and When: A Mixed-methods Study of Combinations of Chatbot Guidance Types and Timings
The DOI matches its Crossref record; 27 of the 27 references it cites were found in a registry or an index.
Abstract
The popularity of task-oriented chatbots is constantly growing, but smooth conversational progress with them remains profoundly challenging. In recent years, researchers have argued that chatbot systems should include guidance for users on how to converse with them. Nevertheless, empirical evidence about what to place in such guidance, and when to deliver it, has been lacking. Using a mixed-methods approach that integrates results from a between-subjects experiment and a reflection session, this paper compares the effectiveness of eight combinations of two guidance types (example-based and rule-based) at four guidance timings (service-onboarding, task-intro, after-failure, and upon-request), as measured by users’ task performance, improvement on subsequent tasks, and subjective experience. It establishes that each guidance type and timing has particular strengths and weaknesses, thus that each type/timing combination has a unique impact on performance metrics, learning outcomes, and user experience. On that basis, it presents guidance-design recommendations for future task-oriented chatbots.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
A guideline of selecting and reporting intraclass correlation coefficients for reliability research
Koo, T. K., & Li, M. Y. (2016) · Journal of Chiropractic Medicine · doi:10.1016/j.jcm.2016.02.012 · Cited by 31,709
A Comparison of Alternative Tests of Significance for the Problem of $m$ Rankings
Friedman, M. (1940) · The Annals of Mathematical Statistics · doi:10.1214/aoms/1177731944 · Cited by 2,785 · cited 2×
“A Friedman test [25] indicated that diferences among the rankings were statistically signifcant (�2(8)=320.”
BackgroundMethodGuidelines for Human-AI Interaction
Amershi, S., Weld, D., Vorvoreanu, M., Fourney, A., Nushi, B., Collisson, P., Suh, J., Iqbal, S., Bennett, P. N., Inkpen, K., et al. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300233 · Cited by 2,439 · cited 5×
“Therefore, while some prior work [3, 38] concluded that guidance should be placed at the initial stage of interaction, the present study specifcally clarifes that the example-based guidance would be better provided at the introduction of the task and not at service-onboarding.”
BackgroundInfluentialTask Complexity: A Review and Analysis
Campbell, D. J. (1988) · The Academy of Management Review · doi:10.5465/amr.1988.4306775 · Cited by 1,634
Luger, E., & Sellen, A. (2016) · Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems · doi:10.1145/2858036.2858288 · Cited by 1,176 · cited 2×
“Additionally, given that learnability is an important aspect of usability of an interactive system [60] and one that has previously been used for assessing whether a conversational user interface allows users to achieve successful interaction [52], we also measured learnability of the chatbot.”
BackgroundBrandtzaeg, P. B., & Følstad, A. (2017) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-70284-1_30 · Cited by 788 · cited 1×
“the question of whether catering to such preferences would yield empirically better communicative outcomes remains unanswered, despite a high proportion of chatbots use being outcome-focused [7, 9, 23].”
BackgroundChaves, A. P., & Gerosa, M. A. (2021) · International Journal of Human–Computer Interaction · doi:10.1080/10447318.2020.1841438 · Cited by 572 · cited 1×
“pectations by progressively issuing them clear guidance about AI capability and functionality [3, 15, 21, 33, 35, 38, 48, 52, 56, 58].”
BackgroundToward an Instructionally Oriented Theory of Example‐Based Learning
Renkl, A. (2014) · Cognitive Science · doi:10.1111/cogs.12086 · Cited by 509
Atkinson, R. K., Renkl, A., & Merrill, M. M. (2003) · Journal of Educational Psychology · doi:10.1037/0022-0663.95.4.774 · Cited by 491
Evaluating XAI: A comparison of rule-based and example-based explanations
van der Waa, J., Nieuwburg, E., Cremers, A., & Neerincx, M. (2021) · Artificial Intelligence · doi:10.1016/j.artint.2020.103404 · Cited by 393 · cited 1×
“[74] compared rule-based and example-based explanations of an Explainable AI system that generated explanations for decision support systems (DSS), and found that while the examplebased ones were better at encouraging people’s compliance with the system’s advice, rule-based explanations helped the participants more ac…”
Result
Cited by
AI Knowledge: Improving AI Delegation through Human Enablement
Pinski, M., Adam, M., & Benlian, A. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3580794 · Cited by 62
Chen, C., Lee, S., Jang, E., & Sundar, S. S. (2024) · Proceedings of the Second International Symposium on Trustworthy Autonomous Systems · doi:10.1145/3686038.3686060 · Cited by 21
Neshaei, S. P., Rietsche, R., Su, X., & Wambsganss, T. (2024) · Proceedings of the 29th International Conference on Intelligent User Interfaces · doi:10.1145/3640543.3645169 · Cited by 9
“The users may also benefit from an always-available help button, which offers guidance if they get confused or become unsure how to progress ( 𝐹 6 , also backed by the literature [114]).”
BackgroundGnewuch, U., & Reinkemeier, F. (2026) · MIS Quarterly · doi:10.25300/misq/2025/18742 · Cited by 5
Metadata
- Authors
- Yeh, S.-F., Wu, M.-H., Chen, T.-Y., Lin, Y.-C., Chang, X., Chiang, Y.-H., & Chang, Y.-J.
- Venue
- CHI Conference on Human Factors in Computing Systems
- Publisher
- ACM
- Published
- 2022-04-29
- Pages
- 1-16
- OpenAlex
- W4225011935
- Cited by
- 35
- References
- 27
- FWCI
- 4.49
- Type
- conference paper
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.