How to Guide Task-oriented Chatbot Users, and When: A Mixed-methods Study of Combinations of Chatbot Guidance Types and Timings
The DOI matches its Crossref record; 19 of the 19 references it cites were found in a registry or an index.
Abstract
The popularity of task-oriented chatbots is constantly growing, but smooth conversational progress with them remains profoundly challenging. In recent years, researchers have argued that chatbot systems should include guidance for users on how to converse with them. Nevertheless, empirical evidence about what to place in such guidance, and when to deliver it, has been lacking. Using a mixed-methods approach that integrates results from a between-subjects experiment and a reflection session, this paper compares the effectiveness of eight combinations of two guidance types (example-based and rule-based) at four guidance timings (service-onboarding, task-intro, after-failure, and upon-request), as measured by users’ task performance, improvement on subsequent tasks, and subjective experience. It establishes that each guidance type and timing has particular strengths and weaknesses, thus that each type/timing combination has a unique impact on performance metrics, learning outcomes, and user experience. On that basis, it presents guidance-design recommendations for future task-oriented chatbots.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
A guideline of selecting and reporting intraclass correlation coefficients for reliability research
Koo, T. K., & Li, M. Y. (2016) · Journal of Chiropractic Medicine · doi:10.1016/j.jcm.2016.02.012 · Cited by 31,709
A Comparison of Alternative Tests of Significance for the Problem of $m$ Rankings
Friedman, M. (1940) · The Annals of Mathematical Statistics · doi:10.1214/aoms/1177731944 · Cited by 2,785 · cited 2×
“A Friedman test [25] indicated that diferences among the rankings were statistically signifcant (�2(8)=320.”
BackgroundMethodGuidelines for Human-AI Interaction
Amershi, S., Weld, D., Vorvoreanu, M., Fourney, A., Nushi, B., Collisson, P., Suh, J., Iqbal, S., Bennett, P. N., Inkpen, K., et al. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300233 · Cited by 2,439 · cited 5×
“Therefore, while some prior work [3, 38] concluded that guidance should be placed at the initial stage of interaction, the present study specifcally clarifes that the example-based guidance would be better provided at the introduction of the task and not at service-onboarding.”
BackgroundInfluentialLuger, E., & Sellen, A. (2016) · Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems · doi:10.1145/2858036.2858288 · Cited by 1,176 · cited 2×
“Additionally, given that learnability is an important aspect of usability of an interactive system [60] and one that has previously been used for assessing whether a conversational user interface allows users to achieve successful interaction [52], we also measured learnability of the chatbot.”
BackgroundBrandtzaeg, P. B., & Følstad, A. (2017) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-70284-1_30 · Cited by 788 · cited 1×
“the question of whether catering to such preferences would yield empirically better communicative outcomes remains unanswered, despite a high proportion of chatbots use being outcome-focused [7, 9, 23].”
BackgroundChaves, A. P., & Gerosa, M. A. (2021) · International Journal of Human–Computer Interaction · doi:10.1080/10447318.2020.1841438 · Cited by 572 · cited 1×
“pectations by progressively issuing them clear guidance about AI capability and functionality [3, 15, 21, 33, 35, 38, 48, 52, 56, 58].”
BackgroundEvaluating XAI: A comparison of rule-based and example-based explanations
van der Waa, J., Nieuwburg, E., Cremers, A., & Neerincx, M. (2021) · Artificial Intelligence · doi:10.1016/j.artint.2020.103404 · Cited by 393 · cited 1×
“[74] compared rule-based and example-based explanations of an Explainable AI system that generated explanations for decision support systems (DSS), and found that while the examplebased ones were better at encouraging people’s compliance with the system’s advice, rule-based explanations helped the participants more ac…”
ResultLee, S., & Choi, J. (2017) · International Journal of Human-Computer Studies · doi:10.1016/j.ijhcs.2017.02.005 · Cited by 388
Evaluating and Informing the Design of Chatbots
Jain, M., Kumar, P., Kota, R., & Patel, S. N. (2018) · Proceedings of the 2018 Designing Interactive Systems Conference · doi:10.1145/3196709.3196735 · Cited by 376
Learning from Worked-Out Examples: The Effects of Example Variability and Elicited Self-Explanations
Renkl, A., Stark, R., Gruber, H., & Mandl, H. (1998) · Contemporary Educational Psychology · doi:10.1006/ceps.1997.0959 · Cited by 374
Cited by
AI Knowledge: Improving AI Delegation through Human Enablement
Pinski, M., Adam, M., & Benlian, A. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3580794 · Cited by 62
Chen, C., Lee, S., Jang, E., & Sundar, S. S. (2024) · Proceedings of the Second International Symposium on Trustworthy Autonomous Systems · doi:10.1145/3686038.3686060 · Cited by 21
Neshaei, S. P., Rietsche, R., Su, X., & Wambsganss, T. (2024) · Proceedings of the 29th International Conference on Intelligent User Interfaces · doi:10.1145/3640543.3645169 · Cited by 9
“The users may also benefit from an always-available help button, which offers guidance if they get confused or become unsure how to progress ( 𝐹 6 , also backed by the literature [114]).”
BackgroundGnewuch, U., & Reinkemeier, F. (2026) · MIS Quarterly · doi:10.25300/misq/2025/18742 · Cited by 5
Metadata
- Authors
- Yeh, S.-F., Wu, M.-H., Chen, T.-Y., Lin, Y.-C., Chang, X., Chiang, Y.-H., & Chang, Y.-J.
- Venue
- CHI Conference on Human Factors in Computing Systems
- Publisher
- ACM
- Published
- 2022-04-29
- Pages
- 1-16
- OpenAlex
- W4225011935
- Cited by
- 35
- References
- 19
- FWCI
- 4.49
- Type
- conference paper
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: How to Guide Task-oriented Chatbot Users, and When: A Mixed-methods Study of Combinations of Chatbot Guidance Types and Timings. https://app.cochat.ai/papers/10.1145/3491102.3501941
- The paper
- Yeh, S.-F., Wu, M.-H., Chen, T.-Y., Lin, Y.-C., Chang, X., Chiang, Y.-H., & Chang, Y.-J. (2022). How to Guide Task-oriented Chatbot Users, and When: A Mixed-methods Study of Combinations of Chatbot Guidance Types and Timings. CHI Conference on Human Factors in Computing Systems. https://doi.org/10.1145/3491102.3501941
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.