Investigating the user experience of customer service chatbot interaction: a framework for qualitative analysis of chatbot dialogues
Open accessThe DOI matches its Crossref record; 19 of the 19 references it cites were found in a registry or an index.
Abstract
The uptake of chatbots for customer service depends on the user experience. For such chatbots, user experience in particular concerns whether the user is provided relevant answers to their queries and the chatbot interaction brings them closer to resolving their problem. Dialogue data from interactions between users and chatbots represents a potentially valuable source of insight into user experience. However, there is a need for knowledge of how to make use of these data. Motivated by this, we present a framework for qualitative analysis of chatbot dialogues in the customer service domain. The framework has been developed across several studies involving two chatbots for customer service, in collaboration with the chatbot hosts. We present the framework and illustrate its application with insights from three case examples. Through the case findings, we show how the framework may provide insight into key drivers of user experience, including response relevance and dialogue helpfulness (Case 1), insight to drive chatbot improvement in practice (Case 2), and insight of theoretical and practical relevance for understanding chatbot user types and interaction patterns (Case 3). On the basis of the findings, we discuss the strengths and limitations of the framework, its theoretical and practical implications, and directions for future work.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Design Science in Information Systems Research1
Hevner, A. R., March, S. T., Park, J., & Ram, S. (2004) · MIS Quarterly · doi:10.2307/25148625 · Cited by 10,707
Design science in information systems research
(2004) · Journal of the Association for Information Systems · doi:10.5555/2017212.2017217 · Cited by 8,254
Fitzpatrick, K. K., Darcy, A., & Vierhile, M. (2017) · JMIR Mental Health · doi:10.2196/mental.7785 · Cited by 2,730
AI-based chatbots in customer service and their effects on user compliance
Adam, M., Wessel, M., & Benlian, A. (2021) · Electronic Markets · doi:10.1007/s12525-020-00414-7 · Cited by 1,195 · cited 3×
“For example, introducing anthropomorphic cues in chatbots for customer service have been found to have positive implications for user compliance with a chatbot's feedback requests [6] and emotional connection with the company following chatbot interaction [22].”
BackgroundLuger, E., & Sellen, A. (2016) · Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems · doi:10.1145/2858036.2858288 · Cited by 1,176
Overlapping talk and the organization of turn-taking for conversation
SCHEGLOFF, E. A. (2000) · Language in Society · doi:10.1017/s0047404500001019 · Cited by 1,163
Go, E., & Sundar, S. S. (2019) · Computers in Human Behavior · doi:10.1016/j.chb.2019.01.020 · Cited by 1,089
Hill, J., Randolph Ford, W., & Farreras, I. G. (2015) · Computers in Human Behavior · doi:10.1016/j.chb.2015.02.026 · Cited by 856
Brandtzaeg, P. B., & Følstad, A. (2017) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-70284-1_30 · Cited by 788 · cited 2×
“The promise of efficiency, effectiveness and convenience has been identified as main motivations for chatbot use in general [21] and the pragmatic quality of the dialogue outcome and process is also the premier determinants of user experience in chatbots for customer service [16, 17].”
BackgroundSelf-citationRecipes for Building an Open-Domain Chatbot
Roller, S., Dinan, E., Goyal, N., Ju, D., Williamson, M., Liu, Y., Xu, J., Ott, M., Smith, E. M., Boureau, Y.-L., et al. (2021) · Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume · doi:10.18653/v1/2021.eacl-main.24 · Cited by 784
Cited by
Chatbots in Education and Research: A Critical Examination of Ethical Implications and Solutions
Kooli, C. (2023) · Sustainability · doi:10.3390/su15075614 · Cited by 666
Nicolescu, L., & Tudorache, M. T. (2022) · Electronics · doi:10.3390/electronics11101579 · Cited by 280
Ali, F., Yasar, B., Ali, L., & Dogan, S. (2023) · International Journal of Hospitality Management · doi:10.1016/j.ijhm.2023.103588 · Cited by 185
Dippold, D. (2023) · Journal of Pragmatics · doi:10.1016/j.pragma.2022.12.004 · Cited by 37
Chang, T.-S., & Hsiao, W.-H. (2025) · Aslib Journal of Information Management · doi:10.1108/ajim-12-2023-0551 · Cited by 31
Metadata
- Authors
- Følstad, A., & Taylor, C.
- Venue
- Quality and User Experience
- Published
- 2021-08-20
- Volume
- 6
- Issue
- 1
- ISSN
- 2366-0139 (ISSN-L)
- 2366-0147
- OpenAlex
- W3195580646
- Cited by
- 97
- References
- 19
- FWCI
- 4.76
- Type
- journal article
- Language
- en
- Open access
- hybrid
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Investigating the user experience of customer service chatbot interaction: a framework for qualitative analysis of chatbot dialogues. https://app.cochat.ai/papers/10.1007/s41233-021-00046-5
- The paper
- Følstad, A., & Taylor, C. (2021). Investigating the user experience of customer service chatbot interaction: a framework for qualitative analysis of chatbot dialogues. Quality and User Experience. https://doi.org/10.1007/s41233-021-00046-5
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.