Understanding the user experience of customer service chatbots: An experimental study of chatbot interaction design
Open accessAbstract
Understanding the user experience of chatbots for customer service is essential to realize the potential of this technology. Such chatbots are typically designed for efficient and effective interactions, accentuating pragmatic quality, and there is a need to understand how to make these more pleasant and engaging, strengthening hedonic quality. One promising approach is to design for more humanlike chatbot interactions, that is, interactions resembling those of skilled customer service personnel. In a randomized experiment (n = 35) we investigated two chatbot interaction design features that may strengthen the impression of a humanlike character: (a) topic-led conversations, encouraging customer reflection, in contrast to task-led conversations, aiming for efficient goal completion, and (b) free text interaction, where users interact mainly using their own words, rather than button interaction, where users mainly interact through predefined answer alternatives. dependent variables were participant perceptions of anthropomorphism and social presence, two key concepts related to chatbot human likeness, in addition to pragmatic quality and hedonic quality. To further explore user perceptions of the interaction designs, the study also included semi-structured interviews. Topic-led conversations were found to strengthen anthropomorphism and hedonic quality. A similar effect was not found for free text interaction, reportedly due to lack in chatbot flexibility and adaptivity. Implications for theory and practice are suggested.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The expansion of conversational User Interfaces has witnessed remarkable growth in recent years, exploiting techniques such as natural language processing and Large Language Models (LLM).
“The expansion of conversational User Interfaces has witnessed remarkable growth in recent years (Haugeland et al. 2022), exploiting techniques such as natural language processing and Large Language Models (LLM).”
From AdaptUI: A Framework for the development of Adaptive User Interfaces in Smart Product-Service Systems (2024) · cites this paper as Haugeland et al. 2022
Section 2.2, “Chatbots” (page unavailable), reports substantial renewed interest in chatbots and links it to advances in natural language processing, supporting the growth claim. The paper does not mention large language models, so it does not support that part of the citing sentence. (LLM, conf=0.85)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
The Measurement of Observer Agreement for Categorical Data
Landis, J. R., & Koch, G. G. (1977) · Biometrics · doi:10.2307/2529310 · Cited by 81,148 · see citing works
(2002) · Cited by 13,211 · see citing works
ELIZA—a computer program for the study of natural language communication between man and machine
Weizenbaum, J. (1966) · Communications of the ACM · doi:10.1145/365153.365168 · Cited by 4,452 · see citing works
Machines and Mindlessness: Social Responses to Computers
Nass, C., & Moon, Y. (2000) · Journal of Social Issues · doi:10.1111/0022-4537.00153 · Cited by 4,022 · see citing works
Meuter, M. L., Bitner, M. J., Ostrom, A. L., & Brown, S. W., III (2005) · Journal of Marketing · doi:10.1509/jmkg.69.2.61.60759 · Cited by 1,741 · see citing works
Conversational agents in healthcare: a systematic review
Laranjo, L., Dunn, A. G., Tong, H. L., Kocaballi, A. B., Chen, J., Bashir, R., Surian, D., Gallego, B., Magrabi, F., Lau, A. Y. S., et al. (2018) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocy072 · Cited by 1,388 · see citing works
AI-based chatbots in customer service and their effects on user compliance
Adam, M., Wessel, M., & Benlian, A. (2021) · Electronic Markets · doi:10.1007/s12525-020-00414-7 · Cited by 1,195 · see citing works
Araujo, T. (2018) · Computers in Human Behavior · doi:10.1016/j.chb.2018.03.051 · Cited by 1,156 · see citing works
The Thing and I: Understanding the Relationship Between User and Product
Hassenzahl, M. (2003) · Human-Computer Interaction Series · doi:10.1007/1-4020-2967-5_4 · Cited by 1,097 · see citing works
Go, E., & Sundar, S. S. (2019) · Computers in Human Behavior · doi:10.1016/j.chb.2019.01.020 · Cited by 1,089 · see citing works
Managing the Total Customer Experience
Berry, L. L., Carbone, L. P., & Haeckel, S. H. (2002) · Dialnet (Universidad de la Rioja) · Cited by 1,054 · see citing works
Chatbot e-service and customer satisfaction regarding luxury brands
Chung, M., Ko, E., Joung, H., & Kim, S. J. (2018) · Journal of Business Research · doi:10.1016/j.jbusres.2018.10.004 · Cited by 1,005 · see citing works
Recipes for Building an Open-Domain Chatbot
Roller, S., Dinan, E., Goyal, N., Ju, D., Williamson, M., Liu, Y., Xu, J., Ott, M., Smith, E. M., Boureau, Y.-L., et al. (2021) · Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume · doi:10.18653/v1/2021.eacl-main.24 · Cited by 784 · see citing works
Towards a Human-like Open-Domain Chatbot
Adiwardana, D., Luong, M.-T., So, D. R., Hall, J., Fiedel, N., Thoppilan, R., Yang, Z., Kulshreshtha, A., Nemade, G., Lu, Y., et al. (2020) · arXiv · doi:10.48550/arxiv.2001.09977 · Cited by 614 · see citing works
Parasuraman, A., Berry, L. L., & Zeithaml, V. A. (1991) · Human Resource Management · doi:10.1002/hrm.3930300304 · Cited by 593 · see citing works
A Survey on Dialogue Systems: Recent Advances and New Frontiers
Chen, H., Liu, X., Yin, D., & Tang, J. (2017) · ACM SIGKDD Explorations Newsletter · doi:10.1145/3166054.3166058 · Cited by 579 · see citing works
Chatbots and the new world of HCI
Følstad, A., & Brandtzæg, P. B. (2017) · interactions · doi:10.1145/3085558 · Cited by 520 · see citing works
Self-citationCustomer experience management: a critical review of an emerging idea
Palmer, A. J. (2010) · Journal of Services Marketing · doi:10.1108/08876041011040604 · Cited by 501 · see citing works
Dale, R. (2016) · Natural Language Engineering · doi:10.1017/s1351324916000243 · Cited by 488 · see citing works
Rediscovering the use of chatbots in education: A systematic literature review
Pérez, J. Q., Daradoumis, T., & Puig, J. M. M. (2020) · Computer Applications in Engineering Education · doi:10.1002/cae.22326 · Cited by 483 · see citing works
User Experiences of Social Support From Companion Chatbots in Everyday Contexts: Thematic Analysis
Ta, V., Griffith, C., Boatfield, C., Wang, X., Civitello, M., Bader, H., DeCero, E., & Loggarakis, A. (2020) · Journal of Medical Internet Research · doi:10.2196/16235 · Cited by 461 · see citing works
Lee, S., & Choi, J. (2017) · International Journal of Human-Computer Studies · doi:10.1016/j.ijhcs.2017.02.005 · Cited by 388 · see citing works
Users' experiences with chatbots: findings from a questionnaire study
Følstad, A., & Brandtzæg, P. B. (2020) · Quality and User Experience · doi:10.1007/s41233-020-00033-2 · Cited by 280 · see citing works
Self-citationModelling user experience – An agenda for research and practice
Law, E. L., & van Schaik, P. (2010) · Interacting with Computers · doi:10.1016/j.intcom.2010.04.006 · Cited by 261 · see citing works
Conversational AI: The Science Behind the Alexa PrizePreprint
Ram, A., Prasad, R., Khatri, C., Venkatesh, A., Gabriel, R., Liu, Q., Nunn, J., Hedayatnia, B., Cheng, M., Nagar, A., et al. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1801.03604 · Cited by 261 · see citing works
An Initial Model of Trust in Chatbots for Customer Service—Findings from a Questionnaire Study
Nordheim, C. B., Følstad, A., & Bjørkli, C. A. (2019) · Interacting with Computers · doi:10.1093/iwc/iwz022 · Cited by 233 · see citing works
Self-citationThe art of service recovery: fact or fiction?
Hocutt, M. A., Bowers, M. R., & Donavan, D. T. (2006) · Journal of Services Marketing · doi:10.1108/08876040610665652 · Cited by 217 · see citing works
Liao, Q. V., Mas-ud Hussain, M., Chandar, P., Davis, M., Khazaeni, Y., Crasso, M. P., Wang, D., Muller, M., Shami, N. S., & Geyer, W. (2018) · Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3173577 · Cited by 116 · see citing works
von Wolff, R. M., Hobert, S., & Schümann, M. (2019) · Hawaii International Conference on System Sciences (HICSS) · doi:10.24251/hicss.2019.013 · Cited by 86 · see citing works
Chatbot Personalities Matters: Improving the User Experience of Chatbot Interfaces
Smestad, T. L., & Volden, F. (2019) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-17705-8_15 · Cited by 52 · see citing works
Studies in Conversational UX Design
(2018) · Human-computer interaction series · doi:10.1007/978-3-319-95579-7 · Cited by 51 · see citing works
Assessing the performance of short multi-item questionnaires in aesthetic evaluation of websites
Papachristos, E. (2018) · Behaviour and Information Technology · doi:10.1080/0144929x.2018.1539521 · Cited by 13 · see citing works
Flagged first, then by citations
Research works printed but not linked: 21
- Ashktorab, Z., Jain, M., Liao, Q.V., Weisz, J.D., 2019. Resilient chatbots: repair strategy preferences for conversational breakdowns. In: Proceedings of CHI 2019 (paper no. 254). New York, NY. ACM.
- Brandtzaeg, P.B., Følstad, A., 2017. Why people use chatbots. In: Proceedings of the International Conference on Internet Science -INSCI 2017. Cham, Switzerland. Springer, pp. 377-392.
- Diederich, S., Janssen-Müller, M., Brendel, A.B., Morana, S., 2019. Emulating empathetic behavior in online service encounters with sentiment-adaptive responses: insights from an experiment with a conversational agent. In: Proceedings of ICIS2019 (Paper no. 1560). AISEL.
- Følstad, A., Skjuve, M., 2019. Chatbots for customer service: user experience and motivation. In: Proceedings of Conversational User Interfaces -CUI 2019 (pp. 1-9). New York, NY. ACM.
- Følstad, A., Kvale, K., Haugstveit, I.M., 2014. Customer support as a source of usability insight: why users call support after visiting self-service websites. In: Proceedings of NordiCHI 2014. New York, NY. ACM, pp. 167-170.
- Hassenzahl, M., Burmester, M., Koller, F., 2003. AttrakDiff: ein Fragebogen zur Messung wahrgenommener hedonischer und pragmatischer Qualität. In: Szwillus, G., Ziegler, J. (Eds.), Mensch & Computer 2003: Interaktion in Bewegung. Vieweg+ Teubner Verlag, Wiesbaden, Germany, pp. 187-196.
- Jain, M., Kumar, P., Kota, R., Patel, S.N., 2018. Evaluating and informing the design of chatbots. In: Proceedings of the Designing Interactive Systems Conference -DIS 2018. New York,NY. ACM, pp. 895-906.
- Jenkins, M.-C., Churchill, R., Cox, S., Smith, D., 2007. Analysis of user interaction with service oriented chatbot systems. In: Proceedings of HCI 2007. Heidelberg, Germany. Springer, pp. 76-83.
- Kujala, S., Miron-Shatz, T., 2013. Emotions, experiences and usability in real-life mobile phone use. In: Proceedings of CHI 2013. New York, NY. ACM, pp. 1061-1070.
- Kvale, K., Sell, O.A., Hodnebrog, S., Følstad, A., 2020. Improving conversations: Lessons learnt from manual analysis of chatbot dialogues. In: Proceedings of Conversations 2019-International Workshop on Chatbot Research and Design. Cham, Switzerland. Springer, Cham, Switzerland, pp. 187-200.
- Laban, G., Araujo, T., 2020. Working together with conversational agents: The relationship of perceived cooperation with service performance evaluations. In: Proceedings of Conversations 2019-International Workshop on Chatbot Research and Design. Cham, Switzerland. Springer, Cham, Switzerland, pp. 215-228.
- Li, C.H., Yeh, S.F., Chang, T.J., Tsai, M.H., Chen, K., Chang, Y.J., 2020. A conversation analysis of non-progress and coping strategies with a banking task-oriented chatbot. In: Proceedings of CHI 2020 (paper no. 82). New York, NY. ACM.
- Liebrecht, C., van Hooijdonk, C., 2019. Creating humanlike chatbots: what chatbot developers could learn from webcare employees in adopting a conversational human voice. In: Proceedings of CONVERSATIONS 2019-International Workshop on Chatbot Research and Design. Cham, Switzerland. Springer, pp. 51-64.
- Schuetzler, R.M., Grimes, G.M., Giboney, J.S., 2018. An investigation of conversational agent relevance, presence, and engagement. In: Proceedings of ACIS 2018. AISEL.
- Schuetzler, R.M., Grimes, M., Giboney, J.S., Buckman, J., 2014. Facilitating natural conversational agent interactions: Lessons from a deception experiment. In: Information Systems and Quantitative Analysis Faculty Proceedings & Presentations. Retrieved from. In: https://digitalcommons.unomaha.edu/isqafacproc/16.
- Shevat, A., 2017. Designing bots: Creating Conversational Experiences. O'Reilly Media, Boston, MA.
- Smestad, T.L., Volden, F., 2018. Chatbot personalities matters. Internet Science. INSCI 2018. Springer, Cham, Switzerland, pp. 170-181.
- Valério, F.A., Guimarães, T.G., Prates, R.O., Candello, H., 2017. Here's what I can do: chatbots' strategies to convey their features to users. In: Proceedings of IHC'17 (article no. 28). New York, NY. ACM.
- Valério, F.A., Guimarães, T.G., Prates, R.O., Candello, H., 2020. Comparing users' perception of different chatbot interaction paradigms: a case study. In: Proceedings of IHC'20 (article no. 11). New York, NY. ACM.
- van der Goot, M., Hafkamp, L., Dankfort, Z., 2020. Customer service chatbots: a qualitative interview study into customers' communication journey. In: Proceedings of CONVERSATIONS 2020. Cham, Switzerland. Springer.
- Xu, A., Liu, Z., Guo, Y., Sinha, V., Akkiraju, R., 2017. A new chatbot for customer service on social media. In: Proceedings of CHI 2017. New York, NY. ACM, pp. 3506-3510.
Cited by
Sorted by citations
Learning to Prompt in the Classroom to Understand AI Limits: A Pilot Study
Theophilou, E., Koyutürk, C., Yavari, M., Bursic, S., Donabauer, G., Telari, A., Testa, A., Boiano, R., Hernandez-Leo, D., Ruskov, M., et al. (2023) · Lecture Notes in Computer Science · doi:10.1007/978-3-031-47546-7_33 · Cited by 58 · see citing works
Development and Validation of a Basic Psychological Needs Scale for Technology Use
Moradbakhti, L., Leichtmann, B., & Mara, M. (2024) · Psychological Test Adaptation and Development · doi:10.1027/2698-1866/a000062 · Cited by 41 · see citing works
Carrera-Rivera, A., Larrinaga, F., Lasa, G., Martinez-Arellano, G., & Unamuno, G. (2024) · User Modeling and User-Adapted Interaction · doi:10.1007/s11257-024-09414-0 · Cited by 30 · see citing works
Artificial intelligence in higher education: a PRISMA 2020 review
Jaboob, M., Al-Ansi, A. M., & Riyadh, H. A. (2026) · International Journal of Educational Management · doi:10.1108/ijem-09-2024-0560 · Cited by 10 · see citing works
Designing an AI Chatbot for Team-Based Diabetes Care: An Iterative Human-in-the-Loop Approach
Tseng, Y.-C., Chen, S., Mah, K.-H., & Chen, Y.-C. (2025) · Lecture Notes in Computer Science · doi:10.1007/978-3-031-93736-1_17 · Cited by 5 · see citing works
- Authors
- Haugeland, I. K. F., Følstad, A., Taylor, C., & Bjørkli, C. A.
- Affiliations
- University of Oslo (NO)
- SINTEF (NO)
- Corresponding author
- Følstad, Asbjørn
- Venue
- International Journal of Human-Computer Studies
- Publisher
- Elsevier BV
- Published
- 2022-05
- Volume
- 161
- Pages
- 102788
- ISSN
- 1071-5819 (ISSN-L)
- OpenAlex
- W4210890071
- Cited by
- 235
- References
- 33
- Citations by year
- 2026: 55 · 2025: 83 · 2024: 58 · 2023: 36 · 2022: 3
- FWCI
- 14.80
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- International Journal of Human-Computer Studies (publishedVersion, cc-by)
- NORA - Norwegian Open Research Archives (submittedVersion, cc-by)
- Funders
- Research Council of Norway
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › AI in Service Interactions
- Social Sciences › Psychology › Social Psychology › Social Robot Interaction and HRI
- Physical Sciences › Computer Science › Artificial Intelligence › Speech and dialogue systems
- Keywords
- customer service chatbots, user experience, anthropomorphism, social presence, hedonic quality, pragmatic quality
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
