Evaluating and Informing the Design of Chatbots
The DOI matches its Crossref record; 32 of the 32 references it cites were found in a registry or an index.
Abstract
Text messaging-based conversational agents (CAs), popularly called chatbots, received significant attention in the last two years. However, chatbots are still in their nascent stage: They have a low penetration rate as 84% of the Internet users have not used a chatbot yet. Hence, understanding the usage patterns of first-time users can potentially inform and guide the design of future chatbots. In this paper, we report the findings of a study with 16 first-time chatbot users interacting with eight chatbots over multiple sessions on the Facebook Messenger platform. Analysis of chat logs and user interviews revealed that users preferred chatbots that provided either a 'human-like' natural language conversation ability, or an engaging experience that exploited the benefits of the familiar turn-based messaging interface. We conclude with implications to evolve the design of chatbots, such as: clarify chatbot capabilities, sustain conversation context, handle dialog failures, and end conversations gracefully.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Previous studies have suggested that chatbots should aim to sustain contexts to improve the dialogue efficiency [37].”
From Leveraging Large Language Models to Power Chatbots for Collecting User Self-Reported Data (2024) · cites this paper as [37]
In the citing paper
“There is also a growing base of research to inform design of chatbot interactions, whether this concerns conversational design [6], personalization of chatbots [69], the use of interactive elements in chatbots [57], or the use of social cues to indicate social status and capabilities [32].”
From Future directions for chatbot research: An interdisciplinary research agenda (2021) · cites this paper as [57]
Citations
Cites
Development of NASA-TLX (Task Load Index): Results of Empirical and Theoretical Research
(1988) · Advances in Psychology · doi:10.1016/s0166-4115(08)62386-9 · Cited by 15,003
Norman, D. (2016) · Vahlen eBooks · doi:10.15358/9783800648108 · Cited by 6,374
ELIZA—a computer program for the study of natural language communication between man and machine
Weizenbaum, J. (1966) · Communications of the ACM · doi:10.1145/365153.365168 · Cited by 4,452 · cited 1×
“The first chatbot emerged in 1966 from MIT, called ELIZA [44], which emulated a Rogerian psychotherapist.”
BackgroundELIZA — a computer program for the study of natural language communication between man and machine
Weizenbaum, J. (1983) · Communications of the ACM · doi:10.1145/357980.357991 · Cited by 2,738
Licklider, J. C. R. (1960) · IRE Transactions on Human Factors in Electronics · doi:10.1109/thfe2.1960.4503259 · Cited by 1,938
Embodied Conversational Agents
(2000) · The MIT Press eBooks · doi:10.7551/mitpress/2697.001.0001 · Cited by 1,799
Establishing and maintaining long-term human-computer relationships
Bickmore, T. W., & Picard, R. W. (2005) · ACM Transactions on Computer-Human Interaction · doi:10.1145/1067860.1067867 · Cited by 1,333
Luger, E., & Sellen, A. (2016) · Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems · doi:10.1145/2858036.2858288 · Cited by 1,176
Semantically Conditioned LSTM-based Natural Language Generation for Spoken Dialogue Systems
Wen, T.-H., Gasic, M., Mrkšić, N., Su, P.-H., Vandyke, D., & Young, S. (2015) · Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d15-1199 · Cited by 874
A diary study of task switching and interruptions
Czerwinski, M., Horvitz, E., & Wilhite, S. (2004) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/985692.985715 · Cited by 830
Research works printed but not linked: 16
- 2002. A.L.I.C. E. Foundation website. (2002). Retrieved January 4, 2017 from http://alicebot.org 2002. A.L.I.C.E. Foundation website. (2002). Retrieved January 4, 2017 from http://alicebot.org
- 2013. Mitsuku. ( 2013 ). Retrieved January 4, 2017 from http://www.mitsuku.com 2013. Mitsuku. (2013). Retrieved January 4, 2017 from http://www.mitsuku.com
- 2013. Rose. ( 2013 ). Retrieved January 4, 2017 from http://brilligunderstanding.com/rosedemo.html 2013. Rose. (2013). Retrieved January 4, 2017 from http://brilligunderstanding.com/rosedemo.html
- 2016. Facebook Messenger bots. (2016). Retrieved Dec 1, 2016 from https://chatbottle.co/bots/messenger 2016. Facebook Messenger bots. (2016). Retrieved Dec 1, 2016 from https://chatbottle.co/bots/messenger
- 2017. Facebook Messenger Alterra. (2017) . Retrieved January 30, 2017 from https://www.messenger.com/t/alterra.cc 2017. Facebook Messenger Alterra. (2017). Retrieved January 30, 2017 from https://www.messenger.com/t/alterra.cc
- 2017. Facebook Messenger Call of Duty. (2017) . Retrieved January 30, 2017 from https://www.messenger.com/t/CallofDuty 2017. Facebook Messenger Call of Duty. (2017). Retrieved January 30, 2017 from https://www.messenger.com/t/CallofDuty
- 2017. Facebook Messenger chatShopper. (2017) . Retrieved January 30, 2017 from https://www.messenger.com/t/chatShopper 2017. Facebook Messenger chatShopper. (2017). Retrieved January 30, 2017 from https://www.messenger.com/t/chatShopper
- 2017. Facebook Messenger CNN. (2017) . Retrieved January 30, 2017 from https://www.messenger.com/t/cnn 2017. Facebook Messenger CNN. (2017). Retrieved January 30, 2017 from https://www.messenger.com/t/cnn
- 2017. Facebook Messenger Hi Poncho. (2017) . Retrieved January 30, 2017 from https://www.messenger.com/t/hiponcho 2017. Facebook Messenger Hi Poncho. (2017). Retrieved January 30, 2017 from https://www.messenger.com/t/hiponcho
- 2017. Facebook Messenger Pandorabots. (2017) . Retrieved January 30, 2017 from https://www.messenger.com/t/chatbots.io 2017. Facebook Messenger Pandorabots. (2017). Retrieved January 30, 2017 from https://www.messenger.com/t/chatbots.io
- 2017. Facebook Messenger Swelly. (2017) . Retrieved January 30, 2017 from https://www.messenger.com/t/swell.bot 2017. Facebook Messenger Swelly. (2017). Retrieved January 30, 2017 from https://www.messenger.com/t/swell.bot
- 2017. Facebook Messenger Trivia Blast. (2017) . Retrieved January 30, 2017 from https://www.messenger.com/t/triviablast1 2017. Facebook Messenger Trivia Blast. (2017). Retrieved January 30, 2017 from https://www.messenger.com/t/triviablast1
- Susan Brennan . 1990. Conversation as Direct Manipulation: An Iconoclastic View. The Art of Human-Computer Interface Design ( 1990 ). Susan Brennan. 1990. Conversation as Direct Manipulation: An Iconoclastic View. The Art of Human-Computer Interface Design (1990).
- Orange Hive . 2017. First time bot users deserve good bots. (2017). Retrieved Jan 5, 2018 from https://unfiltered.orangehive.de/ first-time-bot-users-deserve-good-bots/ Orange Hive. 2017. First time bot users deserve good bots. (2017). Retrieved Jan 5, 2018 from https://unfiltered.orangehive.de/ first-time-bot-users-deserve-good-bots/
- J. C. R. Licklider . 1960. IRE Transactions on Human Factors in Electronics HFE-1 (March 1960 ), 4--11. J. C. R. Licklider. 1960. IRE Transactions on Human Factors in Electronics HFE-1 (March 1960), 4--11.
- Statista. 2017. Most popular global mobile messenger apps as of January 2017 . (2017). Retrieved February 7, 2017 from https://www.statista.com/statistics/258749/ most-popular-global-mobile-messenger-apps/ Statista. 2017. Most popular global mobile messenger apps as of January 2017. (2017). Retrieved February 7, 2017 from https://www.statista.com/statistics/258749/ most-popular-global-mobile-messenger-apps/
Cited by
Items 401–420 of 420
Loneliness and social robots : acceptability and trust
Zehnder, E. (2023) · theses.fr (ABES) · doi:10.70675/f0213dfbz8034z4686zb1bfze0e49ca4d52e
Human-Machine Dialogue in the Medical Field. Using Dialog to Collect Important Patient Information
Liednikova, A. (2022) · theses.fr (ABES) · doi:10.70675/b572f8cbz5876z4683z828bzfb09085ddd78 · Cited by 1
Développer des machines parlantes : les chatbots à l’épreuve du marché
Esteban, C. (2022) · theses.fr (ABES) · doi:10.70675/5d8e685cz5a3bz4f85z87e6z5a514d3e4a1f
gentiletti, M. G. (2022) · theses.fr (ABES) · doi:10.70675/f5c153fbza23fz44e1zafe9z47680aed9e5a · Cited by 2
Guidance Frequency in Task-Oriented Chatbots: An Empirical Study Based on Predicted Abandonment
Hsu, C., Tung, H.-C., Chiu, H.-L., Lin, C.-T., & Chang, Y.-J. (2025) · International Conference on Human-Engaged Computing (ICHEC) · doi:10.1145/3786995.3787038
Sen, B., & Lee, Y.-C. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2606.19286
AI Applications and Organizational Sustainability: Investment and Regulation in Telecommunications
Aljibori, H. R., & Al-shamma, M. A. (2026) · IGI Global eBooks · doi:10.4018/979-8-3373-8964-6.ch013
Traubinger, V., Heil, S., Mithari, A. A., & Gaedke, M. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-26717-7_4
Older Adults’ Perceptions of an LLM-based Music Chatbot App
Aziz, F., Law, E. L., & Correa-Delval, M. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-26717-7_11
Assessing Conversational AI Usability: A System Usability Scale Evaluation of Dola AI
Kurniasari, A., & Velisya, N. A. (2026) · Jurnal Ilmiah Teknik · doi:10.56127/juit.v5i2.1324
Chen, X., Ding, S., Hou, S., & Chai, C. (2025) · International Symposium on Computational Intelligence and Design · doi:10.1109/iscid68789.2025.00026
Predictive Analysis of User Experience in Conversational Design Applications
Venugopal, K. (2026) · Advances in computational intelligence and robotics book series · doi:10.4018/979-8-2600-0512-5.ch004
Navigating Menstrual Hygiene Awareness Through Conversational UX in Pakistan
Azhar, F., & Khan, K. L. (2026) · Communication Design Quarterly · doi:10.1145/3794927.3794932
Desenvolvimento de um chatbot sobre a Amazônia Azul
Carlos, G. O. (2021) · doi:10.11606/003132185
User Perception of Chatbots in a Mobile Commerce Environment
de Andrade, N. A. (2026) · International Journal of Business Marketing · doi:10.18568/ijbmkt.10.2.325
A Survey on Generative AI Chatbots in Customer Service: Challenges, Benefits, and Use Cases
Chilakapati, V. K., Keshireddy, S. R., Nagumotu, V. T., Kavuluri, H. V. R., Pathani, A. K., & Dasari, A. (2026) · Journal of Data Analysis and Critical Management · doi:10.64235/8xx2mg62 · Cited by 3
Tantikanedee, C., & Chooprayoon, V. (2021) · TLA Research Journal
The use of artificial intelligence in interviewing witnesses and victims of criminal offences
Vesel, T. (2026) · University of Maribor digital library (University of Maribor)
Developing an intelligent online counselling framework for an employee health and wellness programme
Kanes, I. (2025) · Boloka Institutional Repository (North-west University)
Metadata
- Authors
- Jain, M., Kumar, P., Kota, R., & Patel, S. N.
- Affiliations
- University of Washington (US)
- IBM (India) (IN)
- IBM (Canada) (CA)
- Venue
- Proceedings of the 2018 Designing Interactive Systems Conference
- Publisher
- ACM
- Published
- 2018-06-08
- Pages
- 895-906
- OpenAlex
- W2808216081
- Cited by
- 376
- References
- 32
- Citations by year
- 2026: 18 · 2025: 55 · 2024: 48 · 2023: 61 · 2022: 65 · 2021: 73 · 2020: 32 · 2019: 23 · 2018: 1
- FWCI
- 13.11
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- Designing Interactive Systems Conference (publishedVersion)
- Topics
- Social Sciences › Business, Management and Accounting › Marketing › AI in Service Interactions
- Physical Sciences › Computer Science › Artificial Intelligence › Speech and dialogue systems
- Social Sciences › Psychology › Social Psychology › Social Robot Interaction and HRI
- Keywords
- chatbots, chatbot design, first-time users, user behavior patterns, Facebook Messenger, text messaging, natural language conversation, conversation context, chat logs
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.