Evaluating and Informing the Design of Chatbots
The DOI matches its Crossref record; 32 of the 32 references it cites were found in a registry or an index.
Abstract
Text messaging-based conversational agents (CAs), popularly called chatbots, received significant attention in the last two years. However, chatbots are still in their nascent stage: They have a low penetration rate as 84% of the Internet users have not used a chatbot yet. Hence, understanding the usage patterns of first-time users can potentially inform and guide the design of future chatbots. In this paper, we report the findings of a study with 16 first-time chatbot users interacting with eight chatbots over multiple sessions on the Facebook Messenger platform. Analysis of chat logs and user interviews revealed that users preferred chatbots that provided either a 'human-like' natural language conversation ability, or an engaging experience that exploited the benefits of the familiar turn-based messaging interface. We conclude with implications to evolve the design of chatbots, such as: clarify chatbot capabilities, sustain conversation context, handle dialog failures, and end conversations gracefully.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Previous studies have suggested that chatbots should aim to sustain contexts to improve the dialogue efficiency [37].”
From Leveraging Large Language Models to Power Chatbots for Collecting User Self-Reported Data (2024) · cites this paper as [37]
In the citing paper
“There is also a growing base of research to inform design of chatbot interactions, whether this concerns conversational design [6], personalization of chatbots [69], the use of interactive elements in chatbots [57], or the use of social cues to indicate social status and capabilities [32].”
From Future directions for chatbot research: An interdisciplinary research agenda (2021) · cites this paper as [57]
Citations
Cites
Development of NASA-TLX (Task Load Index): Results of Empirical and Theoretical Research
(1988) · Advances in Psychology · doi:10.1016/s0166-4115(08)62386-9 · Cited by 15,003
Norman, D. (2016) · Vahlen eBooks · doi:10.15358/9783800648108 · Cited by 6,374
ELIZA—a computer program for the study of natural language communication between man and machine
Weizenbaum, J. (1966) · Communications of the ACM · doi:10.1145/365153.365168 · Cited by 4,452 · cited 1×
“The first chatbot emerged in 1966 from MIT, called ELIZA [44], which emulated a Rogerian psychotherapist.”
BackgroundELIZA — a computer program for the study of natural language communication between man and machine
Weizenbaum, J. (1983) · Communications of the ACM · doi:10.1145/357980.357991 · Cited by 2,738
Licklider, J. C. R. (1960) · IRE Transactions on Human Factors in Electronics · doi:10.1109/thfe2.1960.4503259 · Cited by 1,938
Embodied Conversational Agents
(2000) · The MIT Press eBooks · doi:10.7551/mitpress/2697.001.0001 · Cited by 1,799
Establishing and maintaining long-term human-computer relationships
Bickmore, T. W., & Picard, R. W. (2005) · ACM Transactions on Computer-Human Interaction · doi:10.1145/1067860.1067867 · Cited by 1,333
Luger, E., & Sellen, A. (2016) · Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems · doi:10.1145/2858036.2858288 · Cited by 1,176
Semantically Conditioned LSTM-based Natural Language Generation for Spoken Dialogue Systems
Wen, T.-H., Gasic, M., Mrkšić, N., Su, P.-H., Vandyke, D., & Young, S. (2015) · Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d15-1199 · Cited by 874
A diary study of task switching and interruptions
Czerwinski, M., Horvitz, E., & Wilhite, S. (2004) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/985692.985715 · Cited by 830
Research works printed but not linked: 16
- 2002. A.L.I.C. E. Foundation website. (2002). Retrieved January 4, 2017 from http://alicebot.org 2002. A.L.I.C.E. Foundation website. (2002). Retrieved January 4, 2017 from http://alicebot.org
- 2013. Mitsuku. ( 2013 ). Retrieved January 4, 2017 from http://www.mitsuku.com 2013. Mitsuku. (2013). Retrieved January 4, 2017 from http://www.mitsuku.com
- 2013. Rose. ( 2013 ). Retrieved January 4, 2017 from http://brilligunderstanding.com/rosedemo.html 2013. Rose. (2013). Retrieved January 4, 2017 from http://brilligunderstanding.com/rosedemo.html
- 2016. Facebook Messenger bots. (2016). Retrieved Dec 1, 2016 from https://chatbottle.co/bots/messenger 2016. Facebook Messenger bots. (2016). Retrieved Dec 1, 2016 from https://chatbottle.co/bots/messenger
- 2017. Facebook Messenger Alterra. (2017) . Retrieved January 30, 2017 from https://www.messenger.com/t/alterra.cc 2017. Facebook Messenger Alterra. (2017). Retrieved January 30, 2017 from https://www.messenger.com/t/alterra.cc
- 2017. Facebook Messenger Call of Duty. (2017) . Retrieved January 30, 2017 from https://www.messenger.com/t/CallofDuty 2017. Facebook Messenger Call of Duty. (2017). Retrieved January 30, 2017 from https://www.messenger.com/t/CallofDuty
- 2017. Facebook Messenger chatShopper. (2017) . Retrieved January 30, 2017 from https://www.messenger.com/t/chatShopper 2017. Facebook Messenger chatShopper. (2017). Retrieved January 30, 2017 from https://www.messenger.com/t/chatShopper
- 2017. Facebook Messenger CNN. (2017) . Retrieved January 30, 2017 from https://www.messenger.com/t/cnn 2017. Facebook Messenger CNN. (2017). Retrieved January 30, 2017 from https://www.messenger.com/t/cnn
- 2017. Facebook Messenger Hi Poncho. (2017) . Retrieved January 30, 2017 from https://www.messenger.com/t/hiponcho 2017. Facebook Messenger Hi Poncho. (2017). Retrieved January 30, 2017 from https://www.messenger.com/t/hiponcho
- 2017. Facebook Messenger Pandorabots. (2017) . Retrieved January 30, 2017 from https://www.messenger.com/t/chatbots.io 2017. Facebook Messenger Pandorabots. (2017). Retrieved January 30, 2017 from https://www.messenger.com/t/chatbots.io
- 2017. Facebook Messenger Swelly. (2017) . Retrieved January 30, 2017 from https://www.messenger.com/t/swell.bot 2017. Facebook Messenger Swelly. (2017). Retrieved January 30, 2017 from https://www.messenger.com/t/swell.bot
- 2017. Facebook Messenger Trivia Blast. (2017) . Retrieved January 30, 2017 from https://www.messenger.com/t/triviablast1 2017. Facebook Messenger Trivia Blast. (2017). Retrieved January 30, 2017 from https://www.messenger.com/t/triviablast1
- Susan Brennan . 1990. Conversation as Direct Manipulation: An Iconoclastic View. The Art of Human-Computer Interface Design ( 1990 ). Susan Brennan. 1990. Conversation as Direct Manipulation: An Iconoclastic View. The Art of Human-Computer Interface Design (1990).
- Orange Hive . 2017. First time bot users deserve good bots. (2017). Retrieved Jan 5, 2018 from https://unfiltered.orangehive.de/ first-time-bot-users-deserve-good-bots/ Orange Hive. 2017. First time bot users deserve good bots. (2017). Retrieved Jan 5, 2018 from https://unfiltered.orangehive.de/ first-time-bot-users-deserve-good-bots/
- J. C. R. Licklider . 1960. IRE Transactions on Human Factors in Electronics HFE-1 (March 1960 ), 4--11. J. C. R. Licklider. 1960. IRE Transactions on Human Factors in Electronics HFE-1 (March 1960), 4--11.
- Statista. 2017. Most popular global mobile messenger apps as of January 2017 . (2017). Retrieved February 7, 2017 from https://www.statista.com/statistics/258749/ most-popular-global-mobile-messenger-apps/ Statista. 2017. Most popular global mobile messenger apps as of January 2017. (2017). Retrieved February 7, 2017 from https://www.statista.com/statistics/258749/ most-popular-global-mobile-messenger-apps/
Cited by
Rapp, A., Curti, L., & Boldi, A. (2021) · International Journal of Human-Computer Studies · doi:10.1016/j.ijhcs.2021.102630 · Cited by 643
Chaves, A. P., & Gerosa, M. A. (2021) · International Journal of Human–Computer Interaction · doi:10.1080/10447318.2020.1841438 · Cited by 572
“Therefore, conversational Intelligence refers to the ability of a chatbot to effectively converse beyond the technical capability of achieving a conversational goal [Jain et al. 2018b].”
BackgroundMethodResultInfluentialWhat makes you continuously use chatbot services? Evidence from chinese online travel agencies
Li, L., Lee, K. Y., Emokpae, E., & Yang, S.-B. (2021) · Electronic Markets · doi:10.1007/s12525-020-00454-z · Cited by 289
Chatbots, Humbots, and the Quest for Artificial General Intelligence
Grudin, J., & Jacques, R. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300439 · Cited by 275
“Jain et al. [13] enlisted 13 relatively tech - savvy first - time users to try each of eight Facebook Messenger chatbots daily for 3 days.”
BackgroundBorsci, S., Malizia, A., Schmettow, M., van der Velde, F., Tariverdiyeva, G., Balaji, D., & Chamberlain, A. (2022) · Personal and Ubiquitous Computing · doi:10.1007/s00779-021-01582-9 · Cited by 266
Metadata
- Authors
- Jain, M., Kumar, P., Kota, R., & Patel, S. N.
- Affiliations
- University of Washington (US)
- IBM (India) (IN)
- IBM (Canada) (CA)
- Venue
- Proceedings of the 2018 Designing Interactive Systems Conference
- Publisher
- ACM
- Published
- 2018-06-08
- Pages
- 895-906
- OpenAlex
- W2808216081
- Cited by
- 376
- References
- 32
- Citations by year
- 2026: 18 · 2025: 55 · 2024: 48 · 2023: 61 · 2022: 65 · 2021: 73 · 2020: 32 · 2019: 23 · 2018: 1
- FWCI
- 13.11
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- Designing Interactive Systems Conference (publishedVersion)
- Topics
- Social Sciences › Business, Management and Accounting › Marketing › AI in Service Interactions
- Physical Sciences › Computer Science › Artificial Intelligence › Speech and dialogue systems
- Social Sciences › Psychology › Social Psychology › Social Robot Interaction and HRI
- Keywords
- chatbots, chatbot design, first-time users, user behavior patterns, Facebook Messenger, text messaging, natural language conversation, conversation context, chat logs
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Evaluating and Informing the Design of Chatbots. https://app.cochat.ai/papers/10.1145/3196709.3196735
- The paper
- Jain, M., Kumar, P., Kota, R., & Patel, S. N. (2018). Evaluating and Informing the Design of Chatbots. Proceedings of the 2018 Designing Interactive Systems Conference. https://doi.org/10.1145/3196709.3196735
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.