Evaluating and Informing the Design of Chatbots
The DOI matches its Crossref record; 32 of the 32 references it cites were found in a registry or an index.
Abstract
Text messaging-based conversational agents (CAs), popularly called chatbots, received significant attention in the last two years. However, chatbots are still in their nascent stage: They have a low penetration rate as 84% of the Internet users have not used a chatbot yet. Hence, understanding the usage patterns of first-time users can potentially inform and guide the design of future chatbots. In this paper, we report the findings of a study with 16 first-time chatbot users interacting with eight chatbots over multiple sessions on the Facebook Messenger platform. Analysis of chat logs and user interviews revealed that users preferred chatbots that provided either a 'human-like' natural language conversation ability, or an engaging experience that exploited the benefits of the familiar turn-based messaging interface. We conclude with implications to evolve the design of chatbots, such as: clarify chatbot capabilities, sustain conversation context, handle dialog failures, and end conversations gracefully.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Previous studies have suggested that chatbots should aim to sustain contexts to improve the dialogue efficiency [37].”
From Leveraging Large Language Models to Power Chatbots for Collecting User Self-Reported Data (2024) · cites this paper as [37]
In the citing paper
“There is also a growing base of research to inform design of chatbot interactions, whether this concerns conversational design [6], personalization of chatbots [69], the use of interactive elements in chatbots [57], or the use of social cues to indicate social status and capabilities [32].”
From Future directions for chatbot research: An interdisciplinary research agenda (2021) · cites this paper as [57]
Citations
Cites
Development of NASA-TLX (Task Load Index): Results of Empirical and Theoretical Research
(1988) · Advances in Psychology · doi:10.1016/s0166-4115(08)62386-9 · Cited by 15,003
Norman, D. (2016) · Vahlen eBooks · doi:10.15358/9783800648108 · Cited by 6,374
ELIZA—a computer program for the study of natural language communication between man and machine
Weizenbaum, J. (1966) · Communications of the ACM · doi:10.1145/365153.365168 · Cited by 4,452 · cited 1×
“The first chatbot emerged in 1966 from MIT, called ELIZA [44], which emulated a Rogerian psychotherapist.”
BackgroundELIZA — a computer program for the study of natural language communication between man and machine
Weizenbaum, J. (1983) · Communications of the ACM · doi:10.1145/357980.357991 · Cited by 2,738
Licklider, J. C. R. (1960) · IRE Transactions on Human Factors in Electronics · doi:10.1109/thfe2.1960.4503259 · Cited by 1,938
Embodied Conversational Agents
(2000) · The MIT Press eBooks · doi:10.7551/mitpress/2697.001.0001 · Cited by 1,799
Establishing and maintaining long-term human-computer relationships
Bickmore, T. W., & Picard, R. W. (2005) · ACM Transactions on Computer-Human Interaction · doi:10.1145/1067860.1067867 · Cited by 1,333
Luger, E., & Sellen, A. (2016) · Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems · doi:10.1145/2858036.2858288 · Cited by 1,176
Semantically Conditioned LSTM-based Natural Language Generation for Spoken Dialogue Systems
Wen, T.-H., Gasic, M., Mrkšić, N., Su, P.-H., Vandyke, D., & Young, S. (2015) · Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d15-1199 · Cited by 874
A diary study of task switching and interruptions
Czerwinski, M., Horvitz, E., & Wilhite, S. (2004) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/985692.985715 · Cited by 830
A Conversational Agent as Museum Guide – Design and Evaluation of a Real-World Application
Kopp, S., Gesellensetter, L., Krämer, N. C., & Wachsmuth, I. (2005) · Lecture Notes in Computer Science · doi:10.1007/11550617_28 · Cited by 393
Social Dialongue with Embodied Conversational Agents
Bickmore, T., & Cassell, J. (2005) · Text, Speech and Language Technology · doi:10.1007/1-4020-3933-6_2 · Cited by 324 · cited 1×
“Embodied CAs have a graphical front-end as opposed to a robotic body, and attempt to be human-like by employing non-verbal behaviors, such as gestures and expressions, in addition to speech [13, 29].”
BackgroundA review of large-vocabulary continuous-speech
Young, S. (1996) · IEEE Signal Processing Magazine · doi:10.1109/79.536824 · Cited by 216
Ravenclaw: dialog management using hierarchical task decomposition and an expectation agenda
Bohus, D., & Rudnicky, A. I. (2003) · 8th European Conference on Speech Communication and Technology (Eurospeech 2003) · doi:10.21437/eurospeech.2003-255 · Cited by 195
Automatic Online Evaluation of Intelligent Assistants
Jiang, J., Hassan Awadallah, A., Jones, R., Ozertem, U., Zitouni, I., Gurunath Kulkarni, R., & Khan, O. Z. (2015) · Proceedings of the 24th International Conference on World Wide Web · doi:10.1145/2736277.2741669 · Cited by 148
Close Engagements with Artificial Companions: Key social, psychological, ethical and design issues
(2010) · Natural Language Processing · doi:10.1075/nlp.8 · Cited by 130
Semantically Conditioned LSTM-based Natural Language Generation for Spoken Dialogue SystemsPreprint
Wen, T.-H., Gašić, M., Mrkšić, N., et al. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1508.01745 · Cited by 124
Liao, Q. V., Mas-ud Hussain, M., Chandar, P., Davis, M., Khazaeni, Y., Crasso, M. P., Wang, D., Muller, M., Shami, N. S., & Geyer, W. (2018) · Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3173577 · Cited by 116 · cited 9×
“Recent works [28, 30, 31, 33, 42] evaluated CAs from an HCI perspective.”
BackgroundResultInfluentialDARPA communicator dialog travel planning systems: the june 2000 data collection
Walker, M., Aberdeen, J., Boland, J., Bratt, E., Garofolo, J., Hirschman, L., Le, A., Lee, S., Narayanan, S., Papineni, K., et al. (2001) · 7th European Conference on Speech Communication and Technology (Eurospeech 2001) · doi:10.21437/eurospeech.2001-355 · Cited by 109
How Do You Want Your Chatbot? An Exploratory Wizard-of-Oz Study with Young, Urban Indians
Medhi Thies, I., Menon, N., Magapu, S., Subramony, M., & O’Neill, J. (2017) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-67744-6_28 · Cited by 104
Liao, Q. V., Davis, M., Geyer, W., Muller, M., & Shami, N. S. (2016) · Proceedings of the 2016 ACM Conference on Designing Interactive Systems · doi:10.1145/2901790.2901842 · Cited by 92
Rosenfeld, R., Olsen, D., & Rudnicky, A. (2001) · Interactions · doi:10.1145/384076.384085 · Cited by 83
Convey: Exploring the Use of a Context View for Chatbots
Jain, M., Kota, R., Kumar, P., & Patel, S. N. (2018) · Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3174042 · Cited by 74
Self-citationA comparison between Alice and Elizabeth chatbot systems
Shawar, B. A., & Atwell, E. (2002) · White Rose Research Online (University of Leeds, The University of Sheffield, University of York) · Cited by 59
What would you Ask a conversational Agent? Observations of Human-Agent Dialogues in a Museum Setting
Robinson, S. M., Traum, D. R., Ittycheriah, M., & Henderer, J. (2008) · Language Resources and Evaluation Conference (LREC) · doi:10.63317/3bs9odrfj3mf · Cited by 57
"Oh dear stacy!": social interaction, elaboration, and learning with teachable agents
Ogan, A., Finkelstein, S., Mayfield, E., D'Adamo, C., Matsuda, N., & Cassell, J. (2012) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/2207676.2207684 · Cited by 57
How to Pass the Turing Test by Cheating
Hutchens, J. L. (1997) · Cited by 51
Embodied Conversational Agents
Huang, H. (2017) · Wiley eBooks · doi:10.1002/9781118976005.ch26 · Cited by 15
Wen, T.-H., Gašić, M., Mrkšić, N., Su, P.-H., Vandyke, D., & Young, S. J. (2015) · Apollo (University of Cambridge) · Cited by 11
Dialogues in Context: An Objective User-Oriented Evaluation Approach for Virtual Human Dialogue
Robinson, S. M., Roque, A., & Traum, D. R. (2010) · Language Resources and Evaluation Conference (LREC) · doi:10.63317/44fxd5anv8uv · Cited by 7
Talking Eye: autonomous creature as accomplice for human
Suzuki, N., Ishii, K., & Okada, M. (2002) · Asia-Pacific Computer and Human Interaction · doi:10.1109/apchi.1998.704479 · Cited by 7
Proceedings. 3rd Asia Pacific Computer Human Interaction (Cat. No.98EX110)
(1998) · Asia-Pacific Computer and Human Interaction · doi:10.1109/apchi.1998.704118 · Cited by 3
Research works printed but not linked: 16
- 2002. A.L.I.C. E. Foundation website. (2002). Retrieved January 4, 2017 from http://alicebot.org 2002. A.L.I.C.E. Foundation website. (2002). Retrieved January 4, 2017 from http://alicebot.org
- 2013. Mitsuku. ( 2013 ). Retrieved January 4, 2017 from http://www.mitsuku.com 2013. Mitsuku. (2013). Retrieved January 4, 2017 from http://www.mitsuku.com
- 2013. Rose. ( 2013 ). Retrieved January 4, 2017 from http://brilligunderstanding.com/rosedemo.html 2013. Rose. (2013). Retrieved January 4, 2017 from http://brilligunderstanding.com/rosedemo.html
- 2016. Facebook Messenger bots. (2016). Retrieved Dec 1, 2016 from https://chatbottle.co/bots/messenger 2016. Facebook Messenger bots. (2016). Retrieved Dec 1, 2016 from https://chatbottle.co/bots/messenger
- 2017. Facebook Messenger Alterra. (2017) . Retrieved January 30, 2017 from https://www.messenger.com/t/alterra.cc 2017. Facebook Messenger Alterra. (2017). Retrieved January 30, 2017 from https://www.messenger.com/t/alterra.cc
- 2017. Facebook Messenger Call of Duty. (2017) . Retrieved January 30, 2017 from https://www.messenger.com/t/CallofDuty 2017. Facebook Messenger Call of Duty. (2017). Retrieved January 30, 2017 from https://www.messenger.com/t/CallofDuty
- 2017. Facebook Messenger chatShopper. (2017) . Retrieved January 30, 2017 from https://www.messenger.com/t/chatShopper 2017. Facebook Messenger chatShopper. (2017). Retrieved January 30, 2017 from https://www.messenger.com/t/chatShopper
- 2017. Facebook Messenger CNN. (2017) . Retrieved January 30, 2017 from https://www.messenger.com/t/cnn 2017. Facebook Messenger CNN. (2017). Retrieved January 30, 2017 from https://www.messenger.com/t/cnn
- 2017. Facebook Messenger Hi Poncho. (2017) . Retrieved January 30, 2017 from https://www.messenger.com/t/hiponcho 2017. Facebook Messenger Hi Poncho. (2017). Retrieved January 30, 2017 from https://www.messenger.com/t/hiponcho
- 2017. Facebook Messenger Pandorabots. (2017) . Retrieved January 30, 2017 from https://www.messenger.com/t/chatbots.io 2017. Facebook Messenger Pandorabots. (2017). Retrieved January 30, 2017 from https://www.messenger.com/t/chatbots.io
- 2017. Facebook Messenger Swelly. (2017) . Retrieved January 30, 2017 from https://www.messenger.com/t/swell.bot 2017. Facebook Messenger Swelly. (2017). Retrieved January 30, 2017 from https://www.messenger.com/t/swell.bot
- 2017. Facebook Messenger Trivia Blast. (2017) . Retrieved January 30, 2017 from https://www.messenger.com/t/triviablast1 2017. Facebook Messenger Trivia Blast. (2017). Retrieved January 30, 2017 from https://www.messenger.com/t/triviablast1
- Susan Brennan . 1990. Conversation as Direct Manipulation: An Iconoclastic View. The Art of Human-Computer Interface Design ( 1990 ). Susan Brennan. 1990. Conversation as Direct Manipulation: An Iconoclastic View. The Art of Human-Computer Interface Design (1990).
- Orange Hive . 2017. First time bot users deserve good bots. (2017). Retrieved Jan 5, 2018 from https://unfiltered.orangehive.de/ first-time-bot-users-deserve-good-bots/ Orange Hive. 2017. First time bot users deserve good bots. (2017). Retrieved Jan 5, 2018 from https://unfiltered.orangehive.de/ first-time-bot-users-deserve-good-bots/
- J. C. R. Licklider . 1960. IRE Transactions on Human Factors in Electronics HFE-1 (March 1960 ), 4--11. J. C. R. Licklider. 1960. IRE Transactions on Human Factors in Electronics HFE-1 (March 1960), 4--11.
- Statista. 2017. Most popular global mobile messenger apps as of January 2017 . (2017). Retrieved February 7, 2017 from https://www.statista.com/statistics/258749/ most-popular-global-mobile-messenger-apps/ Statista. 2017. Most popular global mobile messenger apps as of January 2017. (2017). Retrieved February 7, 2017 from https://www.statista.com/statistics/258749/ most-popular-global-mobile-messenger-apps/
Cited by
Rapp, A., Curti, L., & Boldi, A. (2021) · International Journal of Human-Computer Studies · doi:10.1016/j.ijhcs.2021.102630 · Cited by 643
Chaves, A. P., & Gerosa, M. A. (2021) · International Journal of Human–Computer Interaction · doi:10.1080/10447318.2020.1841438 · Cited by 572
“Therefore, conversational Intelligence refers to the ability of a chatbot to effectively converse beyond the technical capability of achieving a conversational goal [Jain et al. 2018b].”
BackgroundMethodResultInfluentialWhat makes you continuously use chatbot services? Evidence from chinese online travel agencies
Li, L., Lee, K. Y., Emokpae, E., & Yang, S.-B. (2021) · Electronic Markets · doi:10.1007/s12525-020-00454-z · Cited by 289
Chatbots, Humbots, and the Quest for Artificial General Intelligence
Grudin, J., & Jacques, R. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300439 · Cited by 275
“Jain et al. [13] enlisted 13 relatively tech - savvy first - time users to try each of eight Facebook Messenger chatbots daily for 3 days.”
BackgroundBorsci, S., Malizia, A., Schmettow, M., van der Velde, F., Tariverdiyeva, G., Balaji, D., & Chamberlain, A. (2022) · Personal and Ubiquitous Computing · doi:10.1007/s00779-021-01582-9 · Cited by 266
Metadata
- Authors
- Jain, M., Kumar, P., Kota, R., & Patel, S. N.
- Affiliations
- University of Washington (US)
- IBM (India) (IN)
- IBM (Canada) (CA)
- Venue
- Proceedings of the 2018 Designing Interactive Systems Conference
- Publisher
- ACM
- Published
- 2018-06-08
- Pages
- 895-906
- OpenAlex
- W2808216081
- Cited by
- 376
- References
- 32
- Citations by year
- 2026: 18 · 2025: 55 · 2024: 48 · 2023: 61 · 2022: 65 · 2021: 73 · 2020: 32 · 2019: 23 · 2018: 1
- FWCI
- 13.11
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- Designing Interactive Systems Conference (publishedVersion)
- Topics
- Social Sciences › Business, Management and Accounting › Marketing › AI in Service Interactions
- Physical Sciences › Computer Science › Artificial Intelligence › Speech and dialogue systems
- Social Sciences › Psychology › Social Psychology › Social Robot Interaction and HRI
- Keywords
- chatbots, chatbot design, first-time users, user behavior patterns, Facebook Messenger, text messaging, natural language conversation, conversation context, chat logs
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.