"Like Having a Really Bad PA": The Gulf between User Expectation and Experience of Conversational Agents
The DOI matches its Crossref record; 49 of the 51 references it cites were found in a registry or an index.
Abstract
The past four years have seen the rise of conversational agents (CAs) in everyday life. Apple, Microsoft, Amazon, Google and Facebook have all embedded proprietary CAs within their software and, increasingly, conversation is becoming a key mode of human-computer interaction. Whilst we have long been familiar with the notion of computers that speak, the investigative concern within HCI has been upon multimodality rather than dialogue alone, and there is no sense of how such interfaces are used in everyday life. This paper reports the findings of interviews with 14 users of CAs in an effort to understand the current interactional factors affecting everyday use. We find user expectations dramatically out of step with the operation of the systems, particularly in terms of known machine intelligence, system capability and goals. Using Norman's 'gulfs of execution and evaluation' [30] we consider the implications of these findings for the design of future systems.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Whereas, we found that the challenges our participants experienced were more elementary.”
From Exploring How Older Adults Use a Smart Speaker–Based Voice Assistant in Their First Interactions: Qualitative Study (2021) · cites this paper as [32]
In the citing paper
“In addition, it could be more important for novice users to have an appropriate mental model and understand the system rather than an efficient user experience [32].”
From Exploring How Older Adults Use a Smart Speaker–Based Voice Assistant in Their First Interactions: Qualitative Study (2021) · cites this paper as [32]
In the citing paper
“human-AI mixed-initiative presents a generally valuable and underexplored interface paradigm.”
From Mixed-Initiative Creative Interfaces (2017) · cites this paper as [20]
In the citing paper
“Not only do these breakdowns significantly diminish user satisfaction and damage trust, they also decrease the likelihood of continued use [18, 44].”
From System and User Strategies to Repair Conversational Breakdowns of Spoken Dialogue Systems: A Scoping Review (2024) · cites this paper as [44]
In the citing paper
“This is because when voice assistants fail, such as by incorrectly answering a question, it derails user trust [27, 35].”
From A Mixed-Methods Approach to Understanding User Trust after Voice Assistant Failures (2023) · cites this paper as [35]
17 more claims about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
How Many Interviews Are Enough?: An Experiment with Data Saturation and Variability
Guest, G., Bunce, A., & Johnson, L. (2006) · Field Methods · doi:10.1177/1525822x05279903 · Cited by 20,808
(2010) · Sociology for Pharmacists · doi:10.4324/9780203381175_chapter_9 · Cited by 14,152
Norman, D. (2016) · Vahlen eBooks · doi:10.15358/9783800648108 · Cited by 6,374
Thematic networks: an analytic tool for qualitative research
Attride-Stirling, J. (2001) · Qualitative Research · doi:10.1177/146879410100100307 · Cited by 5,506 · cited 2×
“The thematic network is a ‘web-like coding structure’ [2; 385] that allows for a rich understanding of the conceptual interplay occurring within the narrative.”
Background(1997) · Choice Reviews Online · doi:10.5860/choice.34-3702 · Cited by 4,887
ELIZA—a computer program for the study of natural language communication between man and machine
Weizenbaum, J. (1966) · Communications of the ACM · doi:10.1145/365153.365168 · Cited by 4,452
ELIZA — a computer program for the study of natural language communication between man and machine
Weizenbaum, J. (1983) · Communications of the ACM · doi:10.1145/357980.357991 · Cited by 2,738
Licklider, J. C. R. (1960) · IRE Transactions on Human Factors in Electronics · doi:10.1109/thfe2.1960.4503259 · Cited by 1,938 · cited 1×
“When considering human-machine dialogue from an HCI perspective, some of the earliest work can be traced back to vision statements and position papers such as JCR Licklider (1960), ‘Man-machine symbiosis’ [25] and the early work of Richard Bolt [4].”
MethodEmbodied Conversational Agents
(2000) · The MIT Press eBooks · doi:10.7551/mitpress/2697.001.0001 · Cited by 1,799
Interaction Ritual: Essays on Face-to-face Behaviour
Goffman, E. (1972) · Medical Entomology and Zoology · Cited by 1,449
Research works printed but not linked: 14
- Boden Margaret A, 2007, Proceedings of Artificial Companions in Society: Perspectives on the Present and Future. Oxford Internet Institute.
- Davis Martin, Engines of Logic: Mathematicians and the Origin of the Computer
- Glass James R., 1999, Challenges for Spoken Dialogue Systems. In the proceedings of IEEE Workshop on Automatic Speech Recognition and Understanding. (ASRU), CO, USA
- 10.5555/1941863 · doi:10.5555/1941863
- Heuwinkel Kerstin, Your Virtual Butler: The Making-of. Robert Trappl (ed)
- Robnett Licklider Joseph Carl, 1960, HFE-1,(March
- Moore Roger, Spoken Language Processing: Where do we go from Here? In Your Virtual Butler: The Making-of. Robert Trappl (ed)
- Payr Sabine, Your Virtual Butler: The Making-of. Robert Trappl (ed)
- Preece Jenny, 2015, Interaction Design: Beyond Huma- Computer Interaction
- Pulman Stephen, Proceedings of the 2010 Workshop on Companionable Dialogue Systems. Assoc. for Computational Linguistics
- Trappl Robert, From Jeeves Jeannie to Siri, and Then? In Your Virtual Butler: The Making-of. Robert Trappl (ed)
- Vassallo Giorgio, Semantic Computing. Sheu, Yu, Ramamoorthy, Joshi & Zadeh.(eds)
- 10.5555/1870559.1870562 · doi:10.5555/1870559.1870562
- Wilks Yorick, 2010, Amsterdam: John Benjamins.
Cited by
Items 201–300 of 1,007
Siri, Echo and Performance: You have to Suffer Darling
Aylett, M. P., Cowan, B. R., & Clark, L. (2019) · Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290607.3310422 · Cited by 50
“…has become the norm because, firstly, engineers are too lazy to think of other ways of evaluating their systems (see [4, 9]), and secondly, designers and HCI professionals have largely ignored this interactive modality for years [1] and are only beginning to challenge this approach e.g[3, 7, 11].”
BackgroundPlayer-AI Interaction: What Neural Network Games Reveal About AI as Play
Zhu, J., Villareale, J., Javvaji, N., Risi, S., Löwe, M., Weigelt, R., & Harteveld, C. (2021) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445307 · Cited by 67
“For instance, in conversational agents, a widely adopted type of AI-infused products, researchers have reported the wide gap between user expectations and the user experience (UX) of these systems [51] and how users develop their own strategies to work around the obstacles [58].”
BackgroundImproving Conversations: Lessons Learnt from Manual Analysis of Chatbot Dialogues
Kvale, K., Sell, O. A., Hodnebrog, S., & Følstad, A. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-39540-7_13 · Cited by 51
Can a Machine Tend to Teenagers' Emotional Needs?: A Study with Conversational Agents
Kim, J., Kim, Y., Kim, B., Yun, S., Kim, M., & Lee, J. (2018) · Extended Abstracts of the 2018 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3170427.3188548 · Cited by 51
Designing PairBuddy—A Conversational Agent for Pair Programming
Robe, P., & Kuttal, S. K. (2022) · ACM Transactions on Computer-Human Interaction · doi:10.1145/3498326 · Cited by 44
“The resulting conversational breakdowns decrease users’ satisfaction, trust, and willingness to continue talking to conversational agents [113, 114, 152].”
BackgroundA Taxonomy of Linguistic Expressions That Contribute To Anthropomorphism of Language Technologies
DeVrio, A., Cheng, M., Egede, L., Olteanu, A., & Blodgett, S. L. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3714038 · Cited by 85
Liebrecht, C., & van Hooijdonk, C. M. J. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-39540-7_4 · Cited by 62
“Designing chatbots that resemble humans may easily lead to users making wrong assumptions regarding the chatbot capabilities (e.g., Luger and Sellen, 2016).”
BackgroundThe Role of Conversational Grounding in Supporting Symbiosis Between People and Digital Assistants
Cho, J., & Rader, E. (2020) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3392838 · Cited by 49
Sánchez‐Franco, M. J., Arenas‐Márquez, F. J., & Santos, M. A. D. (2021) · Journal of Retailing and Consumer Services · doi:10.1016/j.jretconser.2021.102658 · Cited by 52
CoPrompt: Supporting Prompt Sharing and Referring in Collaborative Natural Language Programming
Feng, L., Yen, R., You, Y., Fan, M., Zhao, J., & Lu, Z. (2024) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642212 · Cited by 61
Do Multilingual Users Prefer Chat-bots that Code-mix? Let's Nudge and Find Out!
Bawa, A., Khadpe, P., Joshi, P., Bali, K., & Choudhury, M. (2020) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3392846 · Cited by 82
“As conversational agents become prevalent with designers aiming to broaden their capabilities, research in the design of conversational systems has grown from developing better language generation and language understanding models to include more human-centered considerations [34, 41, 63].”
BackgroundResponse Quality in Human-Chatbot Collaborative Systems
Jiang, J., & Ahuja, N. (2020) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3397271.3401234 · Cited by 52
Sun, Y., Chen, J., Yao, B., Liu, J., Wang, D., Ma, X., Lu, Y., Xu, Y., & He, L. (2024) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3687035 · Cited by 71
Keep it Short: A Comparison of Voice Assistants’ Response Behavior
Haas, G., Rietzler, M., Jones, M., & Rukzio, E. (2022) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3491102.3517684 · Cited by 38
“Emulating human behavior in a VA raises the users expectations, as it implies human capabilities [27].”
BackgroundInfluentialEthical Considerations on Affective Computing: An Overview
Devillers, L. Y., & Cowie, R. (2023) · Proceedings of the IEEE · doi:10.1109/jproc.2023.3315217 · Cited by 57
“Luger and Sellen [49] gave a memorable description of using one: “like having a really bad PA.” Initial interactions, using humor, did generate positive feelings toward the system.”
BackgroundMildner, T., Cooney, O., Meck, A.-M., Bartl, M., Savino, G.-L., Doyle, P. R., Garaialde, D., Clark, L., Sloan, J., Wenig, N., et al. (2024) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642542 · Cited by 53
“In particular, the use of language, uttered or typed, as a primary input for CUIs poses problems regarding the ease with which users can explore their functionalities [9, 15, 96] or restricting the assessment of a system’s capabilities and boundaries [30, 47, 62].”
InfluentialPark, J., Min, B., Son, K., Song, J. Y., Ma, X., & Kim, J. (2026) · ACM International Conference on Intelligent User Interfaces (IUI) · doi:10.1145/3742413.3789107 · Cited by 53
“While conversational interfaces adaptively provide information through a multi-turn process, conversing requires the user to articulate their input clearly which, if not done, could result in misinterpreted user intent and irrelevant responses [7, 17, 28, 53, 79].”
Designing with Gaze: Tama -- a Gaze Activated Smart-Speaker
McMillan, D., Brown, B., Kawaguchi, I., Jaber, R., Solsona Belenguer, J., & Kuzuoka, H. (2019) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3359278 · Cited by 44
Designers Characterize Naturalness in Voice User Interfaces: Their Goals, Practices, and Challenges
Kim, Y., Reza, M., McGrenere, J., & Yoon, D. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445579 · Cited by 42
“In these studies, various issues have been explored, including how VUIs fit into everyday settings [88], how users perceive social and functional roles in conversation [28], and the disparity between high user expectations and low system capability [63].”
BackgroundInvestigating User Expectations on the Roles of Family-shared AI Speakers
Park, S., & Lim, Y.-K. (2020) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376450 · Cited by 54
Participatory Development and Pilot Testing of an Adolescent Health Promotion Chatbot
Maenhout, L., Peuters, C., Cardon, G., Compernolle, S., Crombez, G., & DeSmet, A. (2021) · Frontiers in Public Health · doi:10.3389/fpubh.2021.724779 · Cited by 54
“Two directions could be taken from here: (1) clarify the purpose of the chatbot to adolescents from the outset, for example by stating which topics it covers when greeting the user to avoid the expectation of small talk (35, 44, 46, 87) or (2) providing the opportunity for small talk in the chatbot.”
BackgroundWei, J., Tag, B., Trippas, J. R., Dingler, T., & Kostakos, V. (2022) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3491102.3517432 · Cited by 46
Yeh, S.-F., Wu, M.-H., Chen, T.-Y., Lin, Y.-C., Chang, X., Chiang, Y.-H., & Chang, Y.-J. (2022) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3491102.3501941 · Cited by 35
“Additionally, given that learnability is an important aspect of usability of an interactive system [60] and one that has previously been used for assessing whether a conversational user interface allows users to achieve successful interaction [52], we also measured learnability of the chatbot.”
BackgroundYuan, Y., Thompson, S., Watson, K., Chase, A., Senthilkumar, A., Brush, A. J. B., & Yarosh, S. (2019) · International Journal of Child-Computer Interaction · doi:10.1016/j.ijcci.2019.04.005 · Cited by 46
Social Media through Voice: Synthesized Voice Qualities and Self-presentation
Zhang, L., Jiang, L., Washington, N., Liu, A. A., Shao, J., Fourney, A., Morris, M. R., & Findlater, L. (2021) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3449235 · Cited by 48
“In fact, the increased level of humanness among smart speaker voices seems to introduce unrealistic expectations of these devices’ intellectual and emotional capabilities, juxtaposing their limitations [12, 37].”
BackgroundTalking with Conversational Agents in Collaborative Action
Porcheron, M., Fischer, J. E., McGregor, M., Brown, B., Luger, E., Candello, H., & O'Hara, K. (2017) · Companion of the 2017 ACM Conference on Computer Supported Cooperative Work and Social Computing · doi:10.1145/3022198.3022666 · Cited by 45
“However, recent research shows that despite grand promises made by manufacturers, existing commercial systems fail to meet users’ expectations [10].”
BackgroundSelf-citationNamvarpour, M. (., Pauwels, H., & Razi, A. (2025) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3757548 · Cited by 45
“Further, the rise of AI-driven interactions may create unrealistic user expectations, sometimes fostering dependency on these systems and undermining critical thinking [49].”
Edwards, J., Liu, H., Zhou, T., Gould, S. J. J., Clark, L., Doyle, P. R., & Cowan, B. R. (2019) · ACM Conference on Conversational User Interfaces (CUI) · doi:10.1145/3342775.3342785 · Cited by 47
“The questions reflect common tasks and requests made to IPAs [16, 19, 36, 41].”
BackgroundResultInfluentialCooking With Agents: Designing Context-aware Voice Interaction
Jaber, R., Zhong, S., Kuoppamäki, S., Hosseini, A., Gessinger, I., Brumby, D. P., Cowan, B. R., & Mcmillan, D. (2024) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642183 · Cited by 56
“Despite advancements in conversational agents, communicating with technology still difers fundamentally from human-to-human dialogue [48, 59].”
Let's Talk Futures: A Literature Review of HCI's Future Orientation
Sanchez, C., Wang, S., Savolainen, K., Epp, F. A., & Salovaara, A. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713759 · Cited by 48
Mutual Theory of Mind for Human-AI CommunicationPreprint
Wang, Q., & Goel, A. K. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2210.03842 · Cited by 57
Kuttal, S. K., Ong, B., Kwasny, K., & Robe, P. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445659 · Cited by 44
“To increase participants’ trust [10, 76, 77, 109], the agent showed uncertainty about its work and asked for verification.”
BackgroundPark, H., & Lee, J. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445133 · Cited by 44
Doyle, P. R., Gessinger, I., Edwards, J., Clark, L., Dumbleton, O., Garaialde, D., Rough, D. J., Bleakley, A., Branigan, H. P., & Cowan, B. R. (2025) · ACM Transactions on Computer-Human Interaction · doi:10.1145/3729170 · Cited by 49
“In addition to influencing a person’s language production in HMD [1, 13, 29], partner models are also said to influence broader interaction behaviours by informing expectations about the types of tasks a machine dialogue partner might be capable of executing [30, 34, 57].”
Voice Assistants for Health Self-Management: Designing for and with Older Adults
Mahmood, A., Cao, S., Stiber, M., Antony, V. N., & Huang, C. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713839 · Cited by 51
Assumptions Checked: How Families Learn About and Use the Echo Dot
Beneteau, E., Guan, Y., Richards, O. K., Zhang, M. R., Kientz, J. A., Yip, J., & Hiniker, A. (2020) · Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies · doi:10.1145/3380993 · Cited by 40
“voice interfaces to respond appropriately and communication breakdowns often result in abandonment and distrust that the device will work [2,8,11,19].”
BackgroundInfluentialConversational Interfaces for Information Search
Liao, Q. V., Geyer, W., Muller, M. J., & Khazaen, Y. (2020) · Human-computer interaction series · doi:10.1007/978-3-030-38825-6_13 · Cited by 57
“Recent work also uses the term “playfulness” to refer to intentional engagement of anthropomorphizing behaviors with an agent system [41].”
BackgroundInfluentialHow students (really) use ChatGPT: Uncovering experiences among undergraduate studentsPreprint
Ammari, T., Chen, M., Zaman, S. M. M., & Garimella, K. V. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.7292200 · Cited by 49
Mavrina, L., Szczuka, J., Strathmann, C., Bohnenkamp, L. M., Krämer, N., & Kopp, S. (2022) · Frontiers in Computer Science · doi:10.3389/fcomp.2022.791704 · Cited by 43
“Luger and Sellen (2016) and Cho et al. (2019) report that users with more experience with technology have more realistic ideas about the assistant, but those with less experience would mostly draw their expectations from familiar characteristics of human-human interaction due to insufficient…”
BackgroundInfluentialJung, J.-Y., Qiu, S., Bozzon, A., & Gadiraju, U. (2022) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3491102.3517653 · Cited by 43
Personal LLM Agents: Insights and Survey about the Capability, Efficiency and Security
Li, Y., Wen, H., Wang, W., Li, X., Yuan, Y., Liu, G., Liu, J., Xu, W., Wang, X., Sun, Y., et al. (2024) · arXiv · doi:10.48550/arxiv.2401.05459 · Cited by 43
Cambre, J., Reig, S., Kravitz, Q., & Kulkarni, C. E. (2020) · Designing Interactive Systems Conference · doi:10.1145/3357236.3395479 · Cited by 45
“In particular, users often anthropomorphize voice assistants, in large part due to human-like names, voices, and personalities the assistants are designed to convey [15,19,30], and interact with them as though they were social actors.”
BackgroundInfluentialPridmore, J., Zimmer, M. T., Vitak, J., Mols, A., Trottier, D., Kumar, P. C., & Liao, Y. (2019) · Surveillance & Society · doi:10.24908/ss.v17i1/2.12936 · Cited by 52
“' In line with prior research (Lopatovska et al. forthcoming; Luger and Sellen 2016) that notes the frequent use of these devices for information-related searches (e.g. , weather information), entertainment (e.g. , playing music) or controlling external devices (e.g. , turning lights on and off),…”
ResultBridging Screen Readers and Voice Assistants for Enhanced Eyes-Free Web Search
Vtyurina, A., Fourney, A., Morris, M. R., Findlater, L., & White, R. W. (2019) · The World Wide Web Conference · doi:10.1145/3308558.3314136 · Cited by 42
VERSE: Bridging Screen Readers and Voice Assistants for Enhanced Eyes-Free Web Search
Vtyurina, A., Fourney, A., Morris, M. R., Findlater, L., & White, R. W. (2019) · Proceedings of the 21st International ACM SIGACCESS Conference on Computers and Accessibility · doi:10.1145/3308561.3353773 · Cited by 42
Metaphors in Voice User Interfaces: A Slippery Fish
Desai, S., & Twidale, M. (2023) · ACM Transactions on Computer-Human Interaction · doi:10.1145/3609326 · Cited by 42
“Although consideration of metaphor occurs in many areas of computer science, including natural language processing [ 154 ], the most substantial discussion of metaphor occurs in HCI, often as a part of a mental model [ 77 , 99 ].”
BackgroundMethodInfluentialDesign Foundations for AI Assisted Decision Making: A Self Determination Theory Approach
De Vreede, T., Raghavan, M., & De Vreede, G.-J. (2021) · Proceedings of the Annual Hawaii International Conference on System Sciences · doi:10.24251/hicss.2021.019 · Cited by 42
“Studies show that chatbots that do not meet users’ relatedness needs often result in user dissatisfaction and frustration [38] [74].”
BackgroundTalk to Me on My Level – Linguistic Alignment for Chatbots
Spillner, L., & Wenig, N. (2021) · International Conference on Human-Computer Interaction with Mobile Devices and Services (MobileHCI) · doi:10.1145/3447526.3472050 · Cited by 50
Devices and Data and Agents, Oh My: How Smart Home Abstractions Prime End-User Mental Models
Clark, M., Newman, M., & Dutta, P. (2017) · Proceedings of the ACM on Interactive Mobile Wearable and Ubiquitous Technologies · doi:10.1145/3132031 · Cited by 42
“For example, while both Google Now and Siri are phone-based voice interfaces, research has found that users treat Google Now like “voice-activated search,” whereas they treat Siri like a social actor due to the latter’s presentation as a human-like agent [11].”
BackgroundVoice+Tactile: Augmenting In-vehicle Voice User Interface with Tactile Touchpad Interaction
Jung, J., Lee, S., Hong, J., Youn, E., & Lee, G. (2020) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376863 · Cited by 47
User interactions with “Alexa” in public academic space
Lopatovska, I., & Oropeza, H. (2018) · Proceedings of the Association for Information Science and Technology · doi:10.1002/pra2.2018.14505501034 · Cited by 41
“Luger and Sellen (2016) interviewed IPA users and found that poor understanding of IPA features and lack of feedback from the device result in frustrating experiences and discontinued use.”
BackgroundWhen There is No Progress with a Task-Oriented Chatbot
Li, C.-H., Chen, K., & Chang, Y.-J. (2019) · International Conference on Human-Computer Interaction with Mobile Devices and Services (MobileHCI) · doi:10.1145/3338286.3344407 · Cited by 41
Transformation through Provocation?
Roussou, M., Perry, S., Katifori, A., Vassos, S., Tzouganatou, A., & McKinney, S. (2019) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300857 · Cited by 42
“It seems that users expect what Cassell refers to as “interactional intelligence” [4], the “social smarts” that would enable engagement [5] [24].”
BackgroundResultOn Conversational Agents in Information Systems Research: Analyzing the Past to Guide Future Work.
Diederich, S., Brendel, A., & Kolbe, L. M. (2019) · WIRTSCHAFTSINFORMATIK · Cited by 73
Brokerbot: A Cryptocurrency Chatbot in the Social-technical Gap of Trust
Lee, M., Frank, L. E., & IJsselsteijn, W. A. (2021) · Computer Supported Cooperative Work (CSCW) · doi:10.1007/s10606-021-09392-6 · Cited by 45
“While chatbots can appear social, this increases users’ expectations of social intelligence that functional chatbots often do not meet (Luger and Sellen 2016), especially in new domains.”
BackgroundImmunizing with information – Inoculation messages against conversational agents’ response failures
Weiler, S., Matt, C., & Hess, T. (2022) · Electronic Markets · doi:10.1007/s12525-021-00509-9 · Cited by 40
“Often CA response failures lead to an immediate discontinuance of CA usage, giving the system no chance “to make it up” to the user (Ben Mimoun et al., 2012; Luger & Sellen, 2016).”
BackgroundUser Perceptions of Extraversion in Chatbots after Repeated Use
Völkel, S. T., Schoedel, R., Kaya, L., & Mayer, S. (2022) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3491102.3502058 · Cited by 53
Exploring the Learnability of Program Synthesizers by Novice Programmers
Jayagopal, D., Lubin, J., & Chasins, S. E. (2022) · Proceedings of the 35th Annual ACM Symposium on User Interface Software and Technology · doi:10.1145/3526113.3545659 · Cited by 40
“Some works offer guidelines for improving human-AI collaboration [2, 5, 28, 29, 36, 42, 52]; others detail the usability or learnability of particular classes of AI tools [17, 44, 49, 58, 74].”
BackgroundHow Do Data Analysts Respond to AI Assistance? A Wizard-of-Oz Study
Gu, K., Grunde-McLaughlin, M., McNutt, A., Heer, J., & Althoff, T. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3641891 · Cited by 40
University of Göttingen, Diederich, S., Lembcke, T.-B., University of Göttingen, Brendel, A. B., Dresden, T., Kolbe, L. M., & University of Göttingen (2021) · AIS Transactions on Human-Computer Interaction · doi:10.17705/1thci.00143 · Cited by 40
“Therefore, our experiment emphasizes the importance of managing and matching user expectations when designing (anthropomorphic) CAs as Luger and Sellen (2016) or Følstad and Brandtzæg (2017) have suggested.”
BackgroundMethodInfluentialPias, S. B. H., Huang, R., Williamson, D. S., Kim, M., & Kapadia, A. C. (2024) · ACM Conference on Conversational User Interfaces (CUI) · doi:10.1145/3640794.3665545 · Cited by 56
“Popular functionalities of voice assistants encompass information seeking, including weather checks, recent news, or cooking instruction [51, 72, 73], entertainment, such as music or jokes, and controlling external devices [72].”
BackgroundBraun, M., Broy, N., Pfleging, B., & Alt, F. (2019) · Journal on Multimodal User Interfaces · doi:10.1007/s12193-019-00301-2 · Cited by 42
Firefox Voice: An Open and Extensible Voice Assistant Built Upon the Web
Cambre, J., Williams, A. C., Razi, A., Bicking, I., Wallin, A., Tsai, J., Kulkarni, C., & Kaye, J. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445409 · Cited by 39
“Through analyses of usage logs [3, 64] and self-report data [43, 73], common patterns have emerged around the most commonly used assistant features: users frequently ask their voice assistants to answer search or informational-type queries, control music, set timers, and manage paired smart home devices [3, 43, 64, 73…”
BackgroundInfluential‘Alexa, I feel for you!’ Observers’ Empathetic Reactions towards a Conversational Agent
Carolus, A., Wienrich, C., Törke, A., Friedel, T., Schwietering, C., & Sperzel, M. (2021) · Frontiers in Computer Science · doi:10.3389/fcomp.2021.682982 · Cited by 36
“First, they constitute ‘ the next natural form of HCI ’ as voice controlling adopts a basic principle of humanity.”
BackgroundShaikh, S. J., & Cruz, I. (2022) · AI & Society · doi:10.1007/s00146-021-01335-5 · Cited by 44
“Intelligent assistants often rely on verbal commands that allow users to delegate assignments to them without entirely disengaging from one’s current task (Luger and Sellen 2016; Goksel and Mutlu 2016; López et al. 2018).”
BackgroundCho, E., Motalebi, N., Sundar, S. S., & Abdullah, S. (2022) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3555164 · Cited by 50
"All in the Same Boat": Tradeoffs of Voice Assistant Ownership for Mixed-Visual-Ability Families
Storer, K. M., Judge, T. K., & Branham, S. M. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376225 · Cited by 35
A Survey Investigating Usage of Virtual Personal AssistantsPreprint
Dubiel, M., Halvey, M. J., & Azzopardi, L. A. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1807.04606 · Cited by 45
“The evaluation of VPAs focused on different aspects that affect their continued use, such as usage experience [7, 10, 11], performance expectations [13], privacy concerns and social acceptability [8, 9].”
BackgroundAI Consent Futures: A Case Study on Voice Data Collection with Clinicians
Wilcox, L. G., Brewer, R. N., & Díaz, F. (2023) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3610107 · Cited by 45
Lin, M. P.-C., Chang, D., & Winne, P. H. (2024) · Educational Technology Research and Development · doi:10.1007/s11423-024-10408-3 · Cited by 40
“We incorporated several pictures of a dog to promote positive engagement (Jain et al., 2018; Luger & Sellen, 2016) and experienced teaching staff modified DD’s natural language (scripts) to make it more enjoyable.”
CoLadder: Manipulating Code Generation via Multi-Level Blocks
Yen, R., Zhu, J., Suh, S., Xia, H., & Zhao, J. (2024) · ACM Symposium on User Interface Software and Technology (UIST) · doi:10.1145/3654777.3676357 · Cited by 45
“Hence, prior research into natural language interfaces suggests the benefit of managing expectations and gradually revealing the capabilities of the system through user interaction and intervention [47, 48, 61, 69].”
BackgroundCo-designing voice user interfaces with teenagers in the context of smart homes
Fitton, D., Read, J. C., Sim, G., & Cassidy, B. (2018) · ACM International Conference on Interaction Design and Children (IDC) · doi:10.1145/3202185.3202744 · Cited by 38
“Prior work has highlighted the practical problems of using voice interaction for VPA contemporary technologies ([4],[13],[14]) and it is perhaps the unfamiliarity of the participants with these technologies that has led to this omission.”
BackgroundBigBlueBot: teaching strategies for successful human-agent interactions
Weisz, J. D., Jain, M., Joshi, N. N., Johnson, J., & Lange, I. (2019) · Proceedings of the 24th International Conference on Intelligent User Interfaces · doi:10.1145/3301275.3302290 · Cited by 38
Kendall, L., Chaudhuri, B., & Bhalla, A. (2020) · Information Systems Frontiers · doi:10.1007/s10796-020-10015-6 · Cited by 37
“Luger and Sellen (2016) explored the interactional factors affecting everyday use of different conversational agents across different devices in the United Kingdom.”
BackgroundInfluentialKocielnik, R., Langevin, R., George, J. S., Akenaga, S., Wang, A., Jones, D. P., Argyle, A., Fockele, C., Anderson, L., Hsieh, D. T., et al. (2021) · CUI 2021 - 3rd Conference on Conversational User Interfaces · doi:10.1145/3469595.3469599 · Cited by 37
“Luger et al.[33] in their work on speech-based CUIs pointed to time-saving as an important consideration for many users.”
BackgroundInfluentialCollecting Qualitative Data at Scale with Large Language Models: A Case Study
Cuevas, A., Scurrell, J. V., Brown, E. M., Entenmann, J., & Daepp, M. I. G. (2025) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3710947 · Cited by 46
Song, T., Tan, Y., Zhu, Z., Feng, Y., & Lee, Y. (2025) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3757633 · Cited by 49
Word Play: A History of Voice Interaction in Digital Games
Allison, F., Carter, M., & Gibbs, M. (2017) · Games and Culture · doi:10.1177/1555412017746305 · Cited by 37
“However, as several scholars have noted, there has been a comparative lack of research on the interface design and user applications of voice interfaces (Aylett et al., 2014; Luger & Sellen, 2016, p. 1; Munteanu et al., 2013).”
Exploring Effects of Conversational Fillers on User Perception of Conversational Agents
Jeong, Y., Lee, J., & Kang, Y. (2019) · Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290607.3312913 · Cited by 31
Xiao, Z., Zhou, M. X., Liao, Q. V., Mark, G. J., Chi, C., Chen, W., & Yang, H. (2019) · Cambridge University Engineering Department Publications Database · doi:10.48550/arxiv.1905.10700 · Cited by 27
“The general consensus is that conversational interfaces offer several advantages over traditional WIMP (Windows, Icons, Menus, and Pointers) interfaces [14, 56].”
BackgroundExploring the Design Space of User-System Communication for Smart-home Routine Assistants
Chiang, Y.-S., Chang, R.-C., Chuang, Y.-L., Chou, S.-Y., Lee, H.-P., Lin, I.-J., Chen, J.-H. J., & Chang, Y.-J. (2020) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376501 · Cited by 40
“Past studies of users’ mental models of intelligent systems have reported that such models can be shaped both by users’ technical backgrounds and by their previous experiences [35, 50].”
BackgroundDesigning Voice Interfaces: Back to the (Curriculum) Basics
Murad, C., & Munteanu, C. (2020) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376522 · Cited by 39
Hybrid intelligence in hospitals: towards a research agenda for collaboration
Mirbabaie, M., Stieglitz, S., & Frick, N. R. J. (2021) · Electronic Markets · doi:10.1007/s12525-021-00457-4 · Cited by 33
“Since there are various notations for the term “ conversational agents ” that are used interchangeably (Luger and Sellen 2016), we linked the most common ones within the search string “ chat-bot ” , “ chatbot ” and “ virtual agent ” .”
BackgroundInfluentialAlgorithm awareness as an important internet skill: the case of voice assistants
Gruber, J., Hargittai, E., Karaoglu, G., & Brombach, L. (2021) · International journal of communication · doi:10.5167/uzh-204503 · Cited by 56
“A study based on interviews with 14 voice assistant users found that half of the participants did not know about all the capabilities of their voice assistants (Luger & Sellen, 2016).”
BackgroundMake your own: The Potential of Chatbot Customization for the Development of User Trust
Wald, R., Heijselaar, E., & Bosse, T. (2021) · ACM Conference on User Modeling, Adaptation and Personalization (UMAP) · doi:10.1145/3450614.3463600 · Cited by 39
Wang, Q., Jing, S., & Goel, A. K. (2022) · Designing Interactive Systems Conference · doi:10.1145/3532106.3533534 · Cited by 48
“…AI agents that interact with emotionally vulnerable populations could have huge consequences on users’ wellbeing— AI agents could display functionalities and social characteristics that either mitigate users’ negative emotional states [20, 45] or add on to users’ emotional burden [9, 41, 81].”
BackgroundDippold, D. (2023) · Journal of Pragmatics · doi:10.1016/j.pragma.2022.12.004 · Cited by 37
What's The Talk on VUI Guidelines? A Meta-Analysis of Guidelines for Voice User Interface Design
Murad, C., Candello, H., & Munteanu, C. (2023) · Proceedings of the 5th International Conference on Conversational User Interfaces · doi:10.1145/3571884.3597129 · Cited by 36
“have more human-like conversational abilities than they currently have [11, 26, 62, 65] not moving much farther from the capabilities of decades-old prototypes such as ELIZA [97].”
BackgroundInfluentialAlghamdi, E., Halvey, M., & Nicol, E. (2024) · ACM Conversational User Interfaces 2024 · doi:10.1145/3640794.3665558 · Cited by 36
Assessing the Utility of the System Usability Scale for Evaluating Voice-based User Interfaces
Ghosh, D., Foong, P. S., Zhang, S., & Zhao, S. (2018) · Proceedings of the Sixth International Symposium of Chinese CHI · doi:10.1145/3202667.3204844 · Cited by 36
Bräuer, P., & Mazarakis, A. (2022) · International Journal of Human-Computer Interaction · doi:10.1080/10447318.2022.2160228 · Cited by 43
“Luger and Sellen (2016) point out that to interact with an IVA, users need time to grasp the scope of interaction options.”
Background‘It’s not a romantic relationship’: Stories of Adoption and Abandonment of Smart Speakers at Home
Voit, A., Niess, J., Eckerth, C., Ernst, M., Weingärtner, H., & Woźniak, P. W. (2020) · International Conference on Mobile and Ubiquitous Multimedia (MUM) · doi:10.1145/3428361.3428469 · Cited by 40
Stieglitz, S., Mirbabaie, M., Möllmann, N. R. J., & Rzyski, J. (2021) · Information Systems Frontiers · doi:10.1007/s10796-021-10201-0 · Cited by 50
“Furthermore, individuals have certain expectations towards VAs that use human-like design characteristics which might not be in line with their actual capabilities or purpose (Luger & Sellen, 2016).”
BackgroundVoice Agents Supporting High-Quality Social Play
Pantoja, L. S., Diederich, K., Crawford, L., & Hourcade, J. P. (2019) · Proceedings of the 18th ACM International Conference on Interaction Design and Children · doi:10.1145/3311927.3323151 · Cited by 35
Petsolari, M., Ibrahim, S. B., & Slovak, P. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642619 · Cited by 35
“…technologies, often called "conversational AI", "dialogue systems", "chat-bots", "virtual assistants" or "conversational agents" (i.e., both text-based and speech-or voice-based modalities for input and output) [128, 175, 221, 273, 279], have increasingly been adopted by parents and children alike.”
BackgroundConversational Agents Trust Calibration: A User-Centred Perspective to Design
Dubiel, M., Daronnat, S., & Leiva, L. A. (2022) · Proceedings of the 4th Conference on Conversational User Interfaces · doi:10.1145/3543829.3544518 · Cited by 27
Wienrich, C., Carolus, A., Markus, A., Augustin, Y., Pfister, J., & Hotho, A. (2023) · Computers · doi:10.3390/computers12040077 · Cited by 39
“Currently, IVAs are not able to initiate conversations and their dialogue flow often fails to meet users’ expectations [104].”
BackgroundTransparency Enhances Positive Perceptions of Social Artificial Intelligence
Xu, Y., Bradford, N., & Garg, R. (2023) · Human Behavior and Emerging Technologies · doi:10.1155/2023/5550418 · Cited by 42
Sugisaki, K., & Bleiker, A. (2020) · Mensch und Computer (MuC) · doi:10.1145/3404983.3405505 · Cited by 43
Finding a New Voice: Transitioning Designers from GUI to VUI Design
Murad, C., Munteanu, C., Cowan, B. R., & Clark, L. (2021) · ACM Conference on Conversational User Interfaces (CUI) · doi:10.1145/3469595.3469617 · Cited by 37
Metadata
- Authors
- Luger, E., & Sellen, A.
- Affiliations
- Microsoft Research (United Kingdom) (GB)
- Venue
- Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems
- Publisher
- ACM
- Published
- 2016-05-07
- Pages
- 5286-5297
- OpenAlex
- W2405187948
- Semantic Scholar
- 814e62bb4f0ec6facd640fbd3e3fcbd189716d20
- Cited by
- 1,176
- References
- 51
- Citations by year
- 2026: 164 · 2025: 175 · 2024: 119 · 2023: 116 · 2022: 107 · 2021: 148 · 2020: 141 · 2019: 126 · 2018: 54 · 2017: 25 · 2016: 1
- FWCI
- 37.00
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- CHI Conference on Human Factors in Computing Systems (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Speech and dialogue systems
- Social Sciences › Business, Management and Accounting › Marketing › AI in Service Interactions
- Physical Sciences › Computer Science › Human-Computer Interaction › Innovative Human-Technology Interaction
- Keywords
- chatbots, voice assistants, everyday use, user expectations, human-computer interaction, multimodality, system capabilities, interface design
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.