"Like Having a Really Bad PA": The Gulf between User Expectation and Experience of Conversational Agents
The DOI matches its Crossref record; 49 of the 51 references it cites were found in a registry or an index.
Abstract
The past four years have seen the rise of conversational agents (CAs) in everyday life. Apple, Microsoft, Amazon, Google and Facebook have all embedded proprietary CAs within their software and, increasingly, conversation is becoming a key mode of human-computer interaction. Whilst we have long been familiar with the notion of computers that speak, the investigative concern within HCI has been upon multimodality rather than dialogue alone, and there is no sense of how such interfaces are used in everyday life. This paper reports the findings of interviews with 14 users of CAs in an effort to understand the current interactional factors affecting everyday use. We find user expectations dramatically out of step with the operation of the systems, particularly in terms of known machine intelligence, system capability and goals. Using Norman's 'gulfs of execution and evaluation' [30] we consider the implications of these findings for the design of future systems.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Whereas, we found that the challenges our participants experienced were more elementary.”
From Exploring How Older Adults Use a Smart Speaker–Based Voice Assistant in Their First Interactions: Qualitative Study (2021) · cites this paper as [32]
In the citing paper
“In addition, it could be more important for novice users to have an appropriate mental model and understand the system rather than an efficient user experience [32].”
From Exploring How Older Adults Use a Smart Speaker–Based Voice Assistant in Their First Interactions: Qualitative Study (2021) · cites this paper as [32]
In the citing paper
“human-AI mixed-initiative presents a generally valuable and underexplored interface paradigm.”
From Mixed-Initiative Creative Interfaces (2017) · cites this paper as [20]
In the citing paper
“Not only do these breakdowns significantly diminish user satisfaction and damage trust, they also decrease the likelihood of continued use [18, 44].”
From System and User Strategies to Repair Conversational Breakdowns of Spoken Dialogue Systems: A Scoping Review (2024) · cites this paper as [44]
In the citing paper
“This is because when voice assistants fail, such as by incorrectly answering a question, it derails user trust [27, 35].”
From A Mixed-Methods Approach to Understanding User Trust after Voice Assistant Failures (2023) · cites this paper as [35]
17 more claims about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
How Many Interviews Are Enough?: An Experiment with Data Saturation and Variability
Guest, G., Bunce, A., & Johnson, L. (2006) · Field Methods · doi:10.1177/1525822x05279903 · Cited by 20,808
(2010) · Sociology for Pharmacists · doi:10.4324/9780203381175_chapter_9 · Cited by 14,152
Norman, D. (2016) · Vahlen eBooks · doi:10.15358/9783800648108 · Cited by 6,374
Thematic networks: an analytic tool for qualitative research
Attride-Stirling, J. (2001) · Qualitative Research · doi:10.1177/146879410100100307 · Cited by 5,506 · cited 2×
“The thematic network is a ‘web-like coding structure’ [2; 385] that allows for a rich understanding of the conceptual interplay occurring within the narrative.”
Background(1997) · Choice Reviews Online · doi:10.5860/choice.34-3702 · Cited by 4,887
ELIZA—a computer program for the study of natural language communication between man and machine
Weizenbaum, J. (1966) · Communications of the ACM · doi:10.1145/365153.365168 · Cited by 4,452
ELIZA — a computer program for the study of natural language communication between man and machine
Weizenbaum, J. (1983) · Communications of the ACM · doi:10.1145/357980.357991 · Cited by 2,738
Licklider, J. C. R. (1960) · IRE Transactions on Human Factors in Electronics · doi:10.1109/thfe2.1960.4503259 · Cited by 1,938 · cited 1×
“When considering human-machine dialogue from an HCI perspective, some of the earliest work can be traced back to vision statements and position papers such as JCR Licklider (1960), ‘Man-machine symbiosis’ [25] and the early work of Richard Bolt [4].”
MethodEmbodied Conversational Agents
(2000) · The MIT Press eBooks · doi:10.7551/mitpress/2697.001.0001 · Cited by 1,799
Interaction Ritual: Essays on Face-to-face Behaviour
Goffman, E. (1972) · Medical Entomology and Zoology · Cited by 1,449
Research works printed but not linked: 14
- Boden Margaret A, 2007, Proceedings of Artificial Companions in Society: Perspectives on the Present and Future. Oxford Internet Institute.
- Davis Martin, Engines of Logic: Mathematicians and the Origin of the Computer
- Glass James R., 1999, Challenges for Spoken Dialogue Systems. In the proceedings of IEEE Workshop on Automatic Speech Recognition and Understanding. (ASRU), CO, USA
- 10.5555/1941863 · doi:10.5555/1941863
- Heuwinkel Kerstin, Your Virtual Butler: The Making-of. Robert Trappl (ed)
- Robnett Licklider Joseph Carl, 1960, HFE-1,(March
- Moore Roger, Spoken Language Processing: Where do we go from Here? In Your Virtual Butler: The Making-of. Robert Trappl (ed)
- Payr Sabine, Your Virtual Butler: The Making-of. Robert Trappl (ed)
- Preece Jenny, 2015, Interaction Design: Beyond Huma- Computer Interaction
- Pulman Stephen, Proceedings of the 2010 Workshop on Companionable Dialogue Systems. Assoc. for Computational Linguistics
- Trappl Robert, From Jeeves Jeannie to Siri, and Then? In Your Virtual Butler: The Making-of. Robert Trappl (ed)
- Vassallo Giorgio, Semantic Computing. Sheu, Yu, Ramamoorthy, Joshi & Zadeh.(eds)
- 10.5555/1870559.1870562 · doi:10.5555/1870559.1870562
- Wilks Yorick, 2010, Amsterdam: John Benjamins.
Cited by
Items 101–200 of 1,007
Toward a design theory for virtual companionship
Strohmann, T., Siemon, D., Khosrawi-Rad, B., & Robra-Bissantz, S. (2023) · Human–Computer Interaction · doi:10.1080/07370024.2022.2084620 · Cited by 80
“Research suggests that such negative perceptions of CAs are due to a mismatch between the mental model and user expectations (Luger & Sellen, 2016).”
BackgroundTeach AI How to Code: Using Large Language Models as Teachable Agents for Programming Education
Jin, H., Lee, S., Shin, H., & Kim, J. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642349 · Cited by 99
“While using better LLMs and a more refined pipeline can alleviate the problem, we argue that reducing learners’ expectation gap in understanding the capabilities of teachable agents and their reasoning process is also fundamental in the context of LBT with AI [2, 36].”
Chaves, A. P., Egbert, J., Hocking, T., Doerry, E., & Gerosa, M. A. (2022) · ACM Transactions on Computer-Human Interaction · doi:10.1145/3487193 · Cited by 76
Coached Conversational Preference Elicitation: A Case Study in Understanding Movie Preferences
Radlinski, F., Balog, K., Byrne, B., & Krishnamoorthi, K. (2019) · Annual Meeting of the Special Interest Group on Discourse and Dialogue · doi:10.18653/v1/w19-5941 · Cited by 126
“Yet user interaction is often relatively unnatural (Luger and Sellen, 2016).”
BackgroundVoice assistants and older people: some open issues
Sayago, S., Neves, B. B., & Cowan, B. R. (2019) · Proceedings of the 1st International Conference on Conversational User Interfaces · doi:10.1145/3342775.3342803 · Cited by 97
Chaves, A. P., & Gerosa, M. A. (2020) · Figshare · doi:10.6084/m9.figshare.13207221 · Cited by 96
Empathy Is All You Need: How a Conversational Agent Should Respond to Verbal Abuse
Chin, H., Molefi, L. W., & Yi, M. Y. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376461 · Cited by 96
“When using the term conversational agent, people might refer to a chatbot, virtual assistant, interface agent, embodied conversational agent, or avatar [6, 41].”
BackgroundLee, K., Lee, K. Y., & Sheehan, L. (2020) · Information Systems Frontiers · doi:10.1007/s10796-019-09975-1 · Cited by 95
“2018), Conversational Agent (CA) (Luger and Sellen 2016), or Home Digital Voice Assistants (HDVAs) (Lei et al.”
BackgroundWhy do people use ChatGPT? Exploring user motivations for generative conversational AI
Skjuve, M., Brandtzaeg, P. B., & Følstad, A. (2024) · First Monday · doi:10.5210/fm.v29i1.13541 · Cited by 95
Exploring User Expectations of Proactive AI Systems
Meurisch, C., Mihale-Wilson, C., Hawlitschek, A., Giger, F., Müller, F., Hinz, O. F., & Mühlhäuser, M. (2020) · Proceedings of the ACM on Interactive Mobile Wearable and Ubiquitous Technologies · doi:10.1145/3432193 · Cited by 112
“However, latest studies [7, 12, 16, 19, 35] have revealed that user expectations of such AI”
BackgroundInfluentialVoice as a Design Material: Sociophonetic Inspired Design Strategies in Human-Computer Interaction
Sutton, S. J., Foulkes, P., Kirk, D., & Lawson, S. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300833 · Cited by 92
“Such arguments are motivated by research questions around the alignment of user expectations with technological possibilities of not just VUIs but conversational interfaces in general (see [9, 16, 42]).”
BackgroundWhat Are You Talking To?: Understanding Children's Perceptions of Conversational Agents
Xu, Y., & Warschauer, M. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376416 · Cited by 71
“This is not surprising as the ability to converse intelligently is a defining feature of CAs [32].”
BackgroundData Protection in AI Services: A Survey
Meurisch, C., & Mühlhäuser, M. (2021) · ACM Computing Surveys · doi:10.1145/3440754 · Cited by 119
“Probably, the most prominent representatives of this class are conversational agents [60], which have made the transition from research prototypes to consumer products in the current decade: Apple Siri (2011), Google Assistant (formerly Google Now, 2012), Microsoft Cortana (2015), Amazon Alexa (2015), to name a few [9…”
BackgroundChildren asking questions: speech interface reformulations and personification preferences
Yarosh, S., Thompson, S., Watson, K., Chase, A., Senthilkumar, A., Yuan, Y., & Brush, A. J. B. (2018) · Proceedings of the 17th ACM Conference on Interaction Design and Children · doi:10.1145/3202185.3202207 · Cited by 84
More than Smart Speakers: Security and Privacy Perceptions of Smart Home Personal Assistants
Abdi, N., Ramokapane, K. M., & Such, J. M. (2019) · Research Portal (King's College London) · Cited by 230
Virtual assistants and self-driving cars
Lugano, G. (2017) · International Conference on ITS Telecommunications · doi:10.1109/itst.2017.7972192 · Cited by 110
“Even in the role of virtual butler, users' expectations seem to exceed the actual performance of virtual assistants [19], thus reducing users'”
BackgroundOnce a Kind Friend is Now a Thing: Understanding How Conversational Agents at Home are Forgotten
Cho, M., Lee, S.-S., & Lee, K.-P. (2019) · Proceedings of the 2019 on Designing Interactive Systems Conference · doi:10.1145/3322276.3322332 · Cited by 90
“In the adoption stage, users tried entertaining commands, such as telling a joke or singing a song, but this only led them to explore more fun things, not other practical commands, so entertainment-based features were not associated with long-term use [24].”
BackgroundInfluentialAn Empirical Study of Older Adult’s Voice Assistant Use for Health Information Seeking
Brewer, R., Pierce, C., Upadhyay, P., & Park, L. (2022) · ACM Transactions on Interactive Intelligent Systems · doi:10.1145/3484507 · Cited by 89
“Beyond ASR errors, prior work suggests there is a need to better understand users’ mental models of information search with voice systems to improve informational utility beyond factual search [27, 32, 77].”
BackgroundEliciting and Analysing Users’ Envisioned Dialogues with Perfect Voice Assistants
Völkel, S. T., Buschek, D., Eiband, M., Cowan, B. R., & Hussmann, H. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445536 · Cited by 76
“This selection of scenarios corresponds to similar analyses of everyday use of voice user interfaces as described by prior research [3, 21, 48] and consumer reports [41, 42].”
BackgroundMethodInfluentialImproving Human-AI Collaboration With Descriptions of AI Behavior
Cabrera, Á. A., Perer, A., & Hong, J. I. (2023) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3579612 · Cited by 87
“often expect AI models to behave as humans do, which often does not match their actual behavior [40] and makes people unaware of how an AI may fail [25, 65].”
BackgroundKim, Y., Lee, J., Kim, S., Park, J., & Kim, J. (2024) · Proceedings of the 29th International Conference on Intelligent User Interfaces · doi:10.1145/3640543.3645148 · Cited by 55
Exploring and Promoting Diagnostic Transparency and Explainability in Online Symptom Checkers
Tsai, C.-H., You, Y., Gui, X., Kou, Y., & Carroll, J. M. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445101 · Cited by 86
“The design of conversational agents offer several advantages over traditional WIMP (Windows, Icons, Menus, and Pointers) interfaces [6, 43]: 1) a natural and familiar way for users to tell the system about themselves, which in turn improves the usability of a system as well as updates the user’s mental model to the sy…”
BackgroundElshan, E., Zierau, N., Engel, C., Janson, A., & Leimeister, J. M. (2022) · Information Systems Frontiers · doi:10.1007/s10796-021-10230-9 · Cited by 77
Leveraging Large Language Models to Power Chatbots for Collecting User Self-Reported Data
Wei, J., Kim, S., Jung, H., & Kim, Y.-H. (2024) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3637364 · Cited by 84
Mirbabaie, M., Stieglitz, S., Brünker, F., Hofeditz, L., Ross, B., & Frick, N. R. J. (2021) · Business & Information Systems Engineering · doi:10.1007/s12599-020-00672-x · Cited by 84
“As most VAs are designed as supportive tools (Lamontagne et al. 2014) and not as equivalent virtual team members, they still remain IS (Luger and Sellen 2016).”
Zargham, N., Reicherts, L., Bonfert, M., Voelkel, S. T., Schoening, J., Malaka, R., & Rogers, Y. (2022) · Proceedings of the 4th Conference on Conversational User Interfaces · doi:10.1145/3543829.3543834 · Cited by 84
“Since today’s conversations with voice assistants are highly constrained, task-oriented, and impersonal [8–10, 12, 22, 29], proactive interactions in such devices could open up new opportunities and potentially empower a broad range of applications [42].”
BackgroundExploring the Role of Conversational Cues in Guided Task Support with Virtual Assistants
Vtyurina, A., & Fourney, A. (2018) · Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3173782 · Cited by 83
“…using voice-based assistants on their phones, personal computers (PCs), and internet of things (IoT) devices to complete short tasks, such as setting up an alarm clock or checking weather forecasts [17, 21], as well as for multi-step tasks such as making a restaurant reservation or booking a taxi.”
BackgroundEvolving Roles and Workflows of Creative Practitioners in the Age of Generative AI
Palani, S., & Ramos, G. (2024) · Creativity and Cognition · doi:10.1145/3635636.3656190 · Cited by 58
Investigating Users' Preferences and Expectations for Always-Listening Voice Assistants
Tabassum, M., Kosiński, T., Frik, A., Malkin, N., Wijesekera, P., Egelman, S., & Lipford, H. R. (2019) · Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies · doi:10.1145/3369807 · Cited by 72
“[21] found that there is a disconnect between user expectations and abilities of current voice assistants: users expect the conversational agents to infer the context of the current task from the previous interactions.”
BackgroundThe Last Decade of HCI Research on Children and Voice-based Conversational Agents
Garg, R., Cui, H., Seligson, S., Zhang, B., Porcheron, M., Clark, L., Cowan, B. R., & Beneteau, E. (2022) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3491102.3502016 · Cited by 70
Martínez-Miranda, J., Martínez, A., Ramos, R., Aguilar, H., Jiménez, L., Arias, H., Rosales, G., & Valencia, E. (2019) · Journal of Medical Systems · doi:10.1007/s10916-019-1387-1 · Cited by 79
Design guidelines for hands-free speech interaction
Murad, C., Munteanu, C., Clark, L., & Cowan, B. R. (2018) · Proceedings of the 20th International Conference on Human-Computer Interaction with Mobile Devices and Services Adjunct · doi:10.1145/3236112.3236149 · Cited by 79
“Another issue was a lack of feedback of how user speech input was processed [12].”
BackgroundInfluentialChatbot-based Emotion Management for Distributed Teams: A Participatory Design Study
Benke, I., Knierim, M. T., & Maedche, A. (2020) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3415189 · Cited by 66
Computers as Bad Social Actors: Dark Patterns and Anti-Patterns in Interfaces that Act Socially
Alberts, L., Lyngs, U., & Kleek, M. G. V. (2024) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3653693 · Cited by 94
Portela, M., & Granell-Canut, C. (2017) · Proceedings of the XVIII International Conference on Human Computer Interaction · doi:10.1145/3123818.3123826 · Cited by 66
Informing Age-Appropriate AI: Examining Principles and Practices of AI for Children
Wang, G., Zhao, J., Van Kleek, M., & Shadbolt, N. (2022) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3491102.3502057 · Cited by 76
“key questions explored by the community is how children connect with the non-living, AI-based systems, which may have a physical representation [83, 107, 126] or sometimes not necessarily [90, 147].”
BackgroundBranham, S. M., & Mukkath Roy, A. R. (2019) · Proceedings of the 21st International ACM SIGACCESS Conference on Computers and Accessibility · doi:10.1145/3308561.3353797 · Cited by 75
Toward Voice Query Clarification
Kiesel, J., Bahrami, A., Stein, B., Anand, A., & Hagen, M. (2018) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3209978.3210160 · Cited by 75
Choice of Voices: A Large-Scale Evaluation of Text-to-Speech Voice Quality for Long-Form Content
Cambre, J., Colnago, J., Maddock, J., Tsai, J., & Kaye, J. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376789 · Cited by 65
“In studies of users’ experiences with today’s common voice assistants, researchers found that the human-like names, personalities, and voices the assistants take on mislead users into expecting human-like intelligence [14, 31].”
BackgroundNeed Help? Designing Proactive AI Assistants for Programming
Chen, V., Zhu, A., Zhao, S., Mozannar, H., Sontag, D. A., & Talwalkar, A. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3714002 · Cited by 105
“Many prior deployed proactive assistants have failed or received significant negative reaction because the actual capabilities of these systems do not meet user expectations [33, 34], as many of the systems relied on pre-set messaging or simple models [15, 29].”
Crowdsourcing Perceptions of Fair Predictors for Machine Learning: A Recidivism Case Study
van Berkel, N., Goncalves, J., Hettiachchi, D., Wijenayake, S., Kelly, R. M., & Kostakos, V. (2019) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3359130 · Cited by 60
Convey: Exploring the Use of a Context View for Chatbots
Jain, M., Kota, R., Kumar, P., & Patel, S. N. (2018) · Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3174042 · Cited by 74
“3174042 between experience and expectation with respect to both intelligence and the user interface of chatbots [13, 14, 17].”
BackgroundInfluentialExternalized domestication: smart speaker assistants, networks and domestication theory
Brause, S. R., & Blank, G. (2020) · Information, Communication & Society · doi:10.1080/1369118x.2020.1713845 · Cited by 65
“From the participants’ accounts it became clear that there is no archetypical user, and that they have been developing a variety of use genres related to SSAs beyond the previously known ‘convenience’ (Luger & Sellen, 2016) and ‘entertainment’ (Lopatovska et al., 2018).”
BackgroundSocially situated artificial intelligence enables learning from human interaction
Krishna, R., Lee, D., Fei-Fei, L., & Bernstein, M. S. (2022) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2115730119 · Cited by 63
Emotional Dialogue Generation using Image-Grounded Language Models
Huber, B., McDuff, D., Brockett, C., Galley, M., & Dolan, B. (2018) · Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3173851 · Cited by 76
“Conversational agents often do not meet users’ expectations [27].”
BackgroundKocaballi, A. B., Quiroz, J. C., Rezazadegan, D., Berkovsky, S., Magrabi, F., Coiera, E., & Laranjo, L. (2020) · Journal of Medical Internet Research · doi:10.2196/15823 · Cited by 73
“Although natural language interfaces facilitate intuitive user-system interactions with minimal training [2], they bring about a new set of challenges mainly caused by the lack of visibility of a system’s operations [3], resulting in unrealistic expectations about the capabilities of a system [4].”
BackgroundCelino, I., & Re Calegari, G. (2020) · International Journal of Human-Computer Studies · doi:10.1016/j.ijhcs.2020.102410 · Cited by 63
“Avoiding the use of NLU, our goal is to overcome the limits in user experience when the system does not fully “understand” user’s utterances, thus failing to meet user expectations, as illustrated by Luger and Sellen (2016).”
BackgroundMethodExploring how politeness impacts the user experience of chatbots for mental health support
Bowman, R., Cooney, O., Newbold, J. W., Thieme, A., Clark, L., Doherty, G. J., & Cowan, B. R. (2023) · International Journal of Human-Computer Studies · doi:10.1016/j.ijhcs.2023.103181 · Cited by 81
Kuang, E., Li, M., Fan, M., & Shinohara, K. (2024) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642168 · Cited by 86
“A single moderator served as the UX assistant to respond to impromptu questions.”
BackgroundValidity of Chatbot Use for Mental Health Assessment: Experimental Study
Schick, A., Feine, J., Morana, S., Maedche, A., & Reininghaus, U. (2022) · JMIR mHealth and uHealth · doi:10.2196/28082 · Cited by 58
Supporting Inclusive Learning Using Chatbots? A Chatbot-Led Interview Study
Gupta, S., & Chen, Y. (2022) · Journal of Information Systems Education · doi:10.62273/tvtf4203 · Cited by 69
“The advancement of artificial intelligent (AI) and conversational agents (Luger & Sellen, 2016), such as chatbots (Grudin & Jacques, 2019), has provided promising opportunities for assisting student life in higher education (Goel & Polepeddi, 2016).”
Kocaballi, A. B., Sezgin, E., Clark, L., Carroll, J. M., Huang, Y., Huh-Yoo, J., Kim, J., Kocielnik, R., Lee, Y.-C., Mamykina, L., et al. (2022) · Journal of Medical Internet Research · doi:10.2196/38525 · Cited by 62
“Difficulties in fostering user engagement have been highlighted when CAs cease to perform their intended utility [68].”
User Experience with Smart Voice Assistants: The Accent Perspective
Pal, D., Arpnikanondt, C., Funilkul, S., & Varadarajan, V. (2019) · 2019 10th International Conference on Computing, Communication and Networking Technologies (ICCCNT) · doi:10.1109/icccnt45670.2019.8944754 · Cited by 60
“The current evaluation of the VA’s focus on three main aspects: the usage experiences [3-5], performance expectations [7-8], and the privacy concerns [9-10].”
BackgroundCha, I., Kim, S. I., Hong, H., Yoo, H., & Lim, Y.-K. (2021) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445116 · Cited by 86
An End-to-End Conversational Style Matching Agent
Hoegen, R., Aneja, D., McDuff, D., & Czerwinski, M. (2019) · Proceedings of the 19th ACM International Conference on Intelligent Virtual Agents · doi:10.1145/3308532.3329473 · Cited by 51
“ons has been done with embodied agents [10, 13]. This large gulf in expectations is perhaps part of the reason why conversational agents are only used for very simple tasks and often disappoint users [23]. It has been shown that rapport and information disclosure are higher with assistants that can engage in social di…”
BackgroundMorrison, C., Cutrell, E., Dhareshwar, A., Doherty, K., Thieme, A., & Taylor, A. (2017) · Proceedings of the 19th International ACM SIGACCESS Conference on Computers and Accessibility · doi:10.1145/3132525.3132530 · Cited by 69
“While intended for the mainstream, it is people with visual disabilities who have become especially proficient power users of conversational agents [30].”
BackgroundDoyle, P. R., Clark, L., & Cowan, B. R. (2021) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445206 · Cited by 78
“Work suggests that early attention to anthropomorphic characteristics leads to high expectations in regard to competence and dependability, which are quickly identified as unrealistic following interactions [88].”
BackgroundInfluentialAbdolrahmani, A., Storer, K. M., Roy, A. R. M., Kuber, R., & Branham, S. M. (2019) · ACM Transactions on Accessible Computing · doi:10.1145/3368426 · Cited by 68
“1), as well as sighted individuals who have their hands or eyes preoccupied [Luger and Sellen 2016]. Additionally, previous research by Rodrigues et al. [2018] found that switching contexts between multiple devices increases users’ cognitive workload.”
BackgroundImproving Worker Engagement Through Conversational Microtask Crowdsourcing
Qiu, S., Gadiraju, U., & Bozzon, A. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376403 · Cited by 65
Monsters, Metaphors, and Machine Learning
Dove, G., & Fayard, A.-L. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376275 · Cited by 59
“However, these artifacts may also encourage unrealistic expectations [99], or fail to respond in a way that follows user intent [176].”
BackgroundLee, L., Braud, T., Hosio, S. J., & Hui, P. (2021) · ACM Computing Surveys · doi:10.1145/3467963 · Cited by 95
“Users feel more uncertain with more complicated and important tasks without appropriate visual confirmations to voice commands [112].”
BackgroundTowards Ecologically Valid Research on Language User Interfaces
de Vries, H., Bahdanau, D., & Manning, C. D. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2007.14435 · Cited by 40
“Such studies on LUIs are commonly found in the Human Computer Interaction (HCI) community, e.g. by conducting interviews with real-world users of commercial personal assistants (Luger and Sellen, 2016; Cowan et al., 2017).”
“Combating replay attacks against voice assistants,”
Pradhan, S., Sun, W., Baig, G., & Qiu, L. (2019) · Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies · doi:10.1145/3351258 · Cited by 67
Cracking the code: Co-coding with AI in creative programming education
Jonsson, M., & Tholander, J. (2022) · Creativity and Cognition · doi:10.1145/3527927.3532801 · Cited by 67
“These accounts resonate with previous research on interactions with voice-based conversational agents, such as a study by Luger and Sellen [22] where they found that the users of such systems systematically tested appropriate speech syntax in order to ’speak its language’ and to create a dynamic mental model of its’ c…”
BackgroundResultInfluentialWu, Y., Rough, D., Bleakley, A., Edwards, J., Cooney, O., Doyle, P. R., Clark, L., & Cowan, B. R. (2020) · 22nd International Conference on Human-Computer Interaction with Mobile Devices and Services · doi:10.1145/3379503.3403563 · Cited by 46
Social Boundaries for Personal Agents in the Interpersonal Space of the Home
Luria, M., Zheng, R., Huffman, B., Huang, S., Zimmerman, J., & Forlizzi, J. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376311 · Cited by 66
Myers, C. M., Furqan, A., & Zhu, J. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300277 · Cited by 58
“Key Finding #1: Programming experience did not have a wide-spread impact on performance metrics Luger and Sellen [17] found that participants more technically knowledgeable were self-reported as beingmore patient with errors and willing to say more utterances to accomplish a task with voice-only and visual VUIs.”
BackgroundInfluentialBaroni, I., Re Calegari, G., Scandolari, D., & Celino, I. (2022) · Human Computation · doi:10.15346/hc.v9i1.134 · Cited by 65
“, 2019), or when chatbots reveal a clear mismatch between the user expectations and the actual interaction experience (Luger and Sellen, 2016).”
BackgroundHow Do Analysts Understand and Verify AI-Assisted Data Analyses?
Gu, K., Shang, R., Althoff, T., Wang, C., & Drucker, S. M. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642497 · Cited by 37
“Likewise, since AI-generated analyses can involve sequences of data operations, it is pertinent to create test cases for both the final output and intermediate steps.”
Ahmed, N., Saha, A. K., Noman, M. A. A., Jim, J. R., Mridha, M. F., & Kabir, M. M. (2024) · Natural Language Processing Journal · doi:10.1016/j.nlp.2024.100112 · Cited by 69
Rapp, A., Lodovico, C. D., & Di, L. (2025) · International Journal of Human-Computer Studies · doi:10.1016/j.ijhcs.2025.103471 · Cited by 79
(2018) · doi:10.1145/3170427.3185372 · Cited by 64
“This technology push may have come at the cost of developing chatbots that actually respond to user needs and desires [5].”
BackgroundSelf-citationMore to Meetings: Challenges in Using Speech-Based Technology to Support Meetings
McGregor, M., & Tang, J. C. (2017) · Proceedings of the 2017 ACM Conference on Computer Supported Cooperative Work and Social Computing · doi:10.1145/2998181.2998335 · Cited by 53
“Luger and Sellen characterise the particular form of command-dialogue assistants like Siri, Cortana, Amazon Echo and GoogleNow as a ‘Conversational Agent’ [24].”
BackgroundMirbabaie, M., Hofeditz, L., Frick, N., & Stieglitz, S. (2021) · AI & Society · doi:10.1007/s00146-021-01239-4 · Cited by 69
“The implementation of AI recently became more distributed in hospitals worldwide (Knijnenburg and Wil-lemsen 2016; Luger and Sellen 2016; Li et al. 2019b), creating discernible benefits assisting medical experts in hospitals (Rauschert et al. 2020; Rong et al. 2020).”
BackgroundRose, E. J., & Björling, E. A. (2017) · Proceedings of the 35th ACM International Conference on the Design of Communication · doi:10.1145/3121113.3121212 · Cited by 60
“As technologies continue to move into all aspects of daily life, including conversational agents [43] and wearable technologies [51, 75, 81], researchers and designers need to consider how to design for engagement between people and things [5].”
BackgroundMeasuring User Experience in Conversational Interfaces: A Comparison of Six Questionnaires
Kocabalil, A. B., Laranjo, L., & Coiera, E. (2018) · Electronic Workshops in Computing · doi:10.14236/ewic/hci2018.21 · Cited by 57
“However, there are also other more emphatic and pragmatic user experience evaluation perspectives involving qualitative methods such as conversational analysis (Porcheron et al. 2018) and in-depth interviews with users (Luger & Sellen 2016).”
BackgroundImpertinent mobiles - Effects of politeness and impoliteness in human-smartphone interaction
Carolus, A., Muench, R., Schmidt, C., & Schneider, F. (2019) · Computers in Human Behavior · doi:10.1016/j.chb.2018.12.030 · Cited by 57
Choi, D., Kwak, D., Cho, M., & Lee, S. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376569 · Cited by 54
“As CAs become more commonly used while people are at home or are on-the-go, HCI researchers have been increasingly working on these types of interfaces [9, 22, 23, 28, 42, 45, 58, 59].”
BackgroundBeyond self-diagnosis: how a chatbot-based symptom checker should respond
You, Y., Tsai, C.-H., Li, Y., Ma, F., Heron, C., & Gui, X. (2023) · ACM Transactions on Computer-Human Interaction · doi:10.1145/3589959 · Cited by 60
“Compared to traditional WIMP (Windows, Icons, Menus, and Pointers) interfaces, chatbots provide a familiar way for users to disclose information about themselves and are flexible enough to accommodate various user requests rather than having a fixed path for users to follow [17, 92, 146, 148].”
BackgroundVilaza, G. N., & McCashin, D. (2021) · Frontiers in Digital Health · doi:10.3389/fdgth.2021.689736 · Cited by 60
“Furthermore, explicability is related to challenges communicating the limitations of chatbots’ artificially created dialogues to end-users (52).”
BackgroundUnderstanding Public Perceptions of AI Conversational Agents: A Cross-Cultural Analysis
Liu, Z., Li, H., Chen, A., Zhang, R., & Lee, Y.-C. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642840 · Cited by 60
Rapp, A., Boldi, A., Curti, L., Perrucci, A., & Simeoni, R. (2023) · International Journal of Human-Computer Interaction · doi:10.1080/10447318.2023.2247596 · Cited by 71
“When users ’ expectations are not met, bad conversations may occur: these have been shown to decrease the users ’ willingness to ask the agent for help with the task (Luger & Sellen, 2016) and to lead to the abandonment of the interaction (Crolic et al., 2022; Jain et al., 2018).”
Manifestation of virtual assistants and robots into daily life: vision and challenges
Rawassizadeh, R., Sen, T., Kim, S. J., Meurisch, C., Keshavarz, H., Mühlhäuser, M., & Pazzani, M. J. (2019) · CCF Transactions on Pervasive Computing and Interaction · doi:10.1007/s42486-019-00014-1 · Cited by 71
“Wireless connectivity greatly enhances user experience, but the usual risks associated with network dependence (including privacy, battery life, and network reliability) persist (Luger and Sellen 2016).”
BackgroundArmutat, S., Wattenberg, M., & Mauritz, N. (2024) · International Conference on Gender Research · doi:10.34190/icgr.7.1.2163 · Cited by 58
A Conversation Analysis of Non-Progress and Coping Strategies with a Banking Task-Oriented Chatbot
Li, C.-H., Yeh, S.-F., Chang, T.-J., Tsai, M.-H., Chen, K., & Chang, Y.-J. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376209 · Cited by 57
User Perception of Text-Based Chatbot Personality
Ruane, E., Farrell, S., & Ventresque, A. (2021) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-68288-0_3 · Cited by 57
“…may be of interest, we decided to create two chatbots with personalities that combine high extraversion with high agreeableness (and vice versa) due to their complementary linguistic presentations and previous work ([15] [14]) that suggests users respond to distinct agent personalities.”
BackgroundSeeking empathy or suggesting a solution? Effects of chatbot messages on service failure recovery
Haupt, M., Rozumowski, A., Freidank, J., & Haas, A. (2023) · Electronic Markets · doi:10.1007/s12525-023-00673-0 · Cited by 57
“Indeed, conversational agents were found to respond more successfully when the input was rather simple, short, and unambiguous (Luger & Sellen, 2016).”
BackgroundGetting smart: towards critical digital literacy pedagogies
Knight, J., Dooly, M., & Barberà, E. (2020) · Social Semiotics · doi:10.1080/10350330.2020.1836815 · Cited by 62
“However, they are also increasingly popular with educational institutions as tools for a range of educational (and perhaps ideological) purposes, particularly when combined with AI systems like chatbots (Luger and Sellen 2016).”
BackgroundOwning and Sharing: Privacy Perceptions of Smart Speaker Users
Meng, N., Keküllüoğlu, D., & Vaniea, K. E. (2021) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3449119 · Cited by 82
“Previous research has focused on smart home device adoption, personification and privacy perceptions [1, 30, 37, 40, 41, 58, 70].”
BackgroundMusicBot: Evaluating Critiquing-Based Music Recommenders with Conversational Interaction
Jin, Y., Cai, W., Chen, L., Htun, N. N., & Verbert, K. (2019) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3357384.3357923 · Cited by 67
“However, a recent study [25] shows the Gulf between user expectation and experience with conversational agents, which may influence users to gauge the effort they spend on looking for a song.”
BackgroundSvenningsson, N., & Faraon, M. (2019) · Proceedings of the 2019 2nd Artificial Intelligence and Cloud Computing Conference · doi:10.1145/3375959.3375973 · Cited by 56
ChatrEx: Designing Explainable Chatbot Interfaces for Enhancing Usefulness, Transparency, and Trust
Khurana, A., Harjandi, P. A., & Chilana, P. K. (2021) · IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC) · doi:10.1109/vl/hcc51201.2021.9576440 · Cited by 68
“To show the real-time system status more interactively (as suggested in [9]), we adopted a design similar to the “Status Tracker” UI [48] and show the step-by-step explanations in the form of latest status and updates in a chronological order (Figure 1.b).”
BackgroundMethodInfluentialKuang, E., Soure, E. J., Fan, M., Zhao, J., & Shinohara, K. (2023) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581247 · Cited by 69
“assistants also showed that participants made use of tactics such as removing words other than keywords, reducing the number of words used, using more specific terms, altering enunciation, and speaking more slowly and clearly [49].”
BackgroundMethodInfluential"I don't know what you're talking about, HALexa": the case for voice user interface guidelines
Murad, C., & Munteanu, C. (2019) · Proceedings of the 1st International Conference on Conversational User Interfaces · doi:10.1145/3342775.3342795 · Cited by 52
“They have been advertised as a natural way to interact with technology [7, 9, 15].”
BackgroundCowan, B. R., Doyle, P. R., Edwards, J., Garaialde, D., Hayes-Brady, A., Branigan, H. P., Cabral, J. P., & Clark, L. (2019) · ACM Conference on Conversational User Interfaces (CUI) · doi:10.1145/3342775.3342786 · Cited by 60
“although we generally see machines as limited communicative partners [7, 14], design decisions used to portray humanness of speech systems may be important drivers of people’s initial partner models [17, 30], acting as a metaphor to support interaction. Crucially, these design decisions can overinflate perceptions of…”
BackgroundResultInfluentialPolite or Direct? Conversation Design of a Smart Display for Older Adults Based on Politeness Theory
Hu, Y., Qu, Y., Maus, A., & Mutlu, B. (2022) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3491102.3517525 · Cited by 43
“Despite the efforts towards building conversational agents in HCI research, the debate has been ongoing on whether voice interfaces should be conversational, natural or human-like [5, 13, 19, 29].”
BackgroundUnderstanding Motivators, Constraints, and Practices of Sharing Internet of Things
Garg, R., & Moreno, C. (2019) · Proceedings of the ACM on Interactive Mobile Wearable and Ubiquitous Technologies · doi:10.1145/3328915 · Cited by 61
Rese, A., & Tränkner, P. (2023) · International Journal of Information Management · doi:10.1016/j.ijinfomgt.2023.102699 · Cited by 70
User Simulation for Evaluating Information Access Systems
Balog, K., & Zhai, C. (2024) · Foundations and Trends® in Information Retrieval · doi:10.1561/1500000098 · Cited by 52
Chatbot commerce—How contextual factors affect Chatbot effectiveness
Hsu, P., Nguyen, T. K., Wang, C., & Huang, P. (2023) · Electronic Markets · doi:10.1007/s12525-023-00629-4 · Cited by 56
“The results (Table 1) reveal that more (e.g., Hill et al., 2015; Luger & Sellen, 2016; Radziwill & Benton, 2017).”
Metadata
- Authors
- Luger, E., & Sellen, A.
- Affiliations
- Microsoft Research (United Kingdom) (GB)
- Venue
- Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems
- Publisher
- ACM
- Published
- 2016-05-07
- Pages
- 5286-5297
- OpenAlex
- W2405187948
- Semantic Scholar
- 814e62bb4f0ec6facd640fbd3e3fcbd189716d20
- Cited by
- 1,176
- References
- 51
- Citations by year
- 2026: 164 · 2025: 175 · 2024: 119 · 2023: 116 · 2022: 107 · 2021: 148 · 2020: 141 · 2019: 126 · 2018: 54 · 2017: 25 · 2016: 1
- FWCI
- 37.00
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- CHI Conference on Human Factors in Computing Systems (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Speech and dialogue systems
- Social Sciences › Business, Management and Accounting › Marketing › AI in Service Interactions
- Physical Sciences › Computer Science › Human-Computer Interaction › Innovative Human-Technology Interaction
- Keywords
- chatbots, voice assistants, everyday use, user expectations, human-computer interaction, multimodality, system capabilities, interface design
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.